SAR (Synthetic Aperture Radar) image and optical image fusion method and device and electronic equipment

Abstract

The invention discloses a method and device for fusing an SAR image and an optical image and electronic equipment, and belongs to the technical field of remote sensing image processing, and the method comprises the following steps: obtaining uncorrected data; the SAR geometric positioning model is combined with the reference topographic data to carry out geometric correction on the SAR image, and filtering processing is carried out; carrying out registration on the SAR image and the optical image; performing NSST multi-scale and multi-direction image decomposition on the registered images to obtain low-frequency images and high-frequency sub-band images in different directions; the MSMG is combined with the FMFD to determine a fusion weight, decision diagram fusion is carried out on the low-frequency image, and a low-frequency sub-image of the fused image is obtained; calculating an image mean value of the high-frequency sub-band image, and fusing the sub-images of the same decomposition level to obtain a high-frequency sub-image of the fused image; and performing NSST inverse transformation on the low-frequency sub-image and the high-frequency sub-image to generate a fused image. According to the invention, high-quality fusion of the SAR image and the optical image is realized.

Description

Technical Field

[0001] The invention relates to a method, a device and electronic equipment for fusing SAR images with optical images, and belongs to the technical field of remote sensing image processing. Background Art

[0002] With the development of aerospace and remote sensing imaging technology, sensors such as visible light, multi / hyperspectral, infrared, synthetic aperture radar, and lidar have emerged, providing a large amount of data covering the air, space, and ground for earth observation. A single sensor faces various limitations such as a single band, narrow application scenarios, and low spatial resolution. These limitations affect the comprehensiveness and breadth of application of remote sensing image data, which also puts higher requirements on data joint processing applications.

[0003] SAR images and optical images are two important data types in satellite earth observation. Each has its own unique advantages and can complement each other. In an ideal imaging environment, optical images can capture detailed surface texture features and spectral information of various types of objects. However, in areas with frequent fog and rainfall, the quality of optical images is affected by the continuous obstruction of clouds. As an active imaging radar, SAR has the characteristics of all-weather, multi-band and multi-polarization. It also has excellent penetration ability and is not affected by external factors when acquiring data, thus overcoming the shortcomings of optical imaging being restricted by weather. However, the side-view imaging technology used by SAR leads to speckle noise in the image, which affects the clarity of the texture of the object and may cause the loss of some local details. Therefore, in order to obtain more comprehensive, accurate and reliable remote sensing image information, overcome the defect of limited information of a single sensor, and improve the comprehensive analysis ability and application value of remote sensing image data, SAR image and optical image registration and fusion technology came into being.

[0004] The difficulty of fusing SAR images with optical images is the preservation of the spectral and texture information of SAR images and optical images. At present, the main image fusion methods are mainly divided into three levels, namely pixel level, feature level and decision level. Pixel-level data fusion is performed at the pixel level. By directly fusing the pixels at the corresponding positions in the original image to generate a new image, it can usually provide more comprehensive context information, but the computational complexity is high and the robustness to image noise is poor. Feature-level data fusion is a method of integrating feature information from different data sources or modalities. By extracting and combining features from different sources, a more informative and expressive feature representation is obtained. Decision-level data fusion is a method of integrating decision or classification results from different data sources or modalities. It does not directly fuse the original data or features, but fuses the final decision or classification results to obtain a more comprehensive and accurate final judgment. Compared with other fusion levels, decision-level fusion is faster in processing time and requires relatively less data, which makes it more resistant to noise interference. However, this fusion method also has certain limitations, especially in terms of image information retention, which may cause serious information loss and lead to texture blur. In order to alleviate this problem, a series of complex preprocessing steps including image enhancement, noise reduction and feature extraction are usually required to improve the quality of the fused image by optimizing the original data to ensure the accuracy and reliability of the fusion result. Summary of the invention

[0005] In order to solve the above-mentioned texture blur problem between the fusion of SAR images and optical images, the present invention proposes a method, device and electronic equipment for fusing SAR images and optical images, which can achieve high-quality fusion of SAR images and optical images.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] In a first aspect, an embodiment of the present invention provides a method for fusing a SAR image with an optical image, comprising the following steps:

[0008] Step 1, obtain uncorrected SAR images, optical images with geographic coordinates and reference terrain data;

[0009] Step 2, using the SAR geometric positioning model combined with reference terrain data to geometrically correct the SAR image, and filtering the corrected SAR image;

[0010] Step 3, registering the SAR image and the optical image using their pixel-by-pixel dense geometric structure features and dense optical flow model;

[0011] Step 4, perform NSST (Nonsubsampled Shearlet Transform) multi-scale and multi-directional image decomposition on the registered SAR image and optical image respectively to obtain their low-frequency images and high-frequency sub-band images in different directions;

[0012] Step 5: Multi-scale morphological gradient (MSMG) is combined with Focus Measurement based on Fractal Dimension (FMFD) to determine the fusion weight, and the low-frequency image is fused by decision graph to obtain the low-frequency sub-image of the fused image.

[0013] Step 6, calculating the image mean of the high-frequency sub-band image, and using a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level, to obtain a high-frequency sub-image of the fused image;

[0014] Step 7: Perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

[0015] As a possible implementation of this embodiment, step 2 includes the following steps:

[0016] A strict geometric positioning model is established according to the geometric imaging parameters of the SAR image, and a mapping relationship between geographic coordinates and image coordinates is established; the geometric positioning model adopts the range-Doppler (RD) model:

[0017] F R (X,Y,Z,x,y)=0

[0018] F D (X,Y,Z,x,y)=0

[0019] Among them, F R is the observation equation of range imaging condition, F D is the Doppler imaging condition observation equation, (X, Y, Z) is the geographic coordinate, (x, y) is the image coordinate;

[0020] Based on the geometric positioning model and the reference terrain data, the SAR image is geometrically corrected to obtain the corrected SAR image, and the resolution of the corrected SAR image is consistent with the optical image;

[0021] The speckle noise in the SAR image is filtered and denoised by using an edge-preserving exponential selection filter, and finally a filtered SAR image is obtained;

[0022] The filter is:

[0023]

[0024] Among them, I s and I o Represent the grayscale values ​​of the SAR image and the filtered image, respectively, and E PI is the filtered SAR image, and its value range is 0 to 1.

[0025] As a possible implementation of this embodiment, step 3 includes the following steps:

[0026] Calculate pixel-by-pixel dense geometric structure features of SAR images and optical images;

[0027] A coarse-to-fine matching strategy is adopted to construct a downsampled Gaussian pyramid of geometric structure feature descriptors, calculate the optical flow starting from the feature map at the highest level of the pyramid, and match the feature map together with the optical flow vector;

[0028] The optical flow is divided into horizontal and vertical directions. The optimal optical flow vector is obtained through iterative convergence of the double-layer cyclic belief propagation algorithm. The final optical flow field is obtained by step-by-step transmission and is used to realize the registration of SAR images and optical images.

[0029] As a possible implementation of this embodiment, the step of calculating pixel-by-pixel dense geometric structure features of the SAR image and the optical image includes the following steps:

[0030] The SAR image gradient is calculated using the ROEWA operator, and the optical image gradient is calculated using the Sobel operator;

[0031] The gradient magnitude and gradient direction of the SAR image and the optical image are calculated respectively, and the gradient direction is normalized to [0, 180], and evenly divided into 8 direction intervals with an interval of 22.5°. The angle θ corresponding to each interval is i The direction index table is [0, 22.5, …, 157.5], and the direction index numbers are recorded as 1 to 8 in sequence;

[0032] Construct a statistical histogram of gradient information pixel by pixel, open a 5×5 statistical window with the pixel as the center, and add the corresponding gradient amplitude to the corresponding direction interval according to the index number to obtain the eight-dimensional gradient feature vector of the pixel;

[0033] The feature vector of each pixel is placed along the z-axis perpendicular to the image plane to obtain the geometric structure feature expression, and filtering and smoothing are performed in the z-axis direction to obtain a three-dimensional image geometric structure feature map.

[0034] As a possible implementation of this embodiment, step 4 includes the following steps:

[0035] The registered SAR image and optical image are processed by non-subsampled shearlet transform respectively;

[0036] The non-subsampled Laplace pyramid filter bank (NSPFB) is used for multi-scale decomposition, and the shearlet filter (SF) is used for multi-directional decomposition to obtain low-frequency images of SAR images and optical images and a series of high-frequency sub-band images in different directions.

[0037] As a possible implementation of this embodiment, step 5 includes the following steps:

[0038] According to the multi-scale structural elements and gradient features, the MSMG is obtained by calculating the weighted sum of all scale gradients;

[0039] Calculate FMFD of SAR images and optical images;

[0040] The decision graphs are fused according to the decision graph rules to obtain low-frequency fused images.

[0041] As a possible implementation of this embodiment, the step 5, specifically,

[0042] MSMG is calculated by image gradient, and the multi-scale structured element is defined as:

[0043]

[0044] Where SE1 represents the basic structural element; τ represents the number of scales; represents the morphological dilation operator; the gradient feature G τ It is represented by the gradient operator from the image:

[0045]

[0046] Where (x, y) represents the pixel coordinates; Θ represents the erosion operator;

[0047] According to the multi-scale structural elements and gradient features, the MSMG of SAR images and optical images is obtained by calculating the weighted sum of all scale gradients:

[0048]

[0049] Among them, MSMG Ls (x,y) represents the SAR image MSMG of pixel (x,y), MSMG Lo (x,y) represents the optical image MSMG of pixel (x,y), w τ Represents the weight when the gradient scale is τ;

[0050] Calculate the FMFD of SAR image and optical image separately:

[0051]

[0052]

[0053] In the formula, are the maximum and minimum grayscale values ​​in the 3×3 window centered at the (x,y)th pixel in the SAR and optical low-frequency images, respectively;

[0054] According to the decision graph rules, the decision graph is fused to obtain the low-frequency fused image L F :

[0055]

[0056] As a possible implementation of this embodiment, step 6 includes the following steps:

[0057] A dual-channel adaptive parameter pulse coupled neural network is used for high-frequency image fusion;

[0058] By comparing the internal activities of the images, a decision graph is constructed to fuse the high-frequency sub-band images and obtain the high-frequency fused sub-band images.

[0059] As a possible implementation of this embodiment, in step 7, the expression of the NSST inverse transform is:

[0060]

[0061] Among them, I F To fuse the images, L F (x,y) is the low-frequency fusion image, It is a high-frequency fusion sub-band image.

[0062] In a second aspect, an embodiment of the present invention provides a device for fusing SAR images and optical images, comprising:

[0063] Data acquisition module, used to acquire uncorrected SAR images, optical images with geographic coordinates and reference terrain data;

[0064] The data processing module is used to geometrically correct the SAR image using the SAR geometric positioning model combined with the reference terrain data, and to filter the corrected SAR image;

[0065] An image registration module is used to register SAR images with optical images using their pixel-by-pixel dense geometric structure features and dense optical flow models;

[0066] The image decomposition module is used to perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain their low-frequency images and high-frequency sub-band images in different directions;

[0067] The low-frequency sub-image fusion module is used to determine the fusion weight by combining MSMG with FMFD, and to perform decision graph fusion on the low-frequency images to obtain the low-frequency sub-image of the fused image;

[0068] The high-frequency sub-image fusion module is used to calculate the image mean of the high-frequency sub-band image, and use a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level to obtain the high-frequency sub-image of the fused image;

[0069] The NSST inverse transformation module is used to perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

[0070] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of any of the above-mentioned methods for fusing SAR images with optical images.

[0071] In a fourth aspect, an embodiment of the present invention provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for fusing SAR images with optical images are executed.

[0072] The beneficial effects of the technical solution of the embodiment of the present invention are as follows:

[0073] The present invention realizes image decomposition based on NSST, utilizes a dual-channel adaptive parameter pulse coupled neural network to realize the fusion of high-frequency sub-band images, utilizes multi-scale morphological gradient combined with focal measurement based on fractal dimension to realize low-frequency image fusion, and finally realizes the fusion of SAR image and optical image through NSST inverse transform. It not only solves the texture blur problem between SAR image and optical image, retains the detail information of SAR image and visible light image respectively, but also realizes the high-quality fusion of SAR image and optical image. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of a method for fusing SAR images and optical images according to an exemplary embodiment;

[0075] Figure 2 The figure is a schematic diagram of the structure of a device for fusing SAR images and optical images according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0077] like Figure 1 As shown, a method for fusing SAR images and optical images provided by an embodiment of the present invention includes the following steps:

[0078] Step 1, obtain uncorrected SAR images, optical images with geographic coordinates and reference terrain data;

[0079] Step 2, using the SAR geometric positioning model combined with reference terrain data to geometrically correct the SAR image, and filtering the corrected SAR image;

[0080] Step 3, registering the SAR image and the optical image using their pixel-by-pixel dense geometric structure features and dense optical flow model;

[0081] Step 4, perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain their low-frequency images and high-frequency sub-band images in different directions;

[0082] Step 5: MSMG combines FMFD to determine the fusion weight, and performs decision graph fusion on the low-frequency image to obtain the low-frequency sub-image of the fused image;

[0083] Step 6, calculating the image mean of the high-frequency sub-band image, and using a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level, to obtain a high-frequency sub-image of the fused image;

[0084] Step 7: Perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

[0085] As a possible implementation of this embodiment, step 2 includes the following steps:

[0086] A strict geometric positioning model is established according to the geometric imaging parameters of the SAR image, and a mapping relationship between geographic coordinates and image coordinates is established; the geometric positioning model adopts the range-Doppler (RD) model:

[0087] F R (X,Y,Z,x,y)=0

[0088] F D (X,Y,Z,x,y)=0

[0089] Among them, F R is the observation equation of range imaging condition, FD is the Doppler imaging condition observation equation, (X, Y, Z) is the geographic coordinate, (x, y) is the image coordinate;

[0090] Based on the geometric positioning model and the reference terrain data, the SAR image is geometrically corrected to obtain the corrected SAR image, and the resolution of the corrected SAR image is consistent with the optical image;

[0091] The speckle noise in the SAR image is filtered and denoised by using an edge-preserving exponential selection filter, and finally a filtered SAR image is obtained;

[0092] The filter is:

[0093]

[0094] Among them, I s and I o Represent the grayscale values ​​of the SAR image and the filtered image, respectively, and E PI is the filtered SAR image, and its value range is 0 to 1.

[0095] As a possible implementation of this embodiment, step 3 includes the following steps:

[0096] Calculate pixel-by-pixel dense geometric structure features of SAR images and optical images;

[0097] A coarse-to-fine matching strategy is adopted to construct a downsampled Gaussian pyramid of geometric structure feature descriptors, calculate the optical flow starting from the feature map at the highest level of the pyramid, and match the feature map together with the optical flow vector;

[0098] The optical flow is divided into horizontal and vertical directions. The optimal optical flow vector is obtained through iterative convergence of the double-layer cyclic belief propagation algorithm. The final optical flow field is obtained by step-by-step transmission and is used to realize the registration of SAR images and optical images.

[0099] As a possible implementation of this embodiment, the step of calculating pixel-by-pixel dense geometric structure features of the SAR image and the optical image includes the following steps:

[0100] The SAR image gradient is calculated using the ROEWA operator, and the optical image gradient is calculated using the Sobel operator;

[0101] The gradient magnitude and gradient direction of the SAR image and the optical image are calculated respectively, and the gradient direction is normalized to [0, 180], and evenly divided into 8 direction intervals with an interval of 22.5°. The angle θ corresponding to each interval is i The direction index table is [0, 22.5, …, 157.5], and the direction index numbers are recorded as 1 to 8 in sequence;

[0102] Construct a statistical histogram of gradient information pixel by pixel, open a 5×5 statistical window with the pixel as the center, and add the corresponding gradient amplitude to the corresponding direction interval according to the index number to obtain the eight-dimensional gradient feature vector of the pixel;

[0103] The feature vector of each pixel is placed along the z-axis perpendicular to the image plane to obtain the geometric structure feature expression, and filtering and smoothing are performed in the z-axis direction to obtain a three-dimensional image geometric structure feature map.

[0104] As a possible implementation of this embodiment, step 4 includes the following steps:

[0105] The registered SAR image and optical image are processed by non-subsampled shearlet transform respectively;

[0106] The non-subsampled Laplacian pyramid filter bank is used for multi-scale decomposition, and the shearlet filter is used for multi-directional decomposition to obtain low-frequency images of SAR images and optical images and a series of high-frequency sub-band images in different directions.

[0107] As a possible implementation of this embodiment, the step 4, specifically,

[0108] Assuming the number of layers is 2, non-subsampled shearlet transform processing is performed:

[0109] Ω AB (ψ)={ψ i,j,k (x)=|detA| j / 2 ψ(B l A j xk); j,l∈Z 2}

[0110] Among them, ψ∈L 2 (R 2 ), where A and B are 2*2 reversible matrices, |det B|=1, A j is the scaling transformation matrix, B l Involving rotation and shearing, Downsampling by size approximately 2 2j ×2 j The trapezoidal support has a slope of l2-j, which is the scale. If f∈L 2 (R 2 ), the continuous shear wave transform is expressed as

[0111] SH ψ = <f,ψ j,l,k >

[0112] Where j≥0, l=-2 j , 2 j -1, k∈Z 2 ;

[0113] The NSST decomposition is represented using the following two related functions:

[0114]

[0115] Where nsst_de represents NSST decomposition, which is used to obtain SAR and optical low-frequency images. and high frequency sub-band images Where l and d represent the number of decomposition layers and decomposition directions, respectively, to achieve multi-scale and multi-directional decomposition of SAR and optical images.

[0116] As a possible implementation of this embodiment, step 5 includes the following steps:

[0117] According to the multi-scale structural elements and gradient features, the MSMG is obtained by calculating the weighted sum of all scale gradients;

[0118] Calculate FMFD of SAR images and optical images;

[0119] The decision graphs are fused according to the decision graph rules to obtain low-frequency fused images.

[0120] As a possible implementation of this embodiment, the step 5, specifically,

[0121] MSMG is calculated by image gradient, and the multi-scale structured element is defined as:

[0122]

[0123] Where SE1 represents the basic structural element; τ represents the number of scales; represents the morphological dilation operator; the gradient feature G τ It is represented by the gradient operator from the image:

[0124]

[0125] Where (x, y) represents the pixel coordinates; Θ represents the erosion operator;

[0126] According to the multi-scale structural elements and gradient features, the MSMG of SAR images and optical images is obtained by calculating the weighted sum of all scale gradients:

[0127]

[0128]

[0129] Among them, MSMG Ls (x,y) represents the SAR image MSMG of pixel (x,y), MSMG Lo(x,y) represents the optical image MSMG of pixel (x,y), w τ Represents the weight when the gradient scale is τ;

[0130] Calculate the FMFD of SAR image and optical image separately:

[0131]

[0132] In the formula, are the maximum and minimum grayscale values ​​in the 3×3 window centered at the (x,y)th pixel in the SAR and optical low-frequency images, respectively;

[0133] According to the decision graph rules, the decision graph is fused to obtain the low-frequency fused image L F :

[0134]

[0135] As a possible implementation of this embodiment, step 6 includes the following steps:

[0136] A dual-channel adaptive parameter pulse coupled neural network is used for high-frequency image fusion;

[0137] By comparing the internal activities of the images, a decision graph is constructed to fuse the high-frequency sub-band images and obtain the high-frequency fused sub-band images.

[0138] As a possible implementation of this embodiment, in step 7, the expression of the NSST inverse transform is:

[0139]

[0140] Among them, I F To fuse the images, L F (x,y) is the low-frequency fusion image, It is a high-frequency fusion sub-band image.

[0141] like Figure 2 As shown, an embodiment of the present invention provides a device for fusing SAR images and optical images, comprising:

[0142] Data acquisition module, used to acquire uncorrected SAR images, optical images with geographic coordinates and reference terrain data;

[0143] The data processing module is used to geometrically correct the SAR image using the SAR geometric positioning model combined with the reference terrain data, and to filter the corrected SAR image;

[0144] An image registration module is used to register SAR images with optical images using their pixel-by-pixel dense geometric structure features and dense optical flow models;

[0145] The image decomposition module is used to perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain their low-frequency images and high-frequency sub-band images in different directions;

[0146] The low-frequency sub-image fusion module is used to determine the fusion weight by combining MSMG with FMFD, and to perform decision graph fusion on the low-frequency images to obtain the low-frequency sub-image of the fused image;

[0147] The high-frequency sub-image fusion module is used to calculate the image mean of the high-frequency sub-band image, and use a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level to obtain the high-frequency sub-image of the fused image;

[0148] The NSST inverse transformation module is used to perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

[0149] The specific process of fusing SAR images and optical images using the device of the present invention is as follows.

[0150] First, obtain uncorrected SAR images, optical images with geographic coordinates, and reference terrain data.

[0151] Second, the SAR image is firstly corrected geometrically using the SAR geometric positioning model combined with reference terrain data. The correction resolution is consistent with the optical image resolution. Then, the appropriate filtering method is selected through the edge preservation coefficient to perform SAR image filtering and denoising.

[0152] The specific steps are:

[0153] ① A rigorous geometric positioning model is established based on the geometric imaging parameters of SAR images. The range-Doppler (RD) model is used as the geometric positioning model. The positioning model equation can be expressed as:

[0154] F R (X,Y,Z,x,y)=0

[0155] F D (X,Y,Z,x,y)=0

[0156] Among them, F R is the observation equation of range imaging condition, F D is the Doppler imaging condition observation equation, (X, Y, Z) is the geographic coordinate, (x, y) is the image coordinate. Thus, the mapping relationship between geographic coordinates and image coordinates can be established.

[0157] ② SAR image geometric correction is carried out by indirect sampling. The corrected image is established in the geographic coordinate system of the optical image, and the elevation value Z is obtained by interpolating the geographic coordinates (X, Y) of a certain pixel plane of the corrected image in combination with the reference terrain data (existing DEM data). According to the positioning model, the original SAR image coordinates (x, y) can be solved from (X, Y, Z) by indirect positioning, and the pixel gray value is obtained by interpolation according to the image coordinates. The corrected image is positioned and sampled pixel by pixel to obtain the corrected SAR image with the same resolution as the optical image.

[0158] ③Calculate the edge preservation index and select the appropriate filter:

[0159]

[0160] In the above formula, I s and I o Represent the grayscale values ​​of the SAR image and the filtered image respectively. The filtered SAR image E PI The value range is from 0 to 1. The larger the value, the more edge information is retained, and the final filtered SAR image is obtained.

[0161] Third, the registration of SAR and optical images is achieved by utilizing the pixel-by-pixel dense geometric structure features and dense optical flow model of SAR and optical images.

[0162] The specific steps are:

[0163] ① Calculate the pixel-by-pixel dense geometric structure features of SAR and optical images respectively. The geometric structure feature calculation first uses the ROEWA operator to calculate the SAR image gradient, and the Sobel operator to calculate the optical image gradient. Then, the gradient size and gradient direction are calculated based on the image gradient, and then the gradient direction is normalized to [0, 180), and evenly divided into 8 direction intervals with an interval of 22.5°. The angle θ corresponding to each interval is i The direction index number is [0, 22.5, ..., 157.5], and the direction index number is recorded as 1 to 8 to obtain the direction index table. Next, the statistical histogram of the gradient information is constructed pixel by pixel. Taking the pixel (a, b) as an example, a 5×5 statistical window is opened with the pixel as the center, and the corresponding gradient amplitude is accumulated to the corresponding direction interval according to the index number, and the eight-dimensional gradient feature vector of the pixel is obtained. Then, the feature vector of each pixel is placed along the z-axis perpendicular to the image plane to obtain the geometric structure feature expression, and the filtering and smoothing are performed in the z-axis direction to obtain the three-dimensional image geometric structure feature map.

[0164] ② Create an objective function, estimate the optical flow vector based on the feature maps of the two images, and match the feature maps with the optical flow vector. The objective function uses the following formula:

[0165]

[0166] E(w)=E data +E dis +E s

[0167] The objective function contains the data item E data , small displacement term E dis and the smoothing term E s . In the above formula, w(p) = (u(p), ν(p)) is the optical flow vector of point p in the image, the horizontal optical flow vector u(p) and the vertical optical flow vector ν(p) are both integers, f1 and f2 are the two feature images we want to match; η and α are weights, and ε contains all pixels in the four-neighborhood space. The data term constrains the image feature descriptor f to match the optical flow vector w(p), the small displacement term limits the flow vector value to be as small as possible, and the smoothing term constrains the flow vectors of adjacent pixels in the image to be similar. In this objective function, the data term and the smoothing term are constrained by the truncated L1 norm, and t and d are used as thresholds to explain the matching of outliers and flow discontinuities, respectively.

[0168] ③ A coarse-to-fine matching strategy is adopted to construct a downsampled Gaussian pyramid of the feature map. The double-layer cyclic belief propagation algorithm is used to divide the optical flow into horizontal and vertical directions. The optimal optical flow vector is obtained through iterative convergence. The optical flow is calculated starting from the feature map at the highest level of the pyramid, and the final optical flow field is obtained by transferring it step by step to achieve the registration of SAR and optical images.

[0169] Fourth, perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain low-frequency images of the SAR image and optical image and a series of high-frequency sub-band images in different directions.

[0170] Specifically, assuming the number of layers is 2,

[0171] Ω AB (ψ)={ψ i,j,k (x)=|det A| j / 2 ψ(B l A j xk); j,l∈Z 2}

[0172] Among them, ψ∈L 2 (R 2 ), where A and B are 2*2 reversible matrices, |det B|=1, A j is the scaling transformation matrix, B l Involving rotation and shearing, Downsampling by size approximately 2 2j ×2 jThe trapezoidal support has a slope of l2-j. k is the scale. If f∈L 2 (R 2 ), the continuous shear wave transform can be expressed as

[0173] SH ψ = <f,ψ j,l,k >

[0174] Where j≥0, l=-2 j , 2 j -1, k∈Z 2 The NSST decomposition is represented using the following two related functions.

[0175]

[0176] In the above formula, nsst_de represents NSST decomposition, which obtains SAR and optical low-frequency images. and high frequency sub-band images Where l and d represent the number of decomposition layers and decomposition directions, respectively, to achieve multi-scale and multi-directional decomposition of SAR and optical images.

[0177] Fifth, the fusion weights of the decomposed low-frequency images are determined according to MSMG combined with FMFD, and decision graph fusion is performed to obtain the low-frequency fused image.

[0178] The specific steps are:

[0179] ① Calculate MSMG through image gradient, and the multi-scale structured element is defined as:

[0180]

[0181] Where SE1 represents the basic structural element; τ represents the number of scales; represents the morphological dilation operator; the gradient feature G τ It can be represented by the gradient operator from the image:

[0182]

[0183] In the above formula, (x, y) represents the pixel coordinates; Θ represents the erosion operator. According to the multi-scale structural elements and gradient features, MSMG can be obtained by calculating the weighted sum of all scale gradients.

[0184]

[0185] Among them, w τ represents the weight when the gradient scale is τ.

[0186] ② Calculate FMFD of SAR and optical images:

[0187]

[0188] In the above formula, are the maximum and minimum grayscale values ​​in the 3×3 window centered at the (x,y)th pixel in the SAR and optical low-frequency images, respectively.

[0189] ③According to the decision diagram rule, the fused low-frequency fusion image L is obtained F :

[0190]

[0191] Sixth, the image mean of the high-frequency sub-band image is calculated, and the PCNN (Pulse Coupled Neural Network) model with dual-channel adaptive parameters is used to fuse the sub-images at the same decomposition level respectively to obtain the high-frequency sub-image of the fused image.

[0192] The specific steps are:

[0193] ① A dual-channel adaptive parameter pulse coupled neural network is used to realize high-frequency image fusion. The initial matrices such as F, L, U, and Y are set to 0. The mathematical model parameters are adaptively adapted through adaptive connection strength, which is specifically obtained by the following formula:

[0194]

[0195]

[0196] in and are the external stimuli of the PCNN model at the position (x, y) in the SAR and optical high-frequency sub-band images, i.e., the grayscale value of the image; and Store separately and The two symmetric inputs of remain unchanged during the iteration;

[0197]

[0198] In the above formula, L N (x,y) is the coupling connection of the (x,y)th neuron at the Nth iteration, C L is the coupling connection input amplitude, W L It is a 3*3 matrix, which represents the synaptic weights of the 8 neighboring neurons of the (x,y)th neuron, and m is a fixed value of 1.

[0199]

[0200] In the formula, and They represent the internal state of the (x, y)th neuron after the Nth iteration corresponding to the SAR and optical high-frequency sub-band images, respectively, and represent the nonlinear modulation of the coupling connection of the corresponding neurons and the external stimulus; U N (x, y) represents the internal activity of the (x, y)th neuron after the Nth iteration of the PCNN model. Its initial value is set to 0 and the value is calculated according to U N The calculation mechanism of (x,y) finally obtains a decision map.

[0201] β s (x,y) and β o (x,y) are and The corresponding connection strength reflects the response of the human visual system to pixels and their surrounding areas. The model uses adaptive connection strength and calculates the normalized FMFD to assign specific connection strength values ​​to the neurons in the model to achieve the adaptation of model parameters.

[0202]

[0203]

[0204] in, are the maximum and minimum grayscale values ​​in the 3×3 window centered at the (x,y)th pixel in the SAR and optical high-frequency sub-band images, respectively; FMFD Hs (x,y),FMFD Ho (x, y) are the FMFD calculated from SAR and optical high-frequency sub-band images, FMFD Hs (x,y) and FMFD Ho The maximum and minimum values ​​in (x, y). In order to control the connection strength between 0 and 1, a constant value is added to the formula; α u is the attenuation coefficient, and its value is estimated using the following formula:

[0205]

[0206] In the above formula, σ s and σ o are the standard deviations of SAR and optical high-frequency sub-band images, respectively, and their values ​​are between 0 and 1. It's U N The attenuation value corresponding to (x,y). Model output value Y N (x,y) is calculated as follows:

[0207]

[0208] Among them, ξN (x,y) is the dynamic threshold obtained by the PCNN model after N iterations; α ξ is the attenuation coefficient; C θ is the dynamic threshold amplitude.

[0209]

[0210] in, are the maximum values ​​in SAR and optical high-frequency sub-band images, as well as are the maximum inter-class variance values ​​of SAR and optical high-frequency sub-band images, respectively.

[0211] ②Construct a decision graph by comparing the internal activities of the image to achieve high-frequency sub-band image fusion:

[0212]

[0213] DM l,d (x, y) is a decision diagram, and the information with greater internal activity is selected as the information of the high-frequency fusion sub-band image.

[0214] Step 7: Perform NSST inverse transformation on the fused low-frequency sub-image and a series of high-frequency sub-images to generate a fused image.

[0215] Specifically, the fused image is calculated by the NSST inverse transform:

[0216]

[0217] Finally, a fused image is generated.

[0218] An electronic device provided by an embodiment of the present invention includes a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the above-mentioned methods for fusing SAR images with optical images.

[0219] Specifically, the above-mentioned memory and processor can be general-purpose memory and processor, which are not specifically limited here. When the processor runs the computer program stored in the memory, the above-mentioned method of fusing SAR images and optical images can be executed.

[0220] Those skilled in the art will appreciate that the structure of the electronic device does not limit the electronic device and may include more or fewer components than shown in the figure, or combine or split certain components, or arrange the components differently.

[0221] In some embodiments, the electronic device may also include a touch screen that can be used to display a graphical user interface (e.g., a startup interface of an application) and receive user operations on the graphical user interface (e.g., startup operations on an application). The specific touch screen may include a display panel and a touch panel. The display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. The touch panel may collect the user's contact or non-contact operations on or near it, and generate pre-set operation instructions, for example, the user uses any suitable object such as a finger, stylus, or accessories on or near the touch panel. In addition, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation and posture, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into information that the processor can process, and then sends it to the processor, and can receive and execute commands from the processor. In addition, the touch panel can be implemented by various types such as resistive, capacitive, infrared and surface acoustic wave, and any technology developed in the future can also be used to implement the touch panel. Further, the touch panel can cover the display panel, and the user can operate on or near the touch panel covered on the display panel according to the graphical user interface displayed on the display panel. After the touch panel detects the operation on or near it, it is transmitted to the processor to determine the user input, and then the processor provides corresponding visual output on the display panel in response to the user input. In addition, the touch panel and the display panel can be implemented as two independent components or integrated.

[0222] Corresponding to the method for starting the above application, an embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for fusing any SAR image with an optical image are executed.

[0223] The application startup device provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0224] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0225] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0226] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional module in the embodiments provided in the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0228] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0229] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for fusing SAR images with optical images, characterized in that: The steps include: Step 1, obtain uncorrected SAR images, optical images with geographic coordinates and reference terrain data; Step 2, using the SAR geometric positioning model combined with reference terrain data to geometrically correct the SAR image, and filtering the corrected SAR image; Step 3, registering the SAR image and the optical image using their pixel-by-pixel dense geometric structure features and dense optical flow model; Step 4, perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain their low-frequency images and high-frequency sub-band images in different directions; Step 5: MSMG combines FMFD to determine the fusion weight, and performs decision graph fusion on the low-frequency image to obtain the low-frequency sub-image of the fused image; Step 6, calculating the image mean of the high-frequency sub-band image, and using a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level, to obtain a high-frequency sub-image of the fused image; Step 7: Perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

2. The method for fusing SAR images and optical images according to claim 1, characterized in that: The step 2 comprises the following steps: A strict geometric positioning model is established according to the geometric imaging parameters of the SAR image, and a mapping relationship between geographic coordinates and image coordinates is established; the geometric positioning model adopts the range-Doppler model: F R (X,Y,Z,x,y)=0 F D (X,Y,Z,x,y)=0 Among them, F R is the observation equation of range imaging condition, F D is the Doppler imaging condition observation equation, (X, Y, Z) is the geographic coordinate, (x, y) is the image coordinate; Based on the geometric positioning model and the reference terrain data, the SAR image is geometrically corrected to obtain the corrected SAR image, and the resolution of the corrected SAR image is consistent with the optical image; The speckle noise in the SAR image is filtered and denoised by using an edge-preserving exponential selection filter, and finally a filtered SAR image is obtained; The filter is: Among them, I s and I o Represent the grayscale values ​​of the SAR image and the filtered image, respectively, and E PI is the filtered SAR image, and its value range is 0 to 1.

3. The method for fusing SAR images and optical images according to claim 1, characterized in that: The step 3 comprises the following steps: Calculate pixel-by-pixel dense geometric structure features of SAR images and optical images; A coarse-to-fine matching strategy is adopted to construct a downsampled Gaussian pyramid of geometric structure feature descriptors, calculate the optical flow starting from the feature map at the highest level of the pyramid, and match the feature map together with the optical flow vector; The optical flow is divided into horizontal and vertical directions. The optimal optical flow vector is obtained through iterative convergence of the double-layer cyclic belief propagation algorithm. The final optical flow field is obtained by step-by-step transmission and is used to realize the registration of SAR images and optical images.

4. The method for fusing SAR images and optical images according to claim 3, characterized in that: The method of calculating pixel-by-pixel dense geometric structure features of the SAR image and the optical image comprises the following steps: The SAR image gradient is calculated using the ROEWA operator, and the optical image gradient is calculated using the Sobel operator; The gradient magnitude and gradient direction of the SAR image and the optical image are calculated respectively, and the gradient direction is normalized to [0, 180], and evenly divided into 8 direction intervals with an interval of 22.5°. The angle θ corresponding to each interval is i The direction index table is [0, 22.5, …, 157.5], and the direction index numbers are recorded as 1 to 8 in sequence; Construct a statistical histogram of gradient information pixel by pixel, open a 5×5 statistical window with the pixel as the center, and add the corresponding gradient amplitude to the corresponding direction interval according to the index number to obtain the eight-dimensional gradient feature vector of the pixel; The feature vector of each pixel is placed along the z-axis perpendicular to the image plane to obtain the geometric structure feature expression, and filtering and smoothing are performed in the z-axis direction to obtain a three-dimensional image geometric structure feature map.

5. The method for fusing SAR images and optical images according to claim 1, characterized in that: The step 4 comprises the following steps: The registered SAR image and optical image are processed by non-subsampled shearlet transform respectively; The non-subsampled Laplacian pyramid filter bank is used for multi-scale decomposition, and the shearlet filter is used for multi-directional decomposition to obtain the low-frequency images of the SAR image and the optical image and a series of high-frequency sub-band images in different directions.

6. The method for fusing SAR images and optical images according to claim 1, characterized in that: The step 5 comprises the following steps: According to the multi-scale structural elements and gradient features, the MSMG is obtained by calculating the weighted sum of all scale gradients; Calculate FMFD of SAR images and optical images; The decision graphs are fused according to the decision graph rules to obtain low-frequency fused images.

7. The method for fusing SAR images and optical images according to claim 1, characterized in that: The step 6 comprises the following steps: A dual-channel adaptive parameter pulse coupled neural network is used for high-frequency image fusion; By comparing the internal activities of the images, a decision graph is constructed to fuse the high-frequency sub-band images and obtain the high-frequency fused sub-band images.

8. The method for fusing SAR images and optical images according to any one of claims 1 to 7, characterized in that: In step 7, the expression of the inverse NSST transform is: Among them, I F To fuse the images, L F (x,y) is the low-frequency fusion image, It is a high-frequency fusion sub-band image.

9. A device for fusing SAR images with optical images, characterized in that: include: Data acquisition module, used to acquire uncorrected SAR images, optical images with geographic coordinates and reference terrain data; The data processing module is used to geometrically correct the SAR image using the SAR geometric positioning model combined with the reference terrain data, and to filter the corrected SAR image; An image registration module is used to register SAR images with optical images using their pixel-by-pixel dense geometric structure features and dense optical flow models; The image decomposition module is used to perform NSST multi-scale and multi-directional image decomposition on the registered SAR image and optical image to obtain their low-frequency images and high-frequency sub-band images in different directions; The low-frequency sub-image fusion module is used to determine the fusion weight by combining MSMG with FMFD, and to perform decision graph fusion on the low-frequency images to obtain the low-frequency sub-image of the fused image; The high-frequency sub-image fusion module is used to calculate the image mean of the high-frequency sub-band image, and use a pulse coupled neural network model with dual-channel adaptive parameters to fuse the sub-images at the same decomposition level to obtain the high-frequency sub-image of the fused image; The NSST inverse transformation module is used to perform NSST inverse transformation on the obtained low-frequency sub-image and high-frequency sub-image to generate a fused image.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for fusing SAR images and optical images as described in any one of claims 1 to 8.

Images