Optical and SAR remote sensing image pixel-level fusion method and system

Abstract

The invention provides an optical and SAR remote sensing image pixel-level fusion method and system, and the method comprises the steps: S1, unifying optical and SAR images to a same geographic coordinate frame through a data preprocessing process, eliminating the influence of specific factors of a sensor, obtaining a fusible image group, and carrying out the fusion; s2, acquiring two types of image information to the greatest extent according to a fusion criterion that the variance of the fused image is the same as that of the optical image and the mean value is the same as that of the SAR image; and S3, evaluating the fused image by using information entropy, average gradient, mutual information and structural similarity as objective evaluation indexes, selecting an optimal fusion result by traversing a combination of a waveband channel of the optical image and a polarization channel of the SAR image, and determining an optimal image channel combination according to the evaluation result. The method has high actual operability and universality, the optical and SAR image fusion process can be simplified, and a good fusion result can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image information fusion, and in particular to a method and system for pixel-level fusion of optical and SAR remote sensing images. Background Art

[0002] Remote sensing imagery is an important means of obtaining surface geospatial information, conducting land resource assessments, and monitoring the environment. In most cases, the information provided by a single sensor is incomplete or insufficient. Therefore, combining imagery collected by optical and SAR sensors can yield complementary information.

[0003] Patent document (CN201810842295.5) discloses a method for fusing interferometric SAR complex images and optical images. The method proposes a fusion technology based on color theory. However, the method is too simple and the fusion information is not well utilized. Patent document (CN202210651992.9) discloses a high-precision fusion method for optical images and SAR intensity images. Patent document (202310312540.2) discloses a method and system for fusing SAR and optical images. A transform domain method is proposed, and the fused image is constructed by means such as wavelet transform. However, it is relatively complicated in actual operation. Patent document (CN202410052659.5) discloses a semantic segmentation method and device for fusing optical and SAR channel correlation. Patent document (CN202410659898.7) discloses a method for fusing visible light images and SAR images. However, in actual application, a large number of samples need to be trained for learning, which is relatively complicated. Summary of the Invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a method and system for pixel-level fusion of optical and SAR remote sensing images.

[0005] The method for pixel-level fusion of optical and SAR remote sensing images provided by the present invention includes:

[0006] Step S1: Through the data preprocessing process, the optical and SAR images are unified into the same geographic coordinate frame and the influence of sensor-specific factors is eliminated to obtain a fusible image group and fuse them;

[0007] Step S2: According to the fusion criteria that the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image, the two types of image information are obtained to the greatest extent;

[0008] Step S3: Use information entropy, average gradient, mutual information and structural similarity as objective evaluation indicators to evaluate the fused image. By traversing the band channel combination of the optical image and the polarization channel combination of the SAR image, the optimal fusion result is selected, and the optimal image channel combination is determined based on the evaluation results.

[0009] Preferably, the data preprocessing process in step S1 includes:

[0010] Selection of overlapping observation areas of optical and SAR images: Obtain overlapping observation areas of optical and SAR images by selecting geographic coordinates;

[0011] SAR image radiation correction: converting the backscatter intensity of the original SAR image into a comparable intensity;

[0012] SAR image speckle filtering: Through filtering, the intensity of speckles in SAR images is reduced, providing a basis for image fusion;

[0013] SAR image amplitude to decibel image conversion: Assume that the polarization channel of the SAR image is The expression for converting the amplitude graph into a decibel graph is:

[0014] σ HH =10log|S HH | 2 ,σ HV =10log|S HV | 2 ,σ HH =10log|S HH | 2 ,σ VV =10log|S VV | 2

[0015] Among them, R HH 、R HV 、R VH 、R VV Respectively represent the amplitude value of the horizontal HH polarization channel, the amplitude value of the cross HV polarization channel, the amplitude value of the cross VH polarization channel, and the amplitude value of the vertical VV polarization channel; σ HH , σ HV Represents the decibel diagram of horizontal HH polarization and the decibel diagram of cross HV polarization respectively; S HH 、S HV 、S VV Respectively represent the amplitude diagram of horizontal HH polarization, the amplitude diagram of cross HV polarization, and the amplitude diagram of vertical VV polarization;

[0016] SAR image terrain correction: First, combine the digital elevation model to select high-precision control points; then perform sensitivity analysis on the errors that affect the geometric positioning accuracy, including orbit determination error, slant range error, azimuth time error, and atmospheric delay error; establish the range-Doppler positioning equation to accurately estimate the geometric positioning error; after compensating for the geometric parameter errors, use geocoding to convert the image coordinate space into a geographic coordinate system.

[0017] Preferably, step S2 includes:

[0018] Assume that the optical image O k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image;

[0019] Let the fused image be F k,l , then the construction of the fused image is expressed as:

[0020] F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n)

[0021] Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of optical and SAR images respectively.

[0022] Preferably, to ensure that the local variance in the fused image is equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are retained in the fused image, the sliding window is processed one by one, and the image row number m and column number n are omitted. The expression is:

[0023]

[0024] Among them, Cov(F k,l ,F k,l ) is expressed as F k,l variance; and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window;

[0025] To ensure that the local mean in the fused image is equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image, the expression is:

[0026] Mean(Fk,l )=ω O μ O +ω R μ R =μ R

[0027] Among them, Mean(F k,l ) is expressed as F k,l The mean of μ O and μ R are the mean values of the optical image and SAR image pixels within the window respectively;

[0028] Solve the above two equations together to obtain two fusion coefficients ω O and ω R , the expression is:

[0029]

[0030] If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then the choice is to retain ω O Greater than ω R Solution; if in the fusion process, in order to remove the occlusion in the optical image, then choose to retain ω O Less than ω R The solution.

[0031] Preferably, the step S3 includes: establishing a fusion image evaluation criterion and an optimization model, and performing an evaluation on the fused image F k,l ,k=1,2,...,l=1,2,...,where k and l represent the band channel of the optical image and the polarization channel of the SAR image respectively. A variety of objective evaluation indicators are used to comprehensively evaluate the target image. The selected objective evaluation indicators are divided into single image evaluation indicators and joint image evaluation indicators. By traversing k=1,2,...,l=1,2,..., the band channel of the optical image and the polarization channel of the SAR image are fused in pairs, and the optimal fusion result is selected;

[0032] Step S3.1: Establish a single image evaluation model based on information entropy. The size of information entropy reflects the amount of information carried by the image. The larger the information entropy of the image, the richer its information content and the better its quality. The information entropy is calculated as follows:

[0033]

[0034] In the formula, H represents the information entropy of the image, P ij Represents the frequency of gray level F(i,j) in the image; i and j represent the row and column of image pixel values respectively;

[0035] Step S3.2: Establish a single image evaluation model based on average gradient. The average gradient is used to represent the rate of change of detail information in the image, further reflecting the texture change and image clarity. As the average gradient of the image increases, the more level changes the image contains, and the higher the image clarity. The expression of the average gradient is:

[0036]

[0037] Where W, Q represents the size of the image, ΔI x ,ΔI y Represent the gradients in the x and y directions respectively;

[0038] Step S3.3: Establish an image joint evaluation model based on mutual information. Mutual information is a physical quantity used to evaluate the similarity between the fused image and the source image. The greater the mutual information, the greater the similarity of the grayscale distribution and the higher the fusion quality. The expression of mutual information is:

[0039]

[0040] Where P represents the grayscale probability density function, I fa , I fb 、 Represents the fused image and the source images a, b and the total mutual information MI of the three images; P f 、P a are the grayscale probability density functions of the fused image and the source image a respectively; P fa is the joint probability density of grayscale between the fused image and the source image a; P b is the grayscale probability density function of the source image b; P fb is the joint probability density of grayscale between the fused image and the source image b; δ represents the adjustment parameter;

[0041] Step S3.4: Establish an image evaluation model based on structural similarity. Structural similarity is used to judge the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The expression of structural similarity is:

[0042]

[0043] Where μ is the mean, σ is the variance, and C is a constant; X and Y represent the two images whose structural similarity is to be calculated; μ x 、μ y are the means of X and Y respectively; C1 and C2 are constants respectively; σ xy Represents the covariance of grayscale between X and Y; σ x , σ y Represents the variance of X and Y grayscale respectively;

[0044] As the image position changes, the brightness and contrast change continuously. In practical applications, it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows:

[0045]

[0046] Where M represents the number of local windows in the image.

[0047] The optical and SAR remote sensing image pixel-level fusion system provided by the present invention comprises:

[0048] Module M1: Through the data preprocessing process, the optical and SAR images are unified into the same geographic coordinate frame and the influence of sensor-specific factors is eliminated to obtain a fusible image group and fuse them;

[0049] Module M2: Based on the fusion criteria that the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image, the two types of image information are obtained to the greatest extent;

[0050] Module M3: Information entropy, average gradient, mutual information and structural similarity are used as objective evaluation indicators to evaluate the fused image. By traversing the combination of the band channel of the optical image and the polarization channel of the SAR image, the optimal fusion result is selected, and the best image channel combination is determined based on the evaluation results.

[0051] Preferably, the data preprocessing process in the module M1 includes:

[0052] Selection of overlapping observation areas of optical and SAR images: Obtain overlapping observation areas of optical and SAR images by selecting geographic coordinates;

[0053] SAR image radiation correction: converting the backscatter intensity of the original SAR image into a comparable intensity;

[0054] SAR image speckle filtering: Through filtering, the intensity of speckles in SAR images is reduced, providing a basis for image fusion;

[0055] SAR image amplitude to decibel image conversion: Assume that the polarization channel of the SAR image is The expression for converting the amplitude graph into a decibel graph is:

[0056] σ HH =10log|S HH | 2 ,σ HV =10log|S HV | 2 ,σ HH =10log|S HH |2 ,σ VV =10log|S VV | 2

[0057] Among them, R HH 、R HV 、R VH 、R VV Respectively represent the amplitude value of the horizontal HH polarization channel, the amplitude value of the cross HV polarization channel, the amplitude value of the cross VH polarization channel, and the amplitude value of the vertical VV polarization channel; σ HH , σ HV Represents the decibel diagram of horizontal HH polarization and the decibel diagram of cross HV polarization respectively; S HH 、S HV 、S VV Respectively represent the amplitude diagram of horizontal HH polarization, the amplitude diagram of cross HV polarization, and the amplitude diagram of vertical VV polarization;

[0058] SAR image terrain correction: First, combine the digital elevation model to select high-precision control points; then perform sensitivity analysis on the errors that affect the geometric positioning accuracy, including orbit determination error, slant range error, azimuth time error, and atmospheric delay error; establish the range-Doppler positioning equation to accurately estimate the geometric positioning error; after compensating for the geometric parameter errors, use geocoding to convert the image coordinate space into a geographic coordinate system.

[0059] Preferably, the module M2 includes:

[0060] Assume that the optical image O k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image;

[0061] Let the fused image be F k,l , then the construction of the fused image is expressed as:

[0062] F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n)

[0063] Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of optical and SAR images respectively.

[0064] Preferably, to ensure that the local variance in the fused image is equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are retained in the fused image, the sliding window is processed one by one, and the image row number m and column number n are omitted. The expression is:

[0065]

[0066] Among them, Cov(F k,l ,F k,l ) is expressed as F k,l variance; and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window;

[0067] To ensure that the local mean in the fused image is equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image, the expression is:

[0068] Mean(F k,l )=ω O μ O +ω R μ R =μ R

[0069] Among them, Mean(F k,l ) is expressed as F k,l The mean of μ O and μ R are the mean values of the optical image and SAR image pixels within the window respectively;

[0070] Solve the above two equations together to obtain two fusion coefficients ω O and ω R , the expression is:

[0071]

[0072] If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then the choice is to retain ω O Greater than ω R Solution; if in the fusion process, in order to remove the occlusion in the optical image, then choose to retain ω O Less than ω R The solution.

[0073] Preferably, the module M3 includes: establishing fusion image evaluation criteria and optimization model, and performing k,l,k=1,2,...,l=1,2,...,where k and l represent the band channel of the optical image and the polarization channel of the SAR image respectively. A variety of objective evaluation indicators are used to comprehensively evaluate the target image. The selected objective evaluation indicators are divided into single image evaluation indicators and joint image evaluation indicators. By traversing k=1,2,...,l=1,2,..., the band channel of the optical image and the polarization channel of the SAR image are fused in pairs, and the optimal fusion result is selected;

[0074] Module M3.1: Establish a single image evaluation model based on information entropy. The size of information entropy reflects the amount of information carried by the image. The larger the information entropy of the image, the richer its information content and the better its quality. The calculation formula of information entropy is as follows:

[0075]

[0076] In the formula, H represents the information entropy of the image, P ij Represents the frequency of gray level F(i,j) in the image; i and j represent the row and column of image pixel values respectively;

[0077] Module M3.2: Establish a single image evaluation model based on average gradient. The average gradient is used to represent the rate of change of detail information in the image, further reflecting the texture changes and image clarity. As the average gradient of the image increases, the more level changes the image contains, the higher the image clarity. The expression of the average gradient is:

[0078]

[0079] Where W, Q represents the size of the image, ΔI x ,ΔI y Represent the gradients in the x and y directions respectively;

[0080] Module M3.3: Establish an image joint evaluation model based on mutual information. Mutual information is a physical quantity used to evaluate the similarity between the fused image and the source image. The greater the mutual information, the greater the similarity of the grayscale distribution and the higher the fusion quality. The expression of mutual information is:

[0081]

[0082] Where P represents the grayscale probability density function, I fa , I fb 、 Represents the fused image and the source images a, b and the total mutual information MI of the three images; P f 、P a are the grayscale probability density functions of the fused image and the source image a respectively; P fais the joint probability density of grayscale between the fused image and the source image a; P b is the grayscale probability density function of the source image b; P fb is the joint probability density of grayscale between the fused image and the source image b; δ represents the adjustment parameter;

[0083] Module M3.4: Establish an image evaluation model based on structural similarity. Structural similarity is used to judge the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The expression of structural similarity is:

[0084]

[0085] Where μ is the mean, σ is the variance, and C is a constant; X and Y represent the two images whose structural similarity is to be calculated; μ x 、μ y are the means of X and Y respectively; C1 and C2 are constants respectively; σ xy Represents the covariance of grayscale between X and Y; σ x , σ y Represents the variance of X and Y grayscale respectively;

[0086] As the image position changes, the brightness and contrast change continuously. In practical applications, it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows:

[0087]

[0088] Where M represents the number of local windows in the image.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] 1. This invention establishes a new method for pixel-level fusion of optical and SAR remote sensing images, which reduces the complexity of traditional optical and SAR image fusion, eliminates the influence of sensor-specific factors, and achieves the best fusion results.

[0091] 2. The present invention provides a pixel-level image fusion method that is computationally efficient, does not require training samples, and simultaneously preserves the complementary information of optical and SAR images to the greatest extent possible, thereby maximizing the information obtained from the two types of images.

[0092] 3. The method of the present invention has strong practical operability and versatility, can simplify the optical and SAR image fusion process, obtain better fusion results, and provide a technical reference for subsequent target detection capability improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0094] Figure 1 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0095] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0096] Example 1

[0097] See also Figure 1 , Figure 1 This is a flowchart of a method for designing a long-time series observation mode for a multi-satellite network of small satellites provided in this embodiment. The method steps are as follows:

[0098] Step S1: Establish a data preprocessing model to unify the optical and SAR images into the same geographic coordinate framework and eliminate the influence of sensor-specific factors to obtain a fusionable image group;

[0099] Step S2: Propose a fusion criterion in which the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image, so as to obtain the two types of image information to the greatest extent;

[0100] Step S3: Select image evaluation criteria to evaluate the fused image information, and also analyze how to select image channels to obtain the best fusion results.

[0101] Furthermore, in the step S1, an optical image O is provided. k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image. Due to the different principles and image characteristics of optical and SAR remote sensing, a data preprocessing model is first established, including: selection of overlapping observation areas for optical and SAR images; SAR image radiometric correction; SAR image speckle filtering; SAR image amplitude-to-decibel (power) conversion; and SAR image terrain correction.

[0102] Furthermore, in step S1:

[0103] 1) Selection of overlapping observation areas for optical and SAR images: Obtain overlapping observation areas for optical and SAR images by selecting geographic coordinates;

[0104] 2) SAR image radiometric correction: Considering the differences in incident angle and other sensor parameters, the backscatter intensity of the original SAR image is converted to a comparable intensity;

[0105] 3) SAR image speckle filtering: Through filtering, the intensity of speckle in SAR images is reduced, providing a basis for image fusion;

[0106] 4) SAR image amplitude to decibel (power) map conversion: Assume that the polarization channel of the SAR image is The process of converting the amplitude graph into a decibel graph can be mathematically expressed as:

[0107] σ HH =10log|S HH | 2 ,σ HV =10log|S HV | 2 ,σ HH =10log|S VV | 2

[0108] 5) SAR image terrain correction: Since SAR images are acquired using side-view imaging, there are effects such as "slope shortening", so correction is required. The correction process requires digital elevation information of the observation area, which can be obtained on the Internet.

[0109] Furthermore, in step S2, the fused image is assumed to be F k,l From a mathematical point of view, the pixel-level fusion of the present invention, whether it is the traditional method based on color theory, transformation theory, or the current method based on deep network, each pixel in the fused image is a linear combination of the corresponding pixels of the image before fusion, so the construction of the fused image can be expressed as a linear combination of the input images.

[0110] F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n)(1)

[0111] Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of the optical and SAR images, respectively. The coefficients control the contribution of the optical and SAR images. The fusion coefficients are determined using the input optical and SAR images, and the fused image can be obtained by equation (1).

[0112] Furthermore, in step S2, setting rules or standards to determine the fusion coefficients is crucial for image fusion, as the selected standards determine the nature of the fusion results. Considering that optical images are rich in detail, while SAR images are susceptible to speckle noise, image fusion aims to preserve both the high spatial information or details in optical images and the radiometric information or color in SAR images, establishing these two standards.

[0113] Furthermore, in step S2, the local variance in the standard 1 fused image should be equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are preserved in the fused image. According to formula (1), this expression can be mathematically expressed as:

[0114]

[0115] The above formula is processed by sliding windows one by one. Without affecting the understanding, the image row number, column number m and n are omitted. and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window.

[0116] Furthermore, in step S2, the local mean in the standard 2 fused image should be equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image. According to formula (1), this expression can be expressed as:

[0117] Mean(F k,l )=ω O μ O +ω R μ R =μ R (3)

[0118] where μ O and μ R are the mean values of the optical image and SAR image pixels within the window, respectively.

[0119] These two criteria will produce two equations, which can determine the two fusion coefficients ω O and ω R , that is, the equation:

[0120]

[0121] Formula (4) is about ω R The quadratic equation of one variable, solve the last two solutions, and then get two ω from formula (3) O If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then we choose to keep ω OGreater than ω R If we want to remove the cloud and fog occlusion in the optical image during the fusion process, we can choose to keep ω O Less than ω R The solution.

[0122] Furthermore, in step S3, the fusion image evaluation criteria and optimization model are established to evaluate the fusion image F. k,l , k = 1, 2, ..., l = 1, 2, ..., where k and l represent the optical image band channel and the SAR image polarization channel, respectively. A comprehensive evaluation of the target image is performed using multiple objective evaluation metrics, which can be categorized as single image evaluation metrics and joint image evaluation metrics. By traversing k = 1, 2, ..., l = 1, 2, ..., the optical image band channel and the SAR image polarization channel are fused pairwise, and the optimal fusion result is selected.

[0123] Step S3.1: A single image evaluation model based on information entropy is recommended. The size of information entropy reflects the amount of information carried by the image. Generally speaking, the greater the information entropy of an image, the richer its information content and the better its quality. The calculation of information entropy is shown in the following formula:

[0124]

[0125] In the formula, H represents the information entropy of the image, P ij Indicates the frequency of gray level F(i,j) appearing in the image.

[0126] Step S3.2: A single image evaluation model based on average gradient is proposed. The average gradient is used to represent the rate of change of detail information in the image, and further reflects the texture changes and image clarity. As the average gradient of the image increases, the more level changes the image contains, and the higher the image clarity. The average gradient can be expressed as the following formula:

[0127]

[0128] Where M, N represent the size of the image, ΔI x ,ΔI y represent the gradients in the x and y directions respectively.

[0129] Step S3.3: Establish an image joint evaluation model based on mutual information. Mutual information (MI) is a physical quantity used to evaluate the similarity between the fused image and the source image. The larger the MI, the more source image information is contained in the fused image. The greater the similarity of grayscale distribution, the higher the fusion quality. The calculation of MI can be expressed as:

[0130]

[0131] Where P represents the joint probability density function, I fa , I fb 、 They represent the fused image and the source images a, b, and the total MI of the three images, and δ is 0.5.

[0132] Step S3.4: Establish an image evaluation model based on structural similarity. Structural Similarity (SSIM) measures the similarity of structural features between two images. Remote sensing images have large imaging ranges and contain many structural features such as streets, lakes, and mountains. SSIM is a good indicator of the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The formula for calculating structural similarity is shown below.

[0133]

[0134] In the formula, μ is the mean, σ is the variance, and C is a constant that increases the generalization ability of the formula. Because the brightness and contrast change with the position of the image, in practical applications it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows:

[0135]

[0136] Where M represents the number of local windows in the image.

[0137] Example 2

[0138] The present invention also provides a pixel-level fusion system for optical and SAR remote sensing images. The pixel-level fusion system for optical and SAR remote sensing images can be implemented by executing the process steps of the pixel-level fusion method for optical and SAR remote sensing images. That is, those skilled in the art can understand the pixel-level fusion method for optical and SAR remote sensing images as a preferred implementation of the pixel-level fusion system for optical and SAR remote sensing images.

[0139] The optical and SAR remote sensing image pixel-level fusion system provided by the present invention includes: a module M1: unifying the optical and SAR images into the same geographic coordinate framework through a data preprocessing process, eliminating the influence of sensor-specific factors, obtaining a fusible image group, and fusing them; a module M2: maximally acquiring the two types of image information based on the fusion criterion that the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image; and a module M3: using information entropy, average gradient, mutual information, and structural similarity as objective evaluation indicators to evaluate the fused image, selecting the optimal fusion result by traversing the band channel combination of the optical image and the polarization channel combination of the SAR image, and determining the optimal image channel combination based on the evaluation result.

[0140] The data preprocessing process in the module M1 includes:

[0141] Selection of overlapping observation areas of optical and SAR images: Obtain overlapping observation areas of optical and SAR images by selecting geographic coordinates;

[0142] SAR image radiation correction: converting the backscatter intensity of the original SAR image into a comparable intensity;

[0143] SAR image speckle filtering: Through filtering, the intensity of speckles in SAR images is reduced, providing a basis for image fusion;

[0144] SAR image amplitude to decibel image conversion: Assume that the polarization channel of the SAR image is The expression for converting the amplitude graph into a decibel graph is:

[0145] σ HH =10log|S HH | 2 ,σ HV =10log|S HV | 2 ,σ HH =10log|S HH | 2 ,σ VV =10log|S VV | 2

[0146] Among them, R HH 、R HV 、R VH 、R VV Respectively represent the amplitude value of the horizontal HH polarization channel, the amplitude value of the cross HV polarization channel, the amplitude value of the cross VH polarization channel, and the amplitude value of the vertical VV polarization channel; σ HH , σ HV Represents the decibel diagram of horizontal HH polarization and the decibel diagram of cross HV polarization respectively; SHH 、S HV 、S VV Respectively represent the amplitude diagram of horizontal HH polarization, the amplitude diagram of cross HV polarization, and the amplitude diagram of vertical VV polarization;

[0147] SAR image terrain correction: First, combine the digital elevation model to select high-precision control points; then perform sensitivity analysis on the errors that affect the geometric positioning accuracy, including orbit determination error, slant range error, azimuth time error, and atmospheric delay error; establish the range-Doppler positioning equation to accurately estimate the geometric positioning error; after compensating for the geometric parameter errors, use geocoding to convert the image coordinate space into a geographic coordinate system.

[0148] The module M2 includes:

[0149] Assume that the optical image O k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image;

[0150] Let the fused image be F k,l , then the construction of the fused image is expressed as:

[0151] F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n)

[0152] Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of optical and SAR images respectively.

[0153] To ensure that the local variance in the fused image is equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are retained in the fused image, the sliding windows are processed one by one, omitting the image row number m and column number n. The expression is:

[0154]

[0155] Among them, Cov(F k,l ,F k,l ) is expressed as F k,l variance; and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window;

[0156] To ensure that the local mean in the fused image is equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image, the expression is:

[0157] Mean(F k,l )=ω O μ O +ω R μ R =μ R

[0158] Among them, Mean(F k,l ) is expressed as F k,l The mean of μ O and μ R are the mean values of the optical image and SAR image pixels within the window respectively;

[0159] Solve the above two equations together to obtain two fusion coefficients ω O and ω R , the expression is:

[0160]

[0161] If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then the choice is to retain ω O Greater than ω R Solution; if in the fusion process, in order to remove the occlusion in the optical image, then choose to retain ω O Less than ω R The solution.

[0162] The module M3 includes: establishing fusion image evaluation criteria and optimization model, and performing fusion image F k,l ,k=1,2,...,l=1,2,...,where k and l represent the band channel of the optical image and the polarization channel of the SAR image respectively. A variety of objective evaluation indicators are used to comprehensively evaluate the target image. The selected objective evaluation indicators are divided into single image evaluation indicators and joint image evaluation indicators. By traversing k=1,2,...,l=1,2,..., the band channel of the optical image and the polarization channel of the SAR image are fused in pairs, and the optimal fusion result is selected;

[0163] Module M3.1: Establish a single image evaluation model based on information entropy. The size of information entropy reflects the amount of information carried by the image. The larger the information entropy of the image, the richer its information content and the better its quality. The calculation formula of information entropy is as follows:

[0164]

[0165] In the formula, H represents the information entropy of the image, Pij Represents the frequency of gray level F(i,j) in the image; i and j represent the row and column of image pixel values respectively;

[0166] Module M3.2: Establish a single image evaluation model based on average gradient. The average gradient is used to represent the rate of change of detail information in the image, further reflecting the texture change and image clarity. As the average gradient of the image increases, the more level changes the image contains, the higher the image clarity. The expression of the average gradient is:

[0167]

[0168] Where W, Q represents the size of the image, ΔI x ,ΔI y Represent the gradients in the x and y directions respectively;

[0169] Module M3.3: Establish an image joint evaluation model based on mutual information. Mutual information is a physical quantity used to evaluate the similarity between the fused image and the source image. The greater the mutual information, the greater the similarity of the grayscale distribution and the higher the fusion quality. The expression of mutual information is:

[0170]

[0171] Where P represents the grayscale probability density function, I fa , I fb 、 Represents the fused image and the source images a, b and the total mutual information MI of the three images; P f 、P a are the grayscale probability density functions of the fused image and the source image a respectively; P fa is the joint probability density of grayscale between the fused image and the source image a; P b is the grayscale probability density function of the source image b; P fb is the joint probability density of grayscale between the fused image and the source image b; δ represents the adjustment parameter;

[0172] Module M3.4: Establish an image evaluation model based on structural similarity. Structural similarity is used to judge the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The expression of structural similarity is:

[0173]

[0174] Where μ is the mean, σ is the variance, and C is a constant; X and Y represent the two images whose structural similarity is to be calculated; μ x 、μ y are the means of X and Y respectively; C1 and C2 are constants respectively; σxy Represents the covariance of grayscale between X and Y; σ x , σ y Represents the variance of X and Y grayscale respectively;

[0175] As the image position changes, the brightness and contrast change continuously. In practical applications, it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows:

[0176]

[0177] Where M represents the number of local windows in the image.

[0178] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0179] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A pixel-level fusion method for optical and SAR remote sensing images, characterized in that: include: Step S1: Through the data preprocessing process, the optical and SAR images are unified into the same geographic coordinate frame and the influence of sensor-specific factors is eliminated to obtain a fusible image group and fuse them; Step S2: According to the fusion criteria that the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image, the two types of image information are obtained to the greatest extent; Step S3: Use information entropy, average gradient, mutual information and structural similarity as objective evaluation indicators to evaluate the fused image. By traversing the band channel combination of the optical image and the polarization channel combination of the SAR image, the optimal fusion result is selected, and the optimal image channel combination is determined based on the evaluation results.

2. The method for pixel-level fusion of optical and SAR remote sensing images according to claim 1, characterized in that: The data preprocessing process in step S1 includes: Selection of overlapping observation areas of optical and SAR images: Obtain overlapping observation areas of optical and SAR images by selecting geographic coordinates; SAR image radiation correction: converting the backscatter intensity of the original SAR image into a comparable intensity; SAR image speckle filtering: Through filtering, the intensity of speckles in SAR images is reduced, providing a basis for image fusion; SAR image amplitude to decibel image conversion: Assume that the polarization channel of the SAR image is The expression for converting the amplitude graph into a decibel graph is: s HH =10log|S HH | 2 ,s HV =10log|S HV | 2 ,s HH =10log|S HH | 2 ,s VV =10log|S VV | 2 Among them, R HH 、R HV 、R VH 、R VV Respectively represent the amplitude value of the horizontal HH polarization channel, the amplitude value of the cross HV polarization channel, the amplitude value of the cross VH polarization channel, and the amplitude value of the vertical VV polarization channel; σ HH , σ HV Represents the decibel diagram of horizontal HH polarization and the decibel diagram of cross HV polarization respectively; S HH 、S HV 、S VV Respectively represent the amplitude diagram of horizontal HH polarization, the amplitude diagram of cross HV polarization, and the amplitude diagram of vertical VV polarization; SAR image terrain correction: First, combine the digital elevation model to select high-precision control points; then perform sensitivity analysis on the errors that affect the geometric positioning accuracy, including orbit determination error, slant range error, azimuth time error, and atmospheric delay error; establish the range-Doppler positioning equation to accurately estimate the geometric positioning error; after compensating for the geometric parameter errors, use geocoding to convert the image coordinate space into a geographic coordinate system.

3. The method for pixel-level fusion of optical and SAR remote sensing images according to claim 2, characterized in that: The step S2 comprises: Assume that the optical image O k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image; Let the fused image be F k,l , then the construction of the fused image is expressed as: F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n) Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of optical and SAR images respectively.

4. The method for pixel-level fusion of optical and SAR remote sensing images according to claim 3, characterized in that: To ensure that the local variance in the fused image is equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are retained in the fused image, the sliding windows are processed one by one, omitting the image row number m and column number n. The expression is: Among them, Cov(F k,l ,F k,l ) is expressed as F k,l variance; and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window; To ensure that the local mean in the fused image is equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image, the expression is: Mean(F k,l )=ω O m O +oh R m R =μ R Among them, Mean(F k,l ) is expressed as F k,l The mean of μ O and μ R are the mean values of the optical image and SAR image pixels within the window respectively; Solve the above two equations together to obtain two fusion coefficients ω O and ω R , the expression is: If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then the choice is to retain ω O Greater than ω R Solution; if in the fusion process, in order to remove the occlusion in the optical image, then choose to retain ω O Less than ω R The solution.

5. The method for pixel-level fusion of optical and SAR remote sensing images according to claim 4, characterized in that: The step S3 includes: establishing fusion image evaluation criteria and optimization model, and performing fusion image F k,l ,k=1,2,...,l=1,2,...,where k and l represent the band channel of the optical image and the polarization channel of the SAR image respectively. A variety of objective evaluation indicators are used to comprehensively evaluate the target image. The selected objective evaluation indicators are divided into single image evaluation indicators and joint image evaluation indicators. By traversing k=1,2,...,l=1,2,..., the band channel of the optical image and the polarization channel of the SAR image are fused in pairs, and the optimal fusion result is selected; Step S3.1: Establish a single image evaluation model based on information entropy. The size of information entropy reflects the amount of information carried by the image. The larger the information entropy of the image, the richer its information content and the better its quality. The information entropy is calculated as follows: In the formula, H represents the information entropy of the image, P ij Represents the frequency of gray level F(i,j) in the image; i and j represent the row and column of image pixel values respectively; Step S3.2: Establish a single image evaluation model based on average gradient. The average gradient is used to represent the rate of change of detail information in the image, further reflecting the texture change and image clarity. As the average gradient of the image increases, the more level changes the image contains, and the higher the image clarity. The expression of the average gradient is: Where W, Q represents the size of the image, ΔI x ,ΔI y Represent the gradients in the x and y directions respectively; Step S3.3: Establish an image joint evaluation model based on mutual information. Mutual information is a physical quantity used to evaluate the similarity between the fused image and the source image. The greater the mutual information, the greater the similarity of the grayscale distribution and the higher the fusion quality. The expression of mutual information is: Where P represents the grayscale probability density function, I fa , I fb 、 Represents the fused image and the source images a, b and the total mutual information MI of the three images; P f 、P a are the grayscale probability density functions of the fused image and the source image a respectively; P fa is the joint probability density of grayscale between the fused image and the source image a; P b is the grayscale probability density function of the source image b; P fb is the joint probability density of grayscale between the fused image and the source image b; δ represents the adjustment parameter; Step S3.4: Establish an image evaluation model based on structural similarity. Structural similarity is used to judge the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The expression of structural similarity is: Where μ is the mean, σ is the variance, and C is a constant; X and Y represent the two images whose structural similarity is to be calculated; μ x 、μ y are the means of X and Y respectively; C1 and C2 are constants respectively; σ xy Represents the covariance of grayscale between X and Y; σ x , σ y Represents the variance of X and Y grayscale respectively; As the image position changes, the brightness and contrast change continuously. In practical applications, it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows: Where M represents the number of local windows in the image.

6. A pixel-level fusion system for optical and SAR remote sensing images, characterized in that: include: Module M1: Through the data preprocessing process, the optical and SAR images are unified into the same geographic coordinate frame and the influence of sensor-specific factors is eliminated to obtain a fusible image group and fuse them; Module M2: Based on the fusion criteria that the variance of the fused image is the same as that of the optical image and the mean is the same as that of the SAR image, the two types of image information are obtained to the greatest extent; Module M3: Information entropy, average gradient, mutual information and structural similarity are used as objective evaluation indicators to evaluate the fused image. By traversing the combination of the band channel of the optical image and the polarization channel of the SAR image, the optimal fusion result is selected, and the best image channel combination is determined based on the evaluation results.

7. The optical and SAR remote sensing image pixel-level fusion system according to claim 6, characterized in that: The data preprocessing process in the module M1 includes: Selection of overlapping observation areas of optical and SAR images: Obtain overlapping observation areas of optical and SAR images by selecting geographic coordinates; SAR image radiation correction: converting the backscatter intensity of the original SAR image into a comparable intensity; SAR image speckle filtering: Through filtering, the intensity of speckles in SAR images is reduced, providing a basis for image fusion; SAR image amplitude to decibel image conversion: Assume that the polarization channel of the SAR image is The expression for converting the amplitude graph into a decibel graph is: s HH =10log|S HH | 2 ,s HV =10log|S HV | 2 ,s HH =10log|S HH | 2 ,s VV =10log|S VV | 2 Among them, R HH 、R HV 、R VH 、R VV Respectively represent the amplitude value of the horizontal HH polarization channel, the amplitude value of the cross HV polarization channel, the amplitude value of the cross VH polarization channel, and the amplitude value of the vertical VV polarization channel; σ HH , σ HV Represents the decibel diagram of horizontal HH polarization and the decibel diagram of cross HV polarization respectively; S HH 、S HV 、S VV Respectively represent the amplitude diagram of horizontal HH polarization, the amplitude diagram of cross HV polarization, and the amplitude diagram of vertical VV polarization; SAR image terrain correction: First, combine the digital elevation model to select high-precision control points; then perform sensitivity analysis on the errors that affect the geometric positioning accuracy, including orbit determination error, slant range error, azimuth time error, and atmospheric delay error; establish the range-Doppler positioning equation to accurately estimate the geometric positioning error; after compensating for the geometric parameter errors, use geocoding to convert the image coordinate space into a geographic coordinate system.

8. The optical and SAR remote sensing image pixel-level fusion system according to claim 7, characterized in that: The module M2 includes: Assume that the optical image O k , where k = 1, 2, ..., represents the band of the optical image; SAR image R l , where l = 1, 2, ..., represents the polarization channel of the SAR image; Let the fused image be F k,l , then the construction of the fused image is expressed as: F k,l (m,n)=ω O (m,n)O k (m,n)+ω R (m,n)R l (m,n) Where m and n are the row and column numbers of the image, ω O (m,n),ω R (m,n) are the fusion coefficients of optical and SAR images respectively.

9. The optical and SAR remote sensing image pixel-level fusion system according to claim 8, characterized in that: To ensure that the local variance in the fused image is equal to the corresponding local variance in the optical image, so that the spatial details described by the variance are retained in the fused image, the sliding windows are processed one by one, omitting the image row number m and column number n. The expression is: Among them, Cov(F k,l ,F k,l ) is expressed as F k,l variance; and are the variances of the optical image and SAR image pixels within the window, σ OR is the covariance of the optical and SAR images within the window; To ensure that the local mean in the fused image is equal to the corresponding local mean in the SAR image, so that the radiation information described by the local mean is retained in the fused image, the expression is: Mean(F k,l )=ω O m O +oh R m R =μ R Among them, Mean(F k,l ) is expressed as F k,l The mean of μ O and μ R are the mean values of the optical image and SAR image pixels within the window respectively; Solve the above two equations together to obtain two fusion coefficients ω O and ω R , the expression is: If the fusion process is expected to transfer the spatial details of the optical image to the fused image, then the choice is to retain ω O Greater than ω R Solution; if in the fusion process, in order to remove the occlusion in the optical image, then choose to retain ω O Less than ω R The solution.

10. The optical and SAR remote sensing image pixel-level fusion system according to claim 9, characterized in that: The module M3 includes: establishing fusion image evaluation criteria and optimization model, and performing fusion image F k,l ,k=1,2,...,l=1,2,...,where k and l represent the band channel of the optical image and the polarization channel of the SAR image respectively. A variety of objective evaluation indicators are used to comprehensively evaluate the target image. The selected objective evaluation indicators are divided into single image evaluation indicators and joint image evaluation indicators. By traversing k=1,2,...,l=1,2,..., the band channel of the optical image and the polarization channel of the SAR image are fused in pairs, and the optimal fusion result is selected; Module M3.1: Establish a single image evaluation model based on information entropy. The size of information entropy reflects the amount of information carried by the image. The larger the information entropy of the image, the richer its information content and the better its quality. The calculation formula of information entropy is as follows: In the formula, H represents the information entropy of the image, P ij Represents the frequency of gray level F(i,j) in the image; i and j represent the row and column of image pixel values respectively; Module M3.2: Establish a single image evaluation model based on average gradient. The average gradient is used to represent the rate of change of detail information in the image, further reflecting the texture changes and image clarity. As the average gradient of the image increases, the more level changes the image contains, the higher the image clarity. The expression of the average gradient is: Where W, Q represents the size of the image, ΔI x ,ΔI y Represent the gradients in the x and y directions respectively; Module M3.3: Establish an image joint evaluation model based on mutual information. Mutual information is a physical quantity used to evaluate the similarity between the fused image and the source image. The greater the mutual information, the greater the similarity of the grayscale distribution and the higher the fusion quality. The expression of mutual information is: Where P represents the grayscale probability density function, I fa , I fb 、 Represents the fused image and the source images a, b and the total mutual information MI of the three images; P f 、P a are the grayscale probability density functions of the fused image and the source image a respectively; P fa is the joint probability density of grayscale between the fused image and the source image a; P b is the grayscale probability density function of the source image b; P fb is the joint probability density of grayscale between the fused image and the source image b; δ represents the adjustment parameter; Module M3.4: Establish an image evaluation model based on structural similarity. Structural similarity is used to judge the structural differences between the fused image and the source image. The greater the structural similarity, the higher the quality of the fused image. The expression of structural similarity is: Where μ is the mean, σ is the variance, and C is a constant; X and Y represent the two images whose structural similarity is to be calculated; μ x 、μ y are the means of X and Y respectively; C1 and C2 are constants respectively; σ xy Represents the covariance of grayscale between X and Y; σ x , σ y Represents the variance of X and Y grayscale respectively; As the image position changes, the brightness and contrast change continuously. In practical applications, it is necessary to divide the image into blocks, calculate the structural similarity of each block separately, and finally take the average value. The calculation formula is as follows: Where M represents the number of local windows in the image.

Images