Multispectral and panchromatic image fusion method based on direction adaptive gain and PCNN

Abstract

The invention discloses a multispectral and panchromatic image fusion method based on direction adaptive gain and PCNN. The method comprises the following steps: performing non-subsampled shearlet transform on a panchromatic image and a multispectral image to obtain respective low-frequency coefficients and high-frequency coefficients; constructing an adaptive direction gain pulse coupling neural network, and fusing the high-frequency coefficient of the panchromatic image and the high-frequency coefficient of the multispectral image through the adaptive direction gain pulse coupling neural network to obtain a fused high-frequency coefficient; fusing the low-frequency coefficient of the panchromatic image and the low-frequency coefficient of the multispectral image according to the comparison result of the energy of the panchromatic image and the energy of the multispectral image and the comparison result of the gradient of the panchromatic image and the gradient of the multispectral image to obtain a fused low-frequency coefficient; and performing non-subsampled shear wave inverse transformation on the fused low-frequency coefficient and the fused high-frequency coefficient to obtain a high-resolution multispectral fusion image. According to the invention, the spatial resolution and spectral fidelity of the fused image are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for fusing multispectral and panchromatic images based on orientation adaptive gain and PCNN. Background Technology

[0002] In the field of satellite remote sensing, panchromatic images (PAN) have no spectral information but high spatial resolution, while multispectral images (MS) have insufficient spatial resolution but possess spectral information. Therefore, it is necessary to combine the advantages of both through fusion technology.

[0003] Among the existing fusion methods for satellite remote sensing images, algorithms combining non-subsampled shear wave transform (NSST) and pulse coupled neural network (PCNN) are widely used. NSST can extract rich image edge and texture information, while PCNN can combine pixel neighborhood information to enhance image edges.

[0004] The algorithm combining NSST and PCNN has the following shortcomings: (1) Traditional PCNN does not consider the directional features of the image when processing high-frequency coefficients, resulting in inaccurate discrimination of high-frequency information that is significant in a single direction in the PAN image; (2) In the fusion of low-frequency coefficients, the weights of energy and gradient are unbalanced, significant edge information is easily masked by energy, and there is a ringing phenomenon caused by gradient diffusion. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, which improves the spatial resolution and spectral fidelity of the fused image.

[0006] The objective of this invention is achieved through the following technical solution: a multispectral and panchromatic image fusion method based on directional adaptive gain and PCNN, comprising: performing a non-subsampled shear wave transform on the panchromatic image to obtain low-frequency and high-frequency coefficients of the panchromatic image; performing a non-subsampled shear wave transform on the multispectral image to obtain low-frequency and high-frequency coefficients of the multispectral image; constructing an adaptive directional gain pulse-coupled neural network; fusing the high-frequency coefficients of the panchromatic image and the high-frequency coefficients of the multispectral image through the adaptive directional gain pulse-coupled neural network to obtain fused high-frequency coefficients; fusing the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image and the gradient comparison results of the panchromatic image and the multispectral image to obtain fused low-frequency coefficients; and performing a non-subsampled shear wave inverse transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fused image.

[0007] In the above-mentioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, the panchromatic image is decomposed into NSST level two decomposition for each channel to obtain the low-frequency coefficients and high-frequency coefficients at 16 angles; the multispectral image is decomposed into NSST level two decomposition for each channel to obtain the low-frequency coefficients and high-frequency coefficients at 16 angles.

[0008] In the above-mentioned method for multispectral and panchromatic image fusion based on adaptive direction gain and PCNN, the construction of the adaptive direction gain pulse-coupled neural network includes: extracting edge gradient information in four directions (horizontal, vertical, 45 degrees, and 270 degrees) from the panchromatic image; based on the edge gradient information, dividing the four directions of each pixel into salient directions, adjacent directions, and insignificant directions, and generating gain maps of the corresponding angles; injecting the gain maps into the iterative process of the PCNN network, determining the saliency of high-frequency coefficients in the panchromatic image by the cumulative activation count, constructing a high-frequency coefficient fusion feature map, that is, selecting coefficients with high activation counts as the fusion result.

[0009] In the above-mentioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, edge gradient information at 16 angles is calculated according to the division of the highest frequency direction of the non-subsampled shear wave transform, and these 16 angle edge gradient information are classified into four directions: horizontal, vertical, 45 degrees and 270 degrees; among them, four angle edge gradient information values ​​are assigned to each direction.

[0010] In the aforementioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, the gain map generation rule is as follows: the gain of the four angles corresponding to the significant direction is set to 2, the gain of the eight angles corresponding to the adjacent direction is set to 1, and the gain of the four angles corresponding to the insignificant direction is set to 0.5; a gain map is generated for each image point (i,j). Where i is the x-coordinate of an image point and j is the y-coordinate of an image point. The image is a coefficient diagram of 16 angles obtained from the non-subsampled shear wave transform. Number each coefficient graph. =1、...、16.

[0011] In the above-mentioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, the low-frequency coefficients of the panchromatic image and the multispectral image are fused according to the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, and the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image to obtain the fused low-frequency coefficients. This includes: generating a low-frequency coefficient segmentation map according to the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, and the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image; and using an erosion function to optimize the edges of the generated low-frequency coefficient segmentation map to obtain the fused low-frequency coefficients.

[0012] In the aforementioned method for fusing multispectral and panchromatic images based on orientation adaptive gain and PCNN, the generation of a low-frequency coefficient segmentation map based on the comparison results of the energy of the panchromatic image and the energy of the multispectral image, and the comparison results of the gradient of the panchromatic image and the gradient of the multispectral image, includes the following steps: If the energy of the panchromatic image > the energy of the multispectral image, and the gradient of the panchromatic image > the gradient of the multispectral image, then it is a unified case, and the product of energy and gradient is used as the criterion, and the low-frequency coefficient segmentation map is obtained according to the criterion; If the energy of the panchromatic image < the energy of the multispectral image, and the gradient of the panchromatic image < the gradient of the multispectral image, then it is a unified case, and the product of energy and gradient is used as the criterion, and the low-frequency coefficient segmentation map is obtained according to the criterion; If the energy of the panchromatic image > the energy of the multispectral image, and the gradient of the panchromatic image < the gradient of the multispectral image, then it is a contradictory case, and the first given formula is used as the criterion, and the low-frequency coefficient segmentation map is obtained according to the criterion; If the energy of the panchromatic image < the energy of the multispectral image, and the gradient of the panchromatic image > the gradient of the multispectral image, then it is a contradictory case, and the second given formula is used as the criterion.

[0013] In the above-mentioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, the first given formula is obtained through the following formula: ; in, The low-frequency coefficients of the fused image, These are the low-frequency coefficients of the source image A. Let A be the image energy at point (i,j). Let B be the image energy at point (i,j). Let A be the image gradient at point (i,j). Let B be the image gradient at point (i,j). The x-axis is... The vertical axis is denoted by .

[0014] In the above-mentioned multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN, the second given formula is obtained through the following formula: ; in, The low-frequency coefficients of the fused image, These are the low-frequency coefficients of the source image A. The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. The number of pixels whose low-frequency coefficients are determined from the 8 neighboring pixels of pixel (i,j) as low-frequency coefficients in the PAN image. The number of pixels whose low-frequency coefficients are determined as low-frequency coefficients in the MS image among the 8 neighboring pixels of pixel (i,j).

[0015] A multispectral and panchromatic image fusion system based on directional adaptive gain and PCNN includes: a first module for performing non-subsampled shear wave transform on the panchromatic image to obtain low-frequency and high-frequency coefficients of the panchromatic image, and performing non-subsampled shear wave transform on the multispectral image to obtain low-frequency and high-frequency coefficients of the multispectral image; a second module for constructing an adaptive directional gain pulse-coupled neural network, and fusing the high-frequency coefficients of the panchromatic image and the high-frequency coefficients of the multispectral image through the adaptive directional gain pulse-coupled neural network to obtain fused high-frequency coefficients; a third module for fusing the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, and the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image to obtain fused low-frequency coefficients; and a fourth module for performing non-subsampled shear wave inverse transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fused image.

[0016] Compared with the prior art, the present invention has the following advantages: (1) The present invention introduces directional gain in high frequency coefficient fusion, which improves the accuracy of preserving directional details of PAN images; (2) In the low-frequency coefficient classification and edge optimization, this invention balances the influence of energy and gradient, and reduces ringing phenomenon; (3) The present invention optimizes spatial resolution and spectral resolution while fusing images, and is applicable to scenarios such as remote sensing image interpretation and target detection. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of the multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the ADG-PCNN network structure provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] Figure 1 This is a flowchart of a multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN provided in an embodiment of the present invention. Figure 1 As shown, this multispectral and panchromatic image fusion method based on orientation adaptive gain and PCNN includes: The low-frequency and high-frequency coefficients of the panchromatic image are obtained by performing a non-subsampled shear wave transform on the panchromatic image, and the low-frequency and high-frequency coefficients of the multispectral image are obtained by performing a non-subsampled shear wave transform on the multispectral image. An adaptive directional gain pulse coupling neural network is constructed, and the high-frequency coefficients of the panchromatic image and the multispectral image are fused to obtain the fused high-frequency coefficients. The low-frequency coefficients of the panchromatic image and the multispectral image are fused based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image and the gradient comparison results of the panchromatic image and the multispectral image to obtain the fused low-frequency coefficients. A non-subsampled shear wave inverse transform is performed on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fused image.

[0020] The panchromatic image is decomposed into low-frequency coefficients and high-frequency coefficients at 16 angles by performing NSST secondary decomposition on each channel; the multispectral image is also decomposed into low-frequency coefficients and high-frequency coefficients at 16 angles by performing NSST secondary decomposition on each channel.

[0021] The construction of an adaptive directional gain pulse-coupled neural network includes: extracting edge gradient information in four directions (horizontal, vertical, 45 degrees, and 270 degrees) from the panchromatic image; based on the edge gradient information, dividing the four directions of each pixel into salient, adjacent, and insignificant directions, and generating gain maps for the corresponding angles; injecting the gain maps into the iterative process of the PCNN network, determining the saliency of high-frequency coefficients in the panchromatic image by the cumulative activation count, and constructing a high-frequency coefficient fusion feature map, i.e., selecting coefficients with high activation counts as the fusion result.

[0022] Based on the division of the highest frequency direction of the non-subsampled shear wave transform, edge gradient information at 16 angles is calculated, and these 16 angle edge gradient information are classified into four directions: horizontal, vertical, 45 degrees and 270 degrees; among them, four angle edge gradient information values ​​are assigned to each direction.

[0023] The gain map generation rules are as follows: the gain of the four angles corresponding to the significant direction is set to 2, the gain of the eight angles corresponding to the adjacent direction is set to 1, and the gain of the four angles corresponding to the insignificant direction is set to 0.5; a gain map is generated for each image point (i,j). Where i is the x-coordinate of an image point and j is the y-coordinate of an image point. The image is a coefficient diagram of 16 angles obtained from the non-subsampled shear wave transform. Number each coefficient graph. =1、...、16.

[0024] The low-frequency coefficients of the panchromatic image and the multispectral image are fused based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, as well as the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image. The fused low-frequency coefficients are obtained by: generating a low-frequency coefficient segmentation map based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, as well as the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image; and optimizing the edges of the generated low-frequency coefficient segmentation map using an erosion function to obtain the fused low-frequency coefficients.

[0025] The generation of a low-frequency coefficient segmentation map based on the energy comparison results of the panchromatic image and the multispectral image, and the gradient comparison results of the panchromatic image and the multispectral image, includes the following steps: If the panchromatic image energy > the multispectral image energy, and the panchromatic image gradient > the multispectral image gradient, this is a unified case, and the product of energy and gradient is used as the criterion to obtain the low-frequency coefficient segmentation map; if the panchromatic image energy < the multispectral image energy, and the panchromatic image gradient < the multispectral image gradient, this is a unified case, and the product of energy and gradient is used as the criterion to obtain the low-frequency coefficient segmentation map; if the panchromatic image energy > the multispectral image energy, and the panchromatic image gradient < the multispectral image gradient, this is a contradictory case, and a first given formula is used as the criterion to obtain the low-frequency coefficient segmentation map; if the panchromatic image energy < the multispectral image energy, and the panchromatic image gradient > the multispectral image gradient, this is a contradictory case, and a second given formula is used as the criterion.

[0026] The first given formula is obtained through the following formula: ; in, The low-frequency coefficients of the fused image, These are the low-frequency coefficients of the source image A. Let A be the image energy at point (i,j). Let B be the image energy at point (i,j). Let A be the image gradient at point (i,j). Let B be the image gradient at point (i,j). The x-axis is... The vertical axis is denoted by .

[0027] The second given formula is obtained through the following formula: ; in, The low-frequency coefficients of the fused image, These are the low-frequency coefficients of the source image A. The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. The number of pixels whose low-frequency coefficients are determined from the 8 neighboring pixels of pixel (i,j) as low-frequency coefficients in the PAN image. The number of pixels whose low-frequency coefficients are determined as low-frequency coefficients in the MS image among the 8 neighboring pixels of pixel (i,j).

[0028] Specifically, the steps of this method are as follows: Step 1: Perform non-subsampled shear wave transform (NSST) on the panchromatic image (PAN) and the multispectral image (MS) respectively to obtain their respective low-frequency coefficients and high-frequency coefficients; In step 1, NSST secondary decomposition is performed on each channel of the PAN and MS images to obtain low-frequency coefficients and high-frequency coefficients at 16 angles.

[0029] Step 2: Fuse the high-frequency coefficients of panchromatic and multispectral images by constructing an adaptive directional gain pulse coupled neural network (ADG-PCNN).

[0030] In step 2, ADG-PCNN iteratively activates the high-frequency coefficients of the panchromatic image and the multispectral image respectively. After comparing the activation results, a high-frequency coefficient fusion feature map is constructed.

[0031] In step 2, ADG-PCNN weights the network activation values ​​of the panchromatic image and multispectral image using gain maps from 16 angles, generates activation results for all angles, and then compares them one by one.

[0032] The specific steps for constructing an adaptive directional gain pulse-coupled neural network (ADG-PCNN) include: Step 2.1: Extract edge gradient information in four directions—horizontal, vertical, 45-degree, and 270-degree—from the PAN image; In step 2.1, the edge gradient information in the four directions is obtained by calculating and merging the high-frequency coefficients of 16 angles using a certain formula.

[0033] Step 2.2: Based on edge gradient information, divide the four directions of each pixel into significant directions, adjacent directions and insignificant directions, and generate gain maps for the corresponding angles.

[0034] In step 2.2, a gain map is generated for the high-frequency coefficients at each angle, resulting in a total of 16 gain maps, which are used for weighting the activation values ​​of the ADG-PCNN network.

[0035] In step 2.2, the gain map generation rule is as follows: the gain of the four NSST angles corresponding to the significant direction is set to 2, the gain of the eight NSST angles corresponding to the adjacent direction is set to 1, and the gain of the four NSST angles corresponding to the insignificant direction is set to 0.5. A gain map is generated for each image point (i,j). .

[0036] Step 2.3: Inject the gain map into the iterative process of the PCNN network, determine the significance of the high-frequency coefficients of PAN by accumulating the number of activations, and construct the high-frequency coefficient fusion feature map, that is, select the coefficients with high activation counts as the fusion result.

[0037] In step 2.3, for each angle, the cumulative number of activations will be determined, and the coefficient with the highest number of activations will be selected as the fusion result for each angle.

[0038] In step 2.3, the proposed ADG-PCNN is used to inject PAN image orientation feature maps. The pulse-coupled neural network (PCNN) uses external activation values ​​S for the network. ij Directional gain is applied to optimize the output of the PCNN. High-frequency coefficient feature maps are then generated based on the PCNN output.

[0039] Step 3: Based on the relationship between energy and gradient, design fusion rules for low-frequency coefficients.

[0040] In step 3, the results of local energy comparison and local gradient comparison of panchromatic and multispectral images are combined to classify and distinguish high-energy low-gradient regions and high-gradient low-energy regions that are easily confused and misjudged.

[0041] In step 3, the discrimination results after combining energy and gradient are divided into 4 categories, and a low-frequency coefficient fusion rule is designed for each category.

[0042] Step 3.1: When the energy and gradient comparison results are consistent, the product of energy and gradient is used as the judgment criterion. When the energy and gradient comparison results are contradictory, the judgment results of neighboring pixels are combined to make a decision. In Step 3.1, the relationship between energy and gradient is divided into 4 possibilities, and a low-frequency coefficient segmentation map generation scheme is generated for these 4 possibilities.

[0043] In step 3.1, if the energy of the panchromatic image is greater than the energy of the multispectral image, and the gradient of the panchromatic image is greater than the gradient of the multispectral image, then it is a unified case, called case 1, and the product of energy and gradient is used as the criterion for judgment. In step 3.1, if the energy of the panchromatic image is less than the energy of the multispectral image, and the gradient of the panchromatic image is less than the gradient of the multispectral image, then it is a unified case, called case 2, and the product of energy and gradient is used as the criterion for judgment. In step 3.1, if the energy of the panchromatic image is greater than the energy of the multispectral image, and the gradient of the panchromatic image is less than the gradient of the multispectral image, then it is a contradictory case, called case 3, and the given formula is used as the basis for judgment. In step 3.1, if the panchromatic image energy is less than the multispectral image energy and the panchromatic image gradient is greater than the multispectral image gradient, then it is a contradictory case, referred to as case 4, and the given formula is used as the criterion for judgment. Step 3.2: Use the erosion function to optimize the edges of the low-frequency coefficient fusion result, i.e., the edges of the low-frequency coefficient segmentation map, to reduce the ringing phenomenon in the fusion image caused by gradient diffusion, thereby generating a low-frequency coefficient fusion feature map; In step 3.2, the optimization strategy for the corrosion function is to first corrode and then diffuse.

[0044] Step 4: Perform NSST inverse transform on the fused low-frequency and high-frequency coefficients to obtain a high-resolution multispectral fused image.

[0045] In step 4, high-frequency coefficient feature maps are used to fuse the high-frequency coefficients of the PAN and MS images, and low-frequency coefficient feature maps are used to fuse the low-frequency coefficients of the PAN and MS images. Then, the high-frequency and low-frequency coefficients are subjected to NSST inverse transform to obtain the fused image.

[0046] This invention optimizes the high-frequency coefficient fusion of PCNN through directional adaptive gain and improves the classification processing rules for low-frequency coefficients, effectively enhancing the fusion quality of multispectral and panchromatic images. It can be widely applied in fields such as satellite remote sensing data processing, environmental monitoring, and urban planning.

[0047] The algorithm flow in this embodiment is as follows: Figure 1 As shown, the specific implementation method consists of the following steps: NSST transform parameter settings: A shear wave is constructed using an affine system, with expansion matrix A set to a=4, shear matrix B set to s=1, decomposition scale set to 2 levels, and the number of highest frequency directions being 16.

[0048] ADG-PCNN network construction: Orientation gradient calculation: Calculate the horizontal, vertical, 45-degree, and 270-degree orientation gradients of the PAN image according to formula (1), where the pixel gray value I ij This is the normalized value of the PAN image.

[0049] g 水平 =(I ij - I i-1,j ) 2 g 垂直 =(I ij - I i-1,j ) 2 g 斜右上 =(I ij - I i-1,j ) 2 g 斜右下 =(I ij - I i-1,j ) 2 (1) Gain map generation: Sort the four-directional gradients of each pixel: the direction with the largest gradient is the significant direction (gain 2); the vertical direction is the insignificant direction (gain 0.5); the rest are adjacent directions (gain 1).

[0050] Generate gain maps (MAPs) in 16 directions. l , l=1,...,16.

[0051] ADG-PCNN network architecture as follows Figure 2 As shown.

[0052] Network iteration parameters: V F =0.1, V L =0.2, V T =0.6, iterative feedback parameter exp (-α) F )=0.2,exp (-α) L )=0.5,exp (-α T =0.2, and the number of iterations is set to 20.

[0053] The iteration follows the formula: (2) in, The activation results are output by the ADG-PCNN network. For the number of iterations, For feedback input, For each element in the activation weight matrix W input to the link, i,j are the coordinates of the pixel to be activated in the image, and k,l are the coordinates of the activation weight matrix. This is the output of the previous iteration cycle. This is the directional gain map of pixel (i,j). External activation value, The iterative feedback parameters are the input feedback parameters. The x-coordinate of the pixel is The ordinate of the pixel is This indicates the direction of the shear wave transformation.

[0054] After iteratively applying the highest frequency coefficients of the PAN and MS images to each direction using the PCNN network, the total number of activations of the network during the iteration cycle is accumulated, i.e.: (3) T represents the number of activations at point (I, j) after the nth iteration along the l direction.

[0055] High-frequency coefficient selection: According to formula (4), select the PAN or MS high-frequency coefficient with the highest cumulative activation count as the fusion result.

[0056] (4) Where A is the PAN image and B is the MS image. Coordinates are The high-frequency NSST coefficients of the pixels in the l direction. Similarly.

[0057] Low-frequency coefficient fusion rules: Energy and gradient calculation: Energy E L (i,j) represents the square of the pixel gray value, and the gradient G L (i,j) represents the gradient magnitude calculated by the Sobel operator.

[0058] Handling four types of situations: For cases 1 and 2: use the product of energy and gradient according to formula (5) to determine the case. (5) Contradictory cases 3 and 4: Decision based on the majority pixel source of the 8-neighborhood according to formulas (6) and (7).

[0059] (6) (7) in The number of pixels in the 8-neighborhood of pixel (i,j) that are considered as low-frequency coefficients in the PAN image. Similarly, among the 8 neighboring pixels of pixel (i,j), the number of pixels with low-frequency coefficients is determined as the number of low-frequency coefficients in the MS image.

[0060] Edge optimization: A 3×3 erosion operator is used to perform one erosion operation on the low-frequency fusion results to reduce gradient diffusion.

[0061] Image fusion quality is generally evaluated from both subjective and objective perspectives.

[0062] This embodiment applies to remote sensing images. Therefore, L4-level remote sensing image products from the JL-1 wide-swath 01B satellite are selected as the source image group to be fused. This resource has been radiometrically calibrated, orthorectified, and atmospherically corrected, supporting direct fusion operations.

[0063] The original remote sensing image data has a panchromatic image size of 31779×31222 and a multispectral image size of 7928×7786×4. All four sets of test source images were obtained by cropping and processing from this data.

[0064] The resulting panchromatic image is 1024×1024 pixels, and the multispectral image is 256×256×4 pixels. Since there is no fused reference image, the original multispectral image data is used as the reference image, and both the panchromatic and multispectral source images are downsampled simultaneously as input images.

[0065] After downsampling, the size of the panchromatic source image is 256×256, the size of the multispectral source image is 64×64×4, and the desired size of the fusion result is 256×256×4.

[0066] Four objective metrics were selected for quantitative evaluation of the algorithm performance: Spectral Angle Mapper (SAM), ErrorRelative Global Accuracy (ERGAS), Correlation Coefficient (CC), and Quaternion Theory-based Quality Index (Q4).

[0067] The spectral angular distortion (SAM) transforms the spectral difference between the fused image and the reference image into a vector angle. The smaller the value, the less spectral distortion there is and the better the fusion effect. The ideal value is 0.

[0068] The SAM calculation formula (8) is shown.

[0069] (8) in and These represent the spectral vectors constructed from all bands of point i in the reference image and the fused image, respectively. This represents the dot product operation. This represents the L2 norm operation. After calculating the spectral angular distortion values ​​of all pixels and averaging them, we can obtain the overall spectral angular distortion value that reflects the fused result compared to the reference image.

[0070] Global Relative Error (ERGAS) examines the relative error of all spectral channels between the fused multispectral image and the reference multispectral image, reflecting the degree of distortion of the fused image in spectral information relative to the reference image. Its ideal value is 0.

[0071] The ERGAS calculation formula (9) is shown.

[0072] (9) in represents the root mean square error of the i-th band of the reference image and the fused image. The formula for calculating the root mean square error is shown in (10). represents the average distortion of the entire image.

[0073] (10) This represents the mean value of the reference image in the i-th band. R is the degradation factor, which is the ratio of the size of the multispectral source image to the fused image; it is typically taken as... .

[0074] Spatial correlation (CC) examines the correlation between the source image and the reference image. The value of CC ranges from [0,1]. A value of 1 indicates that the fused image is consistent with the reference image.

[0075] The CC calculation formula (11) is shown.

[0076] (11) in, From the fused image, From the reference image, These are the corresponding coordinates. and These are the mean values ​​of the fused image and the reference image in that channel, respectively. The CC values ​​of all channels are calculated and then averaged.

[0077] The quaternion-based image quality index Q4 is an extension of the generalized image quality index UIQI (Universal Image Quality Index), primarily used to evaluate remote sensing image data with four spectral bands. Compared to UIQI, Q4 calculation first models each pixel I of the multispectral image as a quaternion, that is, linearly combining the four bands.

[0078] UIQI assesses the correlation, average brightness difference, and contrast difference between the fused image and the reference image. The UIQI index ranges from -1 to 1, with a value of 1 when the fused result is completely identical to the reference image.

[0079] Q4 between the fused image and the reference image can be calculated using formula (12): (12) Where I {i} J {i} This is a multispectral image quadruple after linear combination.

[0080] The UIQI calculation formula (13) is shown.

[0081] (13) in and These represent the average pixel values ​​of the reference image and the fused image, respectively. For reference image and fusion results The covariance.

[0082] Three classic algorithms were selected as comparison algorithms. Among them, IHS and PCA are fusion algorithms based on variable substitution, and NSST-PCNN is a baseline algorithm that applies the classic PCNN to image fusion.

[0083] Table 1 compares the evaluation metrics for test image 1. Table 2 compares the evaluation metrics for test image 2. Table 3 compares the evaluation metrics for test image 3. Table 4 compares the evaluation metrics for test image 4.

[0084] Table 1 Comparison of evaluation metrics for test image 1

[0085] Table 2 Comparison of evaluation metrics for test image 2

[0086] Table 3 Comparison of evaluation metrics for test image 3

[0087] Table 4 Comparison of evaluation metrics for test image 4

[0088] Among the SAM metrics, the algorithm proposed in this chapter performs best. The IHS and PCA related algorithms suffer from severe spectral distortion due to the large amount of spatial information injected. The NSST-PCNN algorithm based on multi-resolution analysis performs better than the algorithm based on component replacement.

[0089] The correlation coefficient (CC) indicates that our algorithm achieves a high correlation with NSST-PCNN, suggesting that they can extract more spatial information from panchromatic images and possess clearer edge textures. In contrast, component substitution-based methods like IHS and PCA show poorer correlation. Furthermore, a horizontal comparison of the four test images shows that simpler edge details result in better correlation, and the improvement from our algorithm is more significant.

[0090] In the ERGAS metric, the algorithm proposed in this section improves both spatial and spectral distortion. Therefore, in flat images, the algorithm in this paper is significantly better than other metrics. However, for test images such as test images 2 and 3, which have relatively complex edges and spectra, the difference is not significant.

[0091] In the Q4 metrics, for test images 1 and 4 with relatively flat image characteristics, the algorithm in this paper maintains high spatial imaging quality. Conversely, for test images 2 and 3 with more complex image characteristics, the metrics of various algorithms are not high, but the algorithm in this paper still maintains its leading position.

[0092] In this embodiment, the unspecified components such as NSST transformation, erosion optimization, and test image interpolation to generate source images can all be implemented using existing technologies. Without departing from the principle of this invention, the methods for optimizing and refining the above three items should also be considered within the scope of protection of this invention.

[0093] The four test image examples shown in this embodiment cover all common remote sensing image types, namely flat images, straight-edge images, densely textured images, and irregular-edge images. The degree of optimization achieved by this invention in objective indicators can be used as a typical value.

[0094] Because of the differences in features between the test images, when using different remote sensing test images for objective index evaluation, the degree of optimization of the algorithm of this invention relative to the comparison algorithm will vary depending on the differences in features.

[0095] This embodiment also provides a multispectral and panchromatic image fusion system based on directional adaptive gain and PCNN. The system includes: a first module for performing non-subsampled shear wave transform on the panchromatic image to obtain low-frequency and high-frequency coefficients of the panchromatic image, and performing non-subsampled shear wave transform on the multispectral image to obtain low-frequency and high-frequency coefficients of the multispectral image; a second module for constructing an adaptive directional gain pulse-coupled neural network, and fusing the high-frequency coefficients of the panchromatic image and the high-frequency coefficients of the multispectral image through the adaptive directional gain pulse-coupled neural network to obtain fused high-frequency coefficients; a third module for fusing the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image based on the energy comparison results of the panchromatic image and the energy comparison results of the multispectral image, and the gradient comparison results of the panchromatic image and the gradient comparison results of the multispectral image to obtain fused low-frequency coefficients; and a fourth module for performing non-subsampled shear wave inverse transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fused image.

[0096] This embodiment specifically optimizes the high- and low-frequency coefficients obtained after non-subsampled shear wave transform (NSST) decomposition, thereby improving the subjective visual effect and objective evaluation of the fusion result. For high-frequency coefficients, an adaptive directional gain pulse coupled neural network (ADG-PCNN) is constructed, and a gain map is generated by combining the directional features of the PAN image to enhance the directional discrimination accuracy of high-frequency coefficient fusion. For low-frequency coefficients, fusion rules are designed based on the relationship between energy and gradient, and edges are optimized through an erosion function to reduce ringing. This invention effectively improves the spatial resolution and spectral fidelity of the fused image and is suitable for scenarios such as satellite remote sensing data processing.

[0097] This embodiment introduces directional gain in high-frequency coefficient fusion, improving the accuracy of preserving directional details in PAN images; this embodiment balances the influence of energy and gradient in low-frequency coefficient classification and edge optimization, reducing ringing phenomena; this embodiment optimizes spatial and spectral resolution in the fused image, making it suitable for scenarios such as remote sensing image interpretation and target detection.

[0098] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for multispectral and panchromatic image fusion based on directional adaptive gain and PCNN, characterized in that The method comprises the following steps: performing non-subsampled shearlet transform on the panchromatic image to obtain low-frequency coefficients and high-frequency coefficients of the panchromatic image, and performing non-subsampled shearlet transform on the multispectral image to obtain low-frequency coefficients and high-frequency coefficients of the multispectral image; constructing an adaptive direction gain pulse coupled neural network, and fusing the high-frequency coefficients of the panchromatic image and the high-frequency coefficients of the multispectral image through the adaptive direction gain pulse coupled neural network to obtain fused high-frequency coefficients; fusing the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image according to a comparison result of energies of the panchromatic image and the multispectral image and a comparison result of gradients of the panchromatic image and the multispectral image to obtain fused low-frequency coefficients; performing non-subsampled shearlet inverse transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fusion image.

2. The method according to claim 1, wherein the method is based on the directional adaptive gain and PCNN for multispectral and panchromatic image fusion. performing NSST two-level decomposition on each channel of the panchromatic image to obtain low-frequency coefficients and high-frequency coefficients of 16 angles of the panchromatic image; performing NSST two-level decomposition on each channel of the multispectral image to obtain low-frequency coefficients and high-frequency coefficients of 16 angles of the multispectral image.

3. The method according to claim 1, wherein the method is based on the directional adaptive gain and PCNN for multispectral and panchromatic image fusion. The construction of the adaptive direction gain pulse coupled neural network comprises the following steps: extracting edge gradient information of four directions of horizontal, vertical, 45 degrees and 270 degrees from the panchromatic image; based on the edge gradient information, dividing the four directions of each pixel point into a significant direction, an adjacent direction and an insignificant direction, and generating a gain map of the corresponding angle; injecting the gain map into an iteration process of a PCNN network, determining the significance of the high-frequency coefficients of the panchromatic image through the cumulative number of activations, and constructing a high-frequency coefficient fusion feature map, that is, selecting coefficients with a high number of activations as the fusion result.

4. The method according to claim 3, wherein the method is based on the directional adaptive gain and PCNN. According to the division of the highest frequency direction of the non-subsampled shearlet transform, the edge gradient information of 16 angles is calculated, and the edge gradient information of the 16 angles is classified into four directions of horizontal, vertical, 45 degrees and 270 degrees; wherein, four angle edge gradient information values are allocated to each direction.

5. The method according to claim 3, wherein the method is based on the directional adaptive gain and PCNN for multispectral and panchromatic image fusion. The generation rule of the gain map is: the 4 angle gains corresponding to the significant direction are set to 2, the 8 angle gains corresponding to the adjacent direction are set to 1, and the 4 angle gains corresponding to the insignificant direction are set to 0.5; a gain map is generated for each image point (i, j) where i is the horizontal coordinate and j is the vertical coordinate, is the number of the coefficient map.

6. The method of claim 1, wherein the method is based on a directional adaptive gain and PCNN for multispectral and panchromatic image fusion. The fusing of the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image according to the comparison result of the energies of the panchromatic image and the multispectral image and the comparison result of the gradients of the panchromatic image and the multispectral image to obtain the fused low-frequency coefficients comprises the following steps: generating a low-frequency coefficient segmentation map according to the comparison result of the energies of the panchromatic image and the multispectral image and the comparison result of the gradients of the panchromatic image and the multispectral image; optimizing the edges of the generated low-frequency coefficient segmentation map using an erosion function to obtain the fused low-frequency coefficients.

7. The method according to claim 6, wherein the method is based on the directional adaptive gain and PCNN. The generation of the low-frequency coefficient segmentation map according to the comparison result of the energies of the panchromatic image and the multispectral image and the comparison result of the gradients of the panchromatic image and the multispectral image comprises the following steps: if the panchromatic image energy > multispectral image energy, and the panchromatic image gradient > multispectral image gradient, it is a unified case, then the product of the energy and the gradient is used as the determination basis, and the low-frequency coefficient segmentation map is obtained according to the determination basis; if the panchromatic image energy < multispectral image energy, and the panchromatic image gradient < multispectral image gradient, it is a unified case, then the product of the energy and the gradient is used as the determination basis, and the low-frequency coefficient segmentation map is obtained according to the determination basis; If the panchromatic image energy is greater than the multispectral image energy, and the panchromatic image gradient is less than the multispectral image gradient, it is a contradictory case, and a first given formula is used as a judgment basis to obtain a low-frequency coefficient segmentation map according to the judgment basis; If the panchromatic image energy is less than the multispectral image energy, and the panchromatic image gradient is greater than the multispectral image gradient, it is a contradictory case, and a second given formula is used as a judgment basis.

8. The method according to claim 7, wherein the method is based on the directional adaptive gain and PCNN. The first given formula is obtained by the following formula: ； wherein, is a low frequency coefficient of the fused image, is a low frequency coefficient of the source image A, is an image energy of the A image at the point (i,j), is an image energy of the B image at the point (i,j), is an image gradient of the A image at the point (i,j), is an image gradient of the B image at the point (i,j), is an abscissa, is an ordinate.

9. The method according to claim 7, wherein the method is based on the directional adaptive gain and PCNN for multispectral and panchromatic image fusion. The second given formula is obtained by the following formula: ； wherein, is the low frequency coefficients of the fused image, is the low frequency coefficients of the source image A, is the horizontal pixel coordinate, is the vertical pixel coordinate, is the number of pixels in the 8-neighborhood of pixel (i,j) whose low frequency coefficients are determined to be low frequency coefficients of the PAN image, is the number of pixels in the 8-neighborhood of pixel (i,j) whose low frequency coefficients are determined to be low frequency coefficients of the MS image.

10. A multispectral and panchromatic image fusion system based on directional adaptive gain and PCNN, characterized in that The method comprises the following steps: A first module is configured to perform non-subsampled shearlet transform on the panchromatic image to obtain low-frequency coefficients and high-frequency coefficients of the panchromatic image, and perform non-subsampled shearlet transform on the multispectral image to obtain low-frequency coefficients and high-frequency coefficients of the multispectral image; A second module is configured to construct an adaptive direction gain pulse coupled neural network, and fuse the high-frequency coefficients of the panchromatic image and the high-frequency coefficients of the multispectral image through the adaptive direction gain pulse coupled neural network to obtain fused high-frequency coefficients; A third module is configured to fuse the low-frequency coefficients of the panchromatic image and the low-frequency coefficients of the multispectral image according to a comparison result of the energy of the panchromatic image and the energy of the multispectral image and a comparison result of the gradient of the panchromatic image and the gradient of the multispectral image to obtain fused low-frequency coefficients; A fourth module is configured to perform non-subsampled shearlet inverse transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a high-resolution multispectral fusion image.

Images