SAR radar and infrared image fusion method for multi-source target enhancement
Through the fusion strategy of non-downsampled shear wave transformation and gradient-size principle, the problems of time-consuming and information loss of SAR and infrared images in the prior art are solved, and efficient and detailed image fusion effect is achieved.
Patent Information
- Application Number
- CN202210201466.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-03
AI Technical Summary
The existing SAR and infrared image fusion methods have time-consuming, information loss and pseudo-Gibbs phenomena, resulting in poor fusion effect, and low-frequency image fusion parameters rely on experience and cumbersome process.
The SAR and infrared images are processed by non-downsampled shear wave transformation, combined with Sobel and Laplacian operators to find the gradient, and fuse according to the absolute value of the gradient to avoid the pseudo-Gibbs phenomenon, retain the image contour edge information, and reduce redundancy.
The fusion process time is shortened, information loss is avoided, and the fusion effect is significantly improved. The fused image contains more scene details and the running time is greatly reduced.
Smart Images

Figure CN114581352B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing, and further relates to a method for fusing infrared and SAR images, which can be used for target detection. Background Art
[0002] Synthetic aperture radar SAR has the advantages of all-day, all-weather, and long detection distance, making it an irreplaceable reconnaissance tool. However, in some cases, due to the influence of environmental conditions, target status, target location, and SAR inherent characteristics, the image information obtained by a single SAR is not enough to better analyze and understand the target or scene. Infrared imaging mainly extracts potential targets based on the infrared radiation difference between the target and the background. At the same time, due to the penetration ability of the infrared imaging system, the infrared radiation of the target cannot be completely concealed by soil layers, water layers, and even concrete layers within a certain range. However, compared with SAR imaging, infrared imaging has the disadvantage of being easily affected by weather such as clouds, rain and fog. Therefore, combining the advantages of SAR images and infrared images, research on the fusion of SAR images and infrared images can greatly enhance the target features and thus improve the accuracy of target detection.
[0003] Xidian University disclosed a multi-scale SAR and infrared image fusion method based on target enhancement in the patent application document with publication number: CN112734683A. This method first performs non-subsampled contourlet transform on infrared images and SAR images to obtain multi-scale decomposition information of the source image, and uses the decomposed low-frequency infrared image to determine the low-frequency fusion weight, and uses the regional energy and gradient features of the high-frequency image to determine the high-frequency fusion coefficient. Although this method can solve the problem of spectral distortion, since the value of the low-frequency image fusion parameter λ depends on experience, different images have different λ values, and it may take several attempts to determine, so the process is relatively cumbersome, and this non-subsampled contourlet is time-consuming and has poor fusion effect.
[0004] In the paper "SAR and infrared image fusion based on shearlet domain gradient maximization strategy", Zhang Yukuan, Zhang Gong and others proposed a scheme to use shearlet transform to perform shearlet transform on infrared and SAR images to obtain infrared low-frequency and high-frequency directional images and SAR low-frequency and high-frequency directional images. The low-frequency image uses the Sobel operator to obtain the infrared low-frequency gradient image and SAR low-frequency gradient image, and the low-frequency fusion coefficient is calculated using the gradient maximization principle. The high-frequency directional image is directly maximized in the transform domain to obtain the high-frequency directional image fusion coefficient, and finally the inverse shearlet transform is used to obtain the final fused image. Although this method reduces the time consumption of the multi-scale SAR and infrared image fusion method based on target enhancement, the shearlet transform it uses is prone to information loss and the appearance of pseudo-Gibbs phenomenon, and no filtering operation is performed on the SAR image, resulting in poor image quality of the fusion effect image. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a multi-source target enhanced SAR radar and infrared image fusion method to achieve good fusion effect while reducing fusion time, enhance target features, and make the fused image include more scene detail information.
[0006] To achieve the above object, the implementation scheme of the present invention includes the following:
[0007] (1) Obtain the original SAR image, filter it, eliminate the spot effect, and obtain a smoothed SAR image;
[0008] (2) Register the original infrared and smoothed SAR images;
[0009] (3) Perform non-subsampled shearlet transform on the registered infrared and SAR images to obtain one infrared and one SAR low-frequency image, as well as 64 infrared and 64 SAR bandpass directional images;
[0010] (4) Using the Sobel operator to obtain the gradient of an infrared image and a SAR low-frequency image;
[0011] (5) Using the Laplacian operator to obtain the gradients of 64 infrared and 64 SAR bandpass images;
[0012] (6) Fusing the low-frequency images of an infrared image and a SAR image, and obtaining a fused low-frequency image according to the principle of taking the maximum absolute value of the gradient;
[0013] (7) The passband directional images of 64 infrared images and 64 SAR images are fused, and 64 fused passband directional images are obtained according to the principle of taking the maximum absolute value of the gradient;
[0014] (8) Perform non-subsampled shearlet inverse transform on the fused low-frequency image and the fused 64 bandpass direction images to obtain the final fused image.
[0015] Compared with the prior art, the present invention has the following advantages:
[0016] 1. The present invention uses non-subsampled shearlet transform, which is time-consuming compared to non-subsampled contourlet transform;
[0017] 2. Since the present invention adopts non-subsampled shearlet transform, compared with shearlet, it avoids the pseudo-Gibbs effect of the image and reduces the redundancy of the image.
[0018] 3. Since the present invention adopts the gradient maximum fusion strategy, the contour edge information of the original infrared image and the SAR image is retained to a great extent, so that the fused image includes more scene detail information and has a good fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flow chart of the implementation of the present invention;
[0020] Figure 2 The diagram is a result of fusing the SAR image with the infrared image using the present invention and the existing fusion method respectively. DETAILED DESCRIPTION
[0021] The embodiments and effects of the present invention will be further described in detail below in conjunction with the accompanying drawings. The following examples are only used to illustrate the present invention and should not be regarded as the scope of the present invention.
[0022] refer to Figure 1 The infrared and SAR image fusion method based on non-subsampled shearlet transform of the present invention comprises the following steps:
[0023] Step 1: Filter the original SAR image to eliminate the speckle effect.
[0024] Since the SAR image used in this example is multiplicative noise, the Frost filtering algorithm is selected for filtering, that is, a smoothing window containing a matrix of (2M+1)(2N+1) pixels is used to traverse each pixel value in the image in turn to obtain the grayscale value g' of the smoothed pixel ij , which is implemented as follows:
[0025] 1.1) Assuming the sliding window size is (2M+1)(2N+1), calculate the average value of the SAR image and standard deviation σ ij :
[0026]
[0027]
[0028] 1.2) Calculate the mean standard deviation A of SAR image pixels ij :
[0029]
[0030] 1.3) Calculate the weight index M of each corresponding pixel in the SAR image ij :
[0031] M ij =exp(-A ij ×T ij ), where Tij is the absolute distance from other pixels in the sliding window to the central pixel;
[0032] 1.4) Calculate the gray value g' of the pixel after filtering ij :
[0033]
[0034] Where: g' ij is the gray value of the pixel after filtering; is the mean variance of the pixel values in the smoothing window; The critical points representing the uniform region and the fully developed region of coherent speckle noise; represents the critical point between the fully developed area of coherent speckle noise and the target area containing the separation point; and C I <C u It indicates that the smoothing window is a uniform area, C u ≤C I ≤C max This indicates that all pixels in the smoothing window are fully developed areas of coherent speckle noise, C I >C max Indicates the target area containing the separation point, and L is the number of imaging views.
[0035] Step 2: align the original infrared image and the SAR image after frost filtering.
[0036] First, extract the CFOG feature descriptors from the infrared and filtered SAR images, and then use the error square sum algorithm SSD for registration. The specific implementation is as follows:
[0037] 2.1) Calculate the horizontal gradient channel g of the infrared and filtered SAR images respectively x and vertical gradient channel g y :
[0038] g x =g' ij *[-1,0,1]
[0039] g y =g ij *[-1,0,1] T
[0040] Among them, g' ij is the grayscale value of the pixel after smoothing, T represents transposition, and * represents convolution;
[0041] 2.2) Calculate the gradient channel g in each direction of the infrared and filtered SAR images i (θ):
[0042] g i(θ) = abs(cosθ×g x +sinθ×g y )
[0043] Where θ is the quantized gradient direction, 1≤i≤m, i represents the gradient direction number, and m is the number of gradient directions; abs represents the absolute value, which can convert the gradient direction θ from [180°, 360°) to [0°, 180°);
[0044] 2.3) Using Gaussian kernel g σ The gradient channel g of the infrared and filtered SAR images i (θ) is convolved to obtain the feature channel after convolution
[0045]
[0046] in x represents the horizontal coordinate, y represents the vertical coordinate, σ is the standard deviation of the Gaussian kernel, and g' ij is the gray value of the pixel after smoothing. represents the gradient of the smoothed image in the θ direction, [] + Indicates that the calculation result is greater than or equal to zero, and * indicates convolution;
[0047] 2.4) Calculate the characteristic channel D1 of the infrared image and the characteristic channel D2 of the smoothed SAR image using 2.1)-2.3) respectively, and then use the error square sum algorithm SSD to calculate the difference S between the two characteristic channels i (v):
[0048]
[0049] Where k is the position of the pixel in the three-dimensional feature representation, D1(k) represents the infrared image feature channel at the pixel k position, D2(kv) is the feature channel of the smoothed SAR image at the pixel kv position, and v is the offset vector; T i (k) is the masking function on D1(k), within the sliding window T i (k) = 1, outside the sliding window T i (k) = 0;
[0050] 2.5) For S i (v) Minimize and realize the registration between the feature channel D1 of the original infrared image and the feature channel D2 of the filtered SAR image:
[0051]
[0052] where v irepresents the minimum offset vector that matches D1 and D2 within the smoothing window i.
[0053] Step 3: Perform non-subsampled shearlet transform on the registered infrared image and SAR image to obtain infrared low-frequency images and SAR bandpass directional images with different number combinations.
[0054] 3.1) Performing non-subsampled pyramid transformation on the registered infrared image and SAR image to obtain the first layer low-frequency image and the first layer bandpass image;
[0055] 3.2) Decompose the first layer of bandpass image into 8 directions to obtain 8 bandpass directional images of the first layer;
[0056] 3.3) performing a non-subsampled pyramid transform on the first layer low-frequency image to obtain a second layer low-frequency image and a second layer bandpass image;
[0057] 3.4) Decomposing the second layer bandpass image into 8 directions to obtain 8 bandpass directional images of the second layer;
[0058] 3.5) performing a non-subsampled pyramid transform on the second-layer low-frequency image to obtain a third-layer low-frequency image and a third-layer bandpass image;
[0059] 3.6) Decomposing the third layer bandpass image into 16 directions to obtain 16 bandpass directional images of the third layer;
[0060] 3.7) performing a non-subsampled pyramid transform on the third layer low-frequency image to obtain a fourth layer low-frequency image and a fourth layer bandpass image;
[0061] 3.8) Decompose the fourth layer bandpass image in 32 directions to obtain 32 bandpass directional images of the fourth layer. The fourth layer low-frequency image is the final low-frequency image required, where the infrared low-frequency image is represented by A L Indicates that SAR low-frequency images are represented by B L Indicates that the first layer has 8 bandpass directional images, the second layer has 8 bandpass directional images, the third layer has 16 bandpass directional images, and the fourth layer has 32 bandpass directional images, a total of 64 bandpass directional images, of which 64 infrared bandpass directional images are represented by A H Indicates that 64 SAR passband images are represented by B H express.
[0062] Step 4: Use the Sobel operator to calculate the gradient of the low-frequency images of the infrared image and the SAR image.
[0063] 4.1) Taking the horizontal right direction as the horizontal positive direction and the vertical upward direction as the vertical positive direction, calculate the horizontal gradient P5 of the low-frequency image pixel respectively x (x,y) and vertical gradient P5 y(x,y):
[0064] P5 x (x,y)=(P3-P1)+2(P6-P4)+(P9-P7)
[0065] P5 y (x,y)=(P7-P1)+2(P8-P2)+(P9-P3)
[0066] Among them, the coordinates of P3 are (x+1, y+1), the coordinates of P1 are (x-1, y+1), the coordinates of P4 are (x-1, y), the coordinates of P6 are (x+1, y), the coordinates of P7 are (x-1, y-1), and the coordinates of P9 are (x+1, y-1);
[0067] 4.2) Horizontal gradient P5 obtained according to 4.1) x (x,y) and vertical gradient P5 y (x, y) calculates the gradient value P5(x, y) and gradient direction θ(x, y) of the low-frequency image:
[0068]
[0069]
[0070] 4.3) For the low-frequency infrared image, keep the edge pixels of the image unchanged, and use the formulas 4.1)-4.2) to calculate the gradient of other pixels to obtain the low-frequency infrared gradient image A' L ;
[0071] 4.4) For the low-frequency SAR image, keep the edge pixels of the image unchanged, and use the formulas 4.1)-4.2) to calculate the gradient of other pixels to obtain the low-frequency SAR gradient image B' L .
[0072] Step 5: Use the second-order differential operator Laplacian to calculate the gradient of the bandpass direction images of the infrared image and the SAR image.
[0073] 5.1) Taking the horizontal right direction as the horizontal positive direction and the vertical upward direction as the vertical positive direction, calculate the gradient L5(x,y) of the pixel points in the pass direction respectively:
[0074] L5(x,y)=L2+L4+L6+L8-4*L5
[0075] The coordinates of L2 are (x, y+1), the coordinates of L4 are (x-1, y), the coordinates of L6 are (x+1, y), and the coordinates of L8 are (x, y-1). * indicates multiplication.
[0076] 5.2) For the 64 infrared bandpass direction images, keep the edge pixels of the image unchanged, and use the formula in 5.1) to calculate the gradient of other pixels to obtain 64 infrared bandpass direction gradient images A' H ;
[0077] 5.2) For the 64 SAR bandpass direction images, keep the edge pixels of the image unchanged, and use the formula in 5.1) to calculate the gradient of other pixels to obtain 64 SAR bandpass direction gradient images B' H .
[0078] Step six, according to the principle of taking the maximum absolute value of the low-frequency image gradient, the low-frequency fusion coefficient after the fusion of the infrared image and the SAR image is obtained.
[0079] The low-frequency infrared gradient image A' calculated in step 4 L and the gradient image B' of low-frequency SAR L For comparison:
[0080] If A' L The gradient value A' at (x,y) L (x,y) absolute value is greater than B' L The gradient value B' at (x,y) L (x, y), then the low-frequency fusion coefficient after the fusion of the infrared image and the SAR image is the pixel value A of the infrared low-frequency image at (x, y). L (x,y),
[0081] Otherwise, the fusion coefficient is taken as the pixel value B of the SAR low-frequency image at (x, y) L (x,y).
[0082] The formula is as follows:
[0083]
[0084] Among them, F L is the low frequency fusion coefficient, F L (x, y) represents the pixel value of the low-frequency fusion coefficient at (x, y), A L (x, y) represents the pixel value of the infrared low-frequency image at (x, y), B L (x, y) represents the pixel value of the SAR low-frequency image at (x, y); A' L (x, y) is the gradient value of the infrared low-frequency gradient image at (x, y), B' L (x, y) is the gradient value of the SAR low-frequency gradient image at (x, y).
[0085] Step 7: Use the absolute value of the gradient of the bandpass direction image to take a large fusion strategy and obtain 64 bandpass direction fusion coefficients.
[0086] The 64 infrared bandpass gradient images A' calculated in step 5 are H and 64 SAR bandpass gradient images B' H For comparison:
[0087] If A' H The gradient value A' at (x,y) H (x,y) absolute value is greater than B' H The gradient value B' at (x,y) H (x, y), then the bandpass fusion coefficient after the fusion of the infrared image and the SAR image is the pixel value A of the infrared low-frequency image at (x, y). H (x,y),
[0088] Otherwise, the fusion coefficient is taken as the pixel value B of the SAR bandpass image at (x, y) H (x,y).
[0089] The formula is as follows:
[0090]
[0091] Among them, F H represents the bandpass fusion coefficient, F H (x, y) represents the pixel value of the bandpass fusion coefficient at (x, y), A H (x, y) is the pixel value of the infrared bandpass image at (x, y), B H (x, y) is the pixel value of the SAR bandpass image at (x, y), A' H (x, y) is the gradient value of the infrared bandpass directional gradient image at (x, y), B' L (x, y) is the gradient value of the SAR bandpass directional gradient image at (x, y).
[0092] Step eight, performing non-subsampled shearlet inverse transform on the low-frequency fusion coefficients obtained in step six and the bandpass direction image fusion coefficients obtained in step seven to obtain a final fused image.
[0093] The effect of the present invention can be further illustrated by the following simulation experiment:
[0094] 1. Simulation Experiment Conditions
[0095] The simulation data takes the measured SAR images and infrared images of the airport runway as an example.
[0096] The simulation software is MATLAB 2015b, the hardware processor is Intel(R) Core(TM) i5-5200U CPU@2.20GHz 2.20GHz, the memory is 500MB, and the onboard RAM is 4GB.
[0097] Evaluation indicators:
[0098] (1) Structural Similarity Index SSIM
[0099] Given two images x and y, the structural similarity of the two images can be found as follows:
[0100]
[0101] where μ x is the mean value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability. L is the dynamic range of pixel values. k1 = 0.01, k2 = 0.03. The range of the structural similarity index is -1 to 1. Figure 1 When the two models are identical, the SSIM value is 1.
[0102] (2) Mean Square Error (MSE)
[0103] This statistical parameter is the mean of the sum of squares of the corresponding pixel differences between the fused image and the original image, and its calculation formula is as follows:
[0104]
[0105] Where n is the number of pixels, i is the index item, y is the pixel value of the fused image at position i, and x is the pixel value of the original infrared image or SAR image at position i.
[0106] (3) Peak signal-to-noise ratio (PNSR)
[0107] The peak signal-to-noise ratio is defined according to the mean square error as:
[0108]
[0109] Among them, MAX I It represents the maximum value of the pixel value of the fused image.
[0110] Based on the non-subsampled shearlet transform algorithm NSST, “9-7” is used as the pyramid filter. The filter size is [32 32 16 16], the number of decomposition layers is 4, and the number of directions of each decomposition layer is [3 3 4 5].
[0111] The fusion algorithm NSCT based on non-subsampled contourlet transform also decomposes into four layers, and the number of decomposition directions of each layer is [3 3 4 4].
[0112] 2. Simulation experiment content
[0113] Experiment 1: Under the above environment and conditions, the present invention simulates the fusion of original infrared and SAR images based on the non-subsampled shearlet transform algorithm and the existing non-subsampled contourlet transform algorithm. The results are as follows: Figure 2 As shown, 2(a) is the original SAR image, Figure 2 (b) is the original infrared image, Figure 2 (c) is the fusion result based on non-subsampled shearlet transform of the present invention, Figure 2 (d) is the existing fusion result based on non-subsampled contourlet transform.
[0114] Experiment 2: The fusion effect of the present invention and the existing non-subsampled contourlet-based method are evaluated from a subjective perspective, and the time efficiency is judged from the running time. The results are shown in Table 1.
[0115] Table 1 Quantitative analysis of the fusion effect of the present invention and the existing algorithm
[0116] Numeric <![CDATA[SSIM A ]]> <![CDATA[SSIM B ]]> <![CDATA[MSE A ]]> <![CDATA[MSE B ]]> <![CDATA[PNSR A ]]> <![CDATA[PNSR B ]]> Run time (seconds) The present invention 0.541 0.690 1542 3717 16.25 12.43 1.96 NSCT 0.594 0.641 1490 3751 16.40 12.39 82.03
[0117] The bold values in the table indicate good fusion effects, among which:
[0118] SSIM A Represents the structural similarity index between the infrared image and the fusion result image,
[0119] SSIM B Represents the structural similarity index between the SAR image and the fusion result image;
[0120] MSE A represents the mean square error between the infrared image and the fusion result image,
[0121] MSE B represents the mean square error between the SAR image and the fusion result image,
[0122] PNSR A Represents the peak signal-to-noise ratio of the infrared image and the fusion result image,
[0123] PNSR B Represents the peak signal-to-noise ratio of the SAR image and the fusion result image.
[0124] Comparing the two groups of experiments, from a subjective point of view, the fusion effects of the present invention based on non-subsampled contourlet transform and the existing method based on non-subsampled shearlet transform are similar; from the objective structural similarity index, mean square error, and peak signal-to-noise ratio, the fusion effects are also similar, but the running time of the present invention is very short, only 1.96 seconds, which is much more efficient than the existing method of 82.03 seconds.
Claims
1. A SAR radar and infrared image fusion method for multi-source target enhancement, characterized in that: include: (1) Obtain the original SAR image, filter it, eliminate the spot effect, and obtain a smoothed SAR image; (2) Register the original infrared and smoothed SAR images; (3) Perform non-subsampled shearlet transform on the registered infrared and SAR images to obtain one infrared and one SAR low-frequency image, as well as 64 infrared and 64 SAR bandpass directional images; (4) Using the Sobel operator to obtain the gradient of an infrared image and a SAR low-frequency image; (5) Using the Laplacian operator to obtain the gradients of 64 infrared and 64 SAR bandpass images; (6) The low-frequency images of an infrared image and a SAR image are fused, and the low-frequency fusion coefficients of the fused image are obtained according to the principle of taking the maximum absolute value of the gradient, which is implemented as follows: where F L (x, y) represents the pixel value of the low-frequency fusion coefficient at (x, y), A L (x, y) represents the pixel value of the infrared low-frequency image at (x, y), B L (x, y) represents the pixel value of the SAR low-frequency image at (x, y), A' L is the gradient value of the infrared low-frequency gradient image at (x, y), B' L is the gradient value of the SAR low-frequency gradient image at (x, y); (7) fusing the 64 infrared and 64 SAR image bandpass directional images, and obtaining the fused 64 bandpass directional coefficients according to the same principle as in step (6); (8) Perform non-subsampled shearing inverse wave transform on the fused low-frequency image and the fused 64 bandpass direction images to obtain the final fused image.
2. The method according to claim 1, characterized in that In (1), the original SAR image is filtered by using the frost filter to eliminate noise, that is, a smoothing window containing a matrix of (2M+1)(2N+1) pixels is used to traverse each pixel value in the image in turn to obtain the grayscale value g of the smoothed pixel. i ' j : Where: g i ' j is the gray value of the pixel after smoothing; g ij is the original gray value of each pixel in the smoothing window; is the average grayscale value of pixels in the smoothing window; M ij is the weight index of each corresponding pixel in the smoothing window; T ij is the absolute distance from the center pixel to its adjacent pixels in the smoothing window; σ ij is the variance of the pixel value in the smoothing window; n 2 is the size of the smoothing window; is the mean variance of the pixel values in the smoothing window; The critical points representing the uniform region and the fully developed region of coherent speckle noise; represents the critical point between the fully developed area of coherent speckle noise and the target area containing the separation point; and C I <C u It indicates that the smoothing window is a uniform area, C u ≤C I ≤C max This indicates that all pixels in the smoothing window are fully developed areas of coherent speckle noise, C I >C max Indicates the target area containing the separation point, and L is the number of imaging views.
3. The method according to claim 1, characterized in that In (2), the original infrared and filtered SAR images are registered using the CFOG feature descriptor, which is implemented as follows: (2a) Calculate the horizontal gradient channel g for the acquired infrared and filtered SAR images respectively x and vertical gradient channel g y : g x =g i ' j *[-1,0,1] g y =g ij *[-1,0,1] T Among them, g i ' j is the grayscale value of the pixel after smoothing, T represents transposition, and * represents convolution; (2b) Calculate the gradient channel g in each direction i (θ): g i (θ)=abs(cosθ×g x +sinθ×g y ), Where θ is the quantized gradient direction, 1≤i≤m, i represents the gradient direction number, and m is the number of gradient directions; abs represents the absolute value, which can convert the gradient direction θ from [180°, 360°) to [0°, 180°); (2c) Using Gaussian kernel g σ For the gradient channel g i (θ) is convolved to obtain the feature channel after convolution in x represents the horizontal coordinate, y represents the vertical coordinate, σ is the standard deviation of the Gaussian kernel, and g′ ij is the gray value of the pixel after smoothing. represents the gradient of the smoothed image in the θ direction, [] + Indicates that the calculation result is greater than or equal to zero, and * indicates convolution; (2d) Using (2a)-(2c), we calculate the infrared characteristic channel D1 and the smoothed SAR image characteristic channel D2 respectively, and then use the error square sum algorithm SSD to calculate the difference S between the two characteristic channels. i (v): Where k is the position of the pixel in the three-dimensional feature representation, D1(k) represents the infrared image feature channel at the pixel k position, D2(kv) is the feature channel of the smoothed SAR image at the pixel kv position, and v is the offset vector; T i (k) is the masking function on D1(k), within the sliding window T i (k) = 1, outside the sliding window T i (k) = 0; (2e) For S i (v) Minimize and realize the registration between the feature channel D1 of the original infrared image and the feature channel D2 of the filtered SAR image: where v i represents the minimum offset vector that matches D1 and D2 within the smoothing window i.
4. The method according to claim 1, characterized in that: In (3), the non-subsampled shearlet transform is performed on the registered infrared and SAR images, which is implemented as follows: (3a) performing non-subsampled pyramid transformation on the registered infrared and SAR images to obtain the first-layer low-frequency image and the first-layer bandpass image; (3b) Decomposing the first layer of bandpass images in 8 directions to obtain 8 bandpass directional images of the first layer; (3c) performing a non-subsampled pyramid transform on the first layer low-frequency image to obtain a second layer low-frequency image and a second layer bandpass image; (3d) Decomposing the second layer bandpass image in 8 directions to obtain 8 bandpass directional images of the second layer; (3e) performing a non-subsampled pyramid transform on the second-layer low-frequency image to obtain a third-layer low-frequency image and a third-layer bandpass image; (3f) performing 16-directional decomposition on the third-layer bandpass image to obtain 16 bandpass directional images of the third layer; (3g) performing a non-subsampled pyramid transform on the third-layer low-frequency image to obtain a fourth-layer low-frequency image and a fourth-layer bandpass image; (3h) Decomposing the fourth layer bandpass image in 32 directions to obtain 32 bandpass direction images of the fourth layer; Finally, we get one infrared low-frequency image and 64 infrared bandpass directional images, one SAR low-frequency image and 64 SAR bandpass directional images.
5. The method according to claim 1, characterized in that In (4), the Sobel operator is used to obtain the gradient of an infrared image and a SAR low-frequency image, which is implemented as follows: (4a) Calculate the horizontal gradient P5 of the pixel point with the horizontal right direction as the horizontal positive direction and the vertical upward direction as the vertical positive direction. x (x,y) and vertical gradient P5 y (x,y): P5 x (x,y)=(P3-P1)+2(P6-P4)+(P9-P7) <h2 style=";text-align:left;direction:ltr">P5<h2 style=";text-align:left;direction:ltr"> y <h2 style=";text-align:left;direction:ltr"> (x,y)=(P7-P1)+2(P8-P2)+(P9-P3) Among them, the coordinates of P3 are (x+1, y+1), the coordinates of P1 are (x-1, y+1), the coordinates of P4 are (x-1, y), the coordinates of P6 are (x+1, y), the coordinates of P7 are (x-1, y-1), and the coordinates of P9 are (x+1, y-1); (4b) According to the horizontal gradient P5 of the pixel point x (x,y) and vertical gradient P5 y (x, y) calculates the gradient value P5(x, y) and the gradient direction θ(x, y): (4c) For an infrared image and a SAR low-frequency image, keep the edge pixels of the image unchanged, and use (4a)-(4b) to calculate the gradient of other pixels to obtain the gradient of the entire image.
6. The method according to claim 4, characterized in that In (5), the Laplacian operator is used to obtain the gradient of 64 infrared and 64 SAR bandpass directional images, which is implemented as follows: (5a) Calculate the horizontal gradient L5(x,y) of the pixel point with the horizontal right direction as the horizontal positive direction and the vertical upward direction as the vertical positive direction: L5(x,y)=L2+L4+L6+L8-4*L5 The coordinates of L2 are (x, y+1), the coordinates of L4 are (x-1, y), the coordinates of L6 are (x+1, y), and the coordinates of L8 are (x, y-1). * indicates multiplication. (5b) For 64 infrared and 64 SAR bandpass directional images, the edge pixels of the image are kept unchanged, and the gradients of other pixels are calculated according to (5a) to obtain the gradient of the entire image.
7. The method according to claim 1, characterized in that In (7), the passband direction images of 64 infrared images and 64 SAR images are fused. According to the principle of taking the maximum absolute value of the gradient, the fusion coefficients of 64 passband directions are obtained, which are implemented as follows: (7a) A bandpass fusion coefficient is obtained by the following formula: Among them, F H (x, y) represents the pixel value of the bandpass fusion coefficient at (x, y), A H (x, y) is the pixel value of the infrared bandpass image at (x, y), B H (x, y) is the pixel value of the SAR bandpass image at (x, y), A' H (x, y) is the gradient value of the red band pass directional gradient image at (x, y), B' L (x, y) is the gradient value of the SAR bandpass directional gradient image at (x, y); (7b) Obtain 64 bandpass fusion coefficients in the same way as (7a).
Citation Information
Patent Citations
Spatial frequency-based multispectral image and full-color image fusion method under overall principle
CN108921809A
Multi-scale SAR and infrared image fusion method based on target enhancement
CN112734683A
