A Visible-Infrared Image Fusion Method Based on Least Squares Filtering

Abstract

This invention relates to a visible light infrared image fusion method based on least squares filtering, belonging to the field of image processing technology. It solves the problems of existing visible light infrared image fusion methods resulting in the loss of details with significant contrast but low pixel intensity, and blurred image edges. The method includes the following steps: obtaining a detail layer image after least squares filtering of a visible light grayscale image; obtaining the average value of the detail layer from the visible light least squares detail layer and the fused result detail layer; and fusing the average values ​​of the fused result background layer and the detail layer to obtain the final fused image. This invention preserves details with significant contrast but low pixel intensity in the image region; the combination of least squares filtering and guided filtering better maintains edge characteristics, allowing the fused image to obtain more effective information.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a visible light and infrared image fusion method based on least squares filtering. Background Technology

[0002] Infrared imaging is increasingly widely used in the imaging field because it can easily obtain the temperature information of a target and can operate day and night. However, infrared imaging also has its limitations; in conditions where temperature is not a factor, it is difficult for infrared imaging to distinguish the target from the background.

[0003] In contrast, visible light intensity images have more image details and texture information, but they are easily affected by external factors such as environment and weather. In cases of low contrast or cluttered backgrounds, the target information is not obvious.

[0004] In recent years, in order to fully utilize the advantages of infrared and visible light images, visible light and infrared image fusion technology has become an important branch in the field of multi-band image fusion. Visible light and infrared image fusion technology combines the effective information of visible light and infrared images, which is of great significance for human visual perception, target detection and recognition.

[0005] Existing visible and infrared image fusion methods include image fusion methods based on pixel saliency-guided filtering and linear fusion methods. The fused image lacks details with obvious contrast but low pixel intensity, and the image edges are relatively blurry. Summary of the Invention

[0006] Based on the above analysis, the present invention aims to provide a visible light infrared image fusion method based on least squares filtering, in order to solve the problems of missing details with obvious contrast but low pixel intensity and blurred image edges in the fused images produced by existing visible light infrared image fusion methods.

[0007] The objective of this invention is mainly achieved through the following technical solutions:

[0008] A visible-infrared image fusion method based on least-squares filtering includes the following steps:

[0009] Step 1: Obtain the detail layer image of the visible light grayscale image after least squares filtering;

[0010] Step 2: Calculate the high horizontal gradient and low vertical gradient map for the input visible light and infrared grayscale images;

[0011] Step 3: Compare the high horizontal gradient and low vertical gradient images of the visible light and infrared grayscale images to obtain the contrast significance image I. Statistical ;

[0012] Step 4: Obtain the guide image P that can be used to compare the light image and the infrared image based on the contrast saliency image. Vis and P IR ;

[0013] Step 5: Perform mean filtering on the input visible light and infrared grayscale images to obtain the detail layer and background layer;

[0014] Step 6: Based on the obtained guide image P, process the input visible light and infrared grayscale images. Vis and P IR Guided filtering with different filters and different blur coefficients was performed to obtain the detail layer weight coefficients Wd of visible light and infrared images. Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR ;

[0015] Step 7: Convert the detail layer weight coefficients Wd obtained in Step 5 into... Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR The detail layer of the fused image is obtained by weighting the visible light, infrared detail layer, and background layer. D and background layer FUSION B ;

[0016] Step 8: Obtain the average value of the least squares detail layer obtained in Step 1 and the fusion result detail layer obtained in Step 7;

[0017] Step 9: Fuse the background layer image obtained in Step 7 and the average value of the detail layer obtained in Step 8 to obtain the final fused image.

[0018] Further, in step 1, the least squares filter is first used to obtain the visible light grayscale image, followed by the image VIS after filtering with the least squares filter. WLS Then, the detail layer image (VIS) of the visible light grayscale image after least-squares filtering is obtained. WLS-D .

[0019] Furthermore, in step 2, the gradient map in the x-direction is subtracted from the gradient map in the y-direction to obtain the high horizontal gradient and low vertical gradient map of the visible light and infrared grayscale images, respectively.

[0020] Furthermore, in step 2, the Sobel operator is used to obtain the gradient map.

[0021] Further, in step 3, for each pixel of the contrast saliency image, the maximum value is taken from the visible light and infrared light maps, which are both high-level horizontal gradients and low-level vertical gradients, to obtain the contrast saliency image I. Statistical .

[0022] Furthermore, in step 4, the guide map image P of the visible light image and the infrared image... Vis and P IR The method for obtaining the value is as follows: compare the image with the high horizontal gradient and low vertical gradient images of visible light and infrared light respectively. The position guidance map value of pixels with the same value is 1, and the position guidance map value of pixels with different values ​​is 0.

[0023] Further, in step 5, mean filtering is applied to the input visible light and infrared grayscale images to obtain their detail layer and background layer, respectively. The background layer has a high degree of smoothness, so a 31×31 mean filter is used to obtain the background layer, and the detail layer is calculated based on the obtained background layer.

[0024] Vis mean-d =Vis-Vis mean-b (6)

[0025] IR mean-d =IR-IR mean-b (7)

[0026] Among them, Vis mean-d IR mean-d These are the detail layers of the mean-filtered visible light and infrared grayscale images, respectively. Vis represents the visible light grayscale image, and IR infrared grayscale image. mean-b IR mean-b These are the background layers of the visible light and infrared grayscale images after mean filtering, respectively.

[0027] Furthermore, in step 6, the weight coefficients Wd of the detail layer of the visible light image are obtained using a guided filtering function. Vis The process is as shown in formula (8); the weight coefficient Wb of the background layer of the visible light image is obtained using the guided filtering function. Vis The process is shown in formula (9):

[0028] Wd Vis =guiderfiler(Vis, P) Vis , k d eps d (8)

[0029] Wb Vis =guiderfiler(Vis, P) Vis , k b eps b (9)

[0030] The weight coefficients Wd of the detail layer of the infrared image are obtained using the guided filtering function. IR The process is as shown in formula (10); the weight coefficient Wb of the background layer of the infrared image is obtained using the guided filtering function. IR The process is shown in formula (11):

[0031] Wd lR =guiderfiler(IR, P) IR k d eps d (10)

[0032] Wb IR =guiderfiler(IR, P) IR k b eps b (11)

[0033] In the formula, guiderfiler() represents the guided filter function; Vis is the visible light grayscale image; IR is the infrared grayscale image; P Vis and P IR These are the guide maps for the visible light and infrared images obtained in step 3, respectively; k b and k d For filters of different sizes; eps b and eps d These represent fuzzy coefficients of different sizes.

[0034] Furthermore, in step 7, during the addition, the weight coefficients need to be normalized to obtain the detail layer FUSION of the fused image. D and background layer FUSION B .

[0035] Furthermore, in step 8, the fusion method is as follows:

[0036]

[0037] The present invention can achieve the following beneficial effects:

[0038] (1) The visible-infrared image fusion method based on least squares filtering of the present invention introduces the concept of image contrast saliency, so that details with obvious contrast but low pixel intensity values ​​in the image region can be preserved; the present invention combines least squares method and guided filtering to better preserve the characteristics of edges, so that the fused image can obtain more effective information. The present invention has important significance for human visual perception, target detection and recognition.

[0039] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0040] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0041] Figure 1 This is a flowchart of the visible light and infrared image fusion method according to an embodiment of the present invention;

[0042] Figure 2 This is a template diagram of the Sobel operator in an embodiment of the present invention;

[0043] Figure 3 This is the original grayscale image of visible light according to an embodiment of the present invention;

[0044] Figure 4 This is the original infrared grayscale image of an embodiment of the present invention;

[0045] Figure 5 This is a visible light image with a high horizontal gradient and a low vertical gradient, as described in an embodiment of the present invention.

[0046] Figure 6 This is an infrared image with high horizontal gradient and low vertical gradient according to an embodiment of the present invention;

[0047] Figure 7 This is a contrast saliency image according to an embodiment of the present invention;

[0048] Figure 8 The image is the result of fusion using the image fusion method of this embodiment of the invention;

[0049] Figure 9 The image is a fused image obtained by using existing pixel-saliency-based guided filtering image fusion methods;

[0050] Figure 10 The image is a linearly fused image obtained using existing technology. Detailed Implementation

[0051] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which constitute a part of the present invention and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0052] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the term "connected" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0053] Throughout the text, the terms “top,” “bottom,” “above,” “below,” and “on top” refer to the relative positions of components of the device, such as the relative positions of the top and bottom substrates within the device. It is understood that the device is multifunctional and independent of its spatial orientation.

[0054] One embodiment of the present invention, such as Figures 1 to 10 As shown, a visible light and infrared image fusion method based on least squares filtering is disclosed. Visible light and infrared light simultaneously pass through a spherical primary mirror to form a visible light image and an infrared image, including the following steps:

[0055] Step 1: Obtain the detail layer image of the visible light grayscale image after least-squares filtering:

[0056] First, a least-squares filter is used to obtain the visible light grayscale image. Then, the image (VIS) is filtered using the least-squares filter. WLS Then, the detail layer image after least-squares filtering of the visible light grayscale image is obtained:

[0057] VIS WLS-D =Vis-Vis WLS (1)

[0058] Among them, VIS WLS-D VIS represents the detail layer after the least squares method of the visible light grayscale image.

[0059] Step 2: Calculate the high horizontal gradient and low vertical gradient maps for the input visible light and infrared grayscale images:

[0060] In this process, visible light grayscale images and infrared grayscale images are simultaneously imaged and acquired through a visible light and infrared composite system based on spherical concentric mirrors to ensure the consistency of the acquired visible light and infrared images.

[0061] By subtracting the gradient map in the y-direction from the gradient map in the x-direction, we can obtain the high horizontal gradient and low vertical gradient maps of the visible light and infrared grayscale images respectively (e.g., Figure 5 , 6 (As shown). The Sobel operator is used to obtain the gradient map; the Sobel operator template is shown below. Figure 2As shown. The input visible light and infrared images are both grayscale images.

[0062] gradient Vis =gradX Vis -gradY Vis (2)

[0063] gradient IR =gradX IR -gradY IR (3)

[0064] Among them, gradien t Vis This represents a high horizontal gradient and low vertical gradient map of a visible light grayscale image; gradient IR A map representing the high horizontal gradient and low vertical gradient of an infrared grayscale image; gradX Vis grad Y Vis These represent the gradient maps in the X and Y directions of a visible light grayscale image, respectively; gradX IR grad Y IR These represent the gradient maps in the X and Y directions of the infrared grayscale image, respectively.

[0065] Step 3: Compare the high horizontal gradient and low vertical gradient images of the visible light and infrared grayscale images to obtain the contrast significance image I. Statistical :

[0066] For each pixel in the contrast saliency image, the maximum value is taken from the two high-horizontal-gradient and low-vertical-gradient matrixes of visible light and infrared light, resulting in the contrast saliency image I. Statistical As shown in Figure 7.

[0067]

[0068] Among them, I Statistical，i，j This represents the contrast saliency value at pixel position (i, j), gradient Vis，i，j The gradient represents the value of a pixel at position (i, j) that has a high horizontal gradient and a low vertical gradient in a visible grayscale image. IR，i，j This represents the value of a pixel at position (i, j) in an infrared grayscale image with a high horizontal gradient and a low vertical gradient.

[0069] Step 4: Obtain the guide map image P from the visible light image and the infrared image based on the contrast saliency image. Vis and P IR :

[0070] The image with significant contrast values ​​is compared with the high horizontal gradient and low vertical gradient images of visible light and infrared light, respectively. The position guidance map value of pixels with the same value is 1, and the position guidance map value of pixels with different values ​​is 0.

[0071]

[0072]

[0073] Among them, P Vis，i，j P represents the visible light guide map value at pixel position (i, j). IR，i，j The value of the infrared guide map is when the pixel position is (i, i).

[0074] Step 5: Perform mean filtering on the input visible light and infrared grayscale images to obtain the detail layer and background layer:

[0075] The input visible light and infrared grayscale images are subjected to mean filtering to obtain their detail layer and background layer, respectively. The background layer has a high degree of smoothness, so a 31×31 mean filter is used to obtain the background layer. The detail layer is then calculated based on the obtained background layer.

[0076] Vis mean-d =Vis-Vis mean-b (7)

[0077] IR mean-d =IR-IR mean-b (8)

[0078] Among them, Vis mean-d IR mean-d These are the detail layers of the mean-filtered visible light and infrared grayscale images, respectively. Vis represents the visible light grayscale image, and IR represents the infrared grayscale image. mean-b IR mean-b These are the background layers of the visible light and infrared grayscale images after mean filtering, respectively.

[0079] Step 6: Based on the obtained guide image P, process the input visible light and infrared grayscale images. Vis and P IR Guided filtering with different filters and different blur coefficients was performed to obtain the detail layer weight coefficients Wd of visible light and infrared images. Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR :

[0080] The weight coefficients Wd of the detail layer of a visible light image are obtained using a guided filter function. VisThe process can be expressed as formula (9); the weight coefficient Wb of the background layer of the visible light image is obtained using the guided filtering function. Vis The process can be expressed as formula (10):

[0081] Wd Vis =guiderfiler(Vis, P) Vis , k d eps d (9)

[0082] Wb Vis =guiderfiler(Vis, P) Vis , k b eps b (10)

[0083] The weight coefficients Wd of the detail layer of the infrared image are obtained using the guided filtering function. IR The process can be expressed as formula (11); the weight coefficient Wb of the background layer of the infrared image is obtained using the guided filtering function. IR The process can be expressed as formula (12):

[0084] Wd IR =guiderfiler(IR, P) IR , k d eps d (11)

[0085] Wb IR =guiderfiler(IR, P) IR , k b eps b (12)

[0086] In the formula, guiderfiler() represents the guided filter function; Vis is the visible light grayscale image; IR is the infrared grayscale image; P Vis and P IR These are the guide maps for the visible light and infrared images obtained in step 3; a larger filter k is used when calculating the background layer. b and a large fuzzy coefficient eps b A smaller filter k is used when calculating the detail layer. d and a smaller fuzzy coefficient eps d Typically, the filter k used when calculating the background layer is... b It is the filter k used when calculating the detail layer. d Twice the blur coefficient eps when calculating the background layer. b With the calculation of the fuzzy coefficient eps of the detail layer d They differ by an order of magnitude.

[0087] Furthermore, the principle of guided filtering is as follows:

[0088]

[0089] Among them, I i For the guide map pixels, q i For the output image pixels, q can be viewed as a local linear transformation of the guide graph I, where q is the output value, and k is the midpoint of the localized window, thus belonging to window ω. k Each pixel can be represented by the corresponding pixel in the guide map via (a) k , b k The coefficients are transformed and calculated.

[0090] The loss function within the filtering window can be written as:

[0091]

[0092] Among them, p i For the input image pixels, a regularization parameter ∈ is introduced to avoid ak from being too large.

[0093] Solving the above equations yields:

[0094]

[0095] The underlined value represents the average of all values ​​calculated within the window.

[0096]

[0097] in It is the variance of the window:

[0098]

[0099] For example, in this embodiment, the filter used to calculate the weight map of the background layer has a window radius of 8 and a blur coefficient of 0.3*0.3. When calculating the weight map of the detail layer, the filter has a window radius of 4 and a blur coefficient of 0.03*0.03.

[0100] Step 7: Convert the detail layer weight coefficients Wd obtained in Step 6 into... Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR The detail layer of the fused image is obtained by weighting the detail layer of the visible light and infrared layers and the background layer. D and background layer FUSIO N B .

[0101] During the addition process, the weight coefficients also need to be normalized to obtain the detail layer FUSION of the fused image. D and background layer FUSIO N B As shown in formulas (13) and (14):

[0102] FUSION D =Vis mean-d *Wd Vis / (Wd Vis +Wd IR )+IR mean-d ×Wd IR / (Wd Vis +Wd IR (13)

[0103] FUSION B =Vis mean-b *Wb Vis / (Wb Vis +Wb IR )+IR mean-b *Wb IR / (Wb Vis +Wb IR (14)

[0104] Step 8: Calculate the average value of the visible light least squares detail layer obtained in Step 1 and the fused detail layer obtained in Step 7. The fusion method is shown in the following formula:

[0105]

[0106] Step 9: Fuse the background layer obtained in Step 7 with the average value of the detail layer obtained in Step 8 to obtain the final fused image. The fusion method is shown in the following formula:

[0107] FUSION = DETAIL D +FUSION B (16)

[0108] As shown in the figure, the image fused with the image fusion method based on pixel saliency guided filtering (e.g.) Figure 9 (as shown) and linearly fused images (such as) Figure 10 Compared to the image fusion method shown in the figure, the image fusion method based on contrast saliency-guided filtering in this embodiment produces a fused image (such as...). Figure 8 As shown, it retains details with obvious contrast but low pixel intensity values, making the image edges clearer and allowing the fused image to obtain more effective information.

[0109] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A visible-infrared image fusion method based on least-squares filtering, characterized in that, Includes the following steps: Step 1: Obtain the detail layer image of the visible light grayscale image after least squares filtering; Step 2: Calculate the high horizontal gradient and low vertical gradient map for the input visible light and infrared grayscale images; Step 3: Compare the high horizontal gradient and low vertical gradient images of the visible light and infrared grayscale images to obtain the contrast significance image I. Statistical ; Step 4: Obtain the guide image P that can be used to compare the light image and the infrared image based on the contrast saliency image. Vis and P IR ; Step 5: Perform mean filtering on the input visible light and infrared grayscale images to obtain the detail layer and background layer; Step 6: Based on the obtained guide image P, process the input visible light and infrared grayscale images. Vis and P IR Guided filtering with different filters and different blur coefficients was performed to obtain the detail layer weight coefficients Wd of visible light and infrared images. Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR ; Step 7: Convert the detail layer weight coefficients Wd obtained in Step 5 into... Vis Wd IR and the weight coefficient Wb of the background layer Vis Wb IR The detail layer of the fused image is obtained by weighting the visible light, infrared detail layer, and background layer. D and background layer FUSION B ; Step 8: Obtain the average value of the least squares detail layer obtained in Step 1 and the fusion result detail layer obtained in Step 7; Step 9: Fuse the average values ​​of the background layer obtained in Step 7 and the detail layer obtained in Step 8 to obtain the final fused image.

2. The visible light and infrared image fusion method based on least squares filtering according to claim 1, characterized in that, In step 1, the least squares filter is first used to obtain the visible light grayscale image, followed by the image VIS after filtering with the least squares filter. WLS Then, the detail layer image (VIS) of the visible light grayscale image after least-squares filtering is obtained. WLS-D .

3. The visible light and infrared image fusion method based on least squares filtering according to claim 1, characterized in that, In step 2, the gradient map in the x-direction is subtracted from the gradient map in the y-direction to obtain the high horizontal gradient and low vertical gradient map of the visible light and infrared grayscale images, respectively.

4. The visible light infrared image fusion method based on least squares filtering according to claim 3, characterized in that, In step 2, the Sobel operator is used to obtain the gradient map.

5. The visible light infrared image fusion method based on least squares filtering according to claim 2, characterized in that, In step 3, for each pixel of the contrast saliency image, the maximum value is taken from the two high-horizontal-gradient and low-vertical-gradient matrixes of visible light and infrared light, to obtain the contrast saliency image I. Statistical .

6. The visible light and infrared image fusion method based on least squares filtering according to claim 5, characterized in that, In step 4, the guide image P of the visible light image and the infrared image. Vis and P IR The method for obtaining the value is as follows: compare the image with the high horizontal gradient and low vertical gradient images of visible light and infrared light respectively. The position guidance map value of pixels with the same value is 1, and the position guidance map value of pixels with different values ​​is 0.

7. The visible light and infrared image fusion method based on least squares filtering according to claim 6, characterized in that, In step 5, mean filtering is applied to the input visible light and infrared grayscale images to obtain their detail layer and background layer, respectively. The background layer has a high degree of smoothness, so a 31×31 mean filter is used to obtain the background layer. The detail layer is then calculated based on the obtained background layer. Show mean-d =Show-Show mean-b (6) IS mean-d =IR-IR mean-b (7) Among them, Vis mean-d IR mean-d These are the detail layers of the mean-filtered visible light and infrared grayscale images, respectively. Vis represents the visible light grayscale image and the IR infrared grayscale image. mean-b IR mean-b These are the background layers of the visible light and infrared grayscale images after mean filtering, respectively.

8. The visible light and infrared image fusion method based on least squares filtering according to claim 7, characterized in that, In step 6, the weight coefficients Wd of the detail layer of the visible light image are obtained using a guided filter function. Vis The process is as shown in formula (8); the weight coefficient Wb of the background layer of the visible light image is obtained using the guided filtering function. Vis The process is shown in formula (9): Wd Vis =guides files(View,P Vis ,k d ,eps d ) (8) Wb Vis =guidesfiles(Show,P Vis ,k b ,eps b ) (9) The weight coefficients Wd of the detail layer of the infrared image are obtained using the guided filtering function. IR The process is as shown in formula (10); the weight coefficient Wb of the background layer of the infrared image is obtained using the guided filtering function. IR The process is shown in formula (11): Wd IR =guide files(IR,P IR ,k d ,eps d ) (10) Wb IR =guide files(IR,P IR ,k b ,eps b ) (11) In the formula, guiderfiler() represents the guided filter function; Vis is the visible light grayscale image; IR is the infrared grayscale image; P Vis and P IR These are the guide maps for the visible light and infrared images obtained in step 3, respectively; k b and k d For filters of different sizes; eps b and eps d These represent fuzzy coefficients of different sizes.

9. The visible light and infrared image fusion method based on least squares filtering according to claim 8, characterized in that, In step 7, during the addition, the weight coefficients also need to be normalized to obtain the detail layer FUSION of the fused image. D and background layer FUSION B .

10. The visible light and infrared image fusion method based on least squares filtering according to claim 9, characterized in that, In step 8, the fusion method is as follows:

Images