Infrared and visible image fusion method using multi-scale bessel filter

Abstract

The application discloses an infrared and visible light image fusion method using a multi-scale Bezier filter, and belongs to the technical field of image fusion, and comprises the following steps: S1: extracting bright and dark features of infrared and visible light images based on a multi-scale Bezier filter; S2: fusing bright and dark feature maps; S3: fusing basic images; and S4: fusing infrared and visible light images. The application designs a multi-scale Bezier filter, and performs multi-scale filtering on infrared images and visible light images by using the Bezier filter, so that significant bright features and dark features can be efficiently extracted from the infrared and visible light images, and a fusion image with rich information is generated. The method is not only superior to an advanced transform domain image fusion method, but also superior to a latest image fusion method based on deep learning, and can be widely applied to various security monitoring systems.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and more specifically to an infrared and visible light image fusion method utilizing a multi-scale Bezel filter. Background Technology

[0002] In recent years, scholars have proposed various methods for fusing infrared and visible light images. These methods can generally be divided into two categories: spatial domain methods and transform domain methods. Spatial domain methods first segment the source image into multiple regions, and then combine the salient regions to obtain a fused image. However, this method is difficult to obtain accurate segmentation results, so it often produces unsatisfactory fused images. Transform domain methods have been the most popular image fusion methods over the past two decades, such as pyramid-based, wavelet-based, and sparse representation-based methods. These methods first transform the infrared and visible light images to a certain domain, then fuse the salient features of the two images in the transform domain, and finally inversely transform the fused transform domain features back to the spatial domain to generate a fused image. The fused images obtained by this type of method are usually more conducive to intuitive understanding by the human visual system. However, the fused images obtained by this type of method often exhibit blurring effects or lose a large amount of visible light image information, thus reducing the visual quality of the fused image.

[0003] Recently, several scholars have proposed image fusion methods based on deep learning (especially convolutional neural networks, CNNs). These methods typically first use CNNs to extract features from each image, then fuse the feature maps of multiple images and reconstruct the final fused image. Although CNN-based methods have achieved great success in image fusion, the fused images generated by these methods often suffer from low contrast or other deficiencies. In addition to the above methods, some scholars have proposed an infrared and visible light image fusion method based on quadtrees and Bézier interpolation. This method first extracts significant bright object features from the infrared image and then directly superimposes them onto the visible light image to generate the fused image. However, images contain both bright and dark features, so the fused image obtained by this method cannot reflect the dark features of the infrared image, and when the corresponding area of ​​the visible light image is overexposed, the fused image also fails to display the bright features of the infrared image. Given the excellent feature extraction capability of the Bézier interpolation operator, this invention designs a multi-scale Bézier filter with superior performance and further proposes an infrared and visible light image fusion method utilizing this multi-scale Bézier filter. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to make the fused image not only retain the visual features in the visible light image that are easier for the human visual system to understand, but also effectively integrate the bright and dark features in the infrared image. This invention provides a method for fusing infrared and visible light images using a multi-scale Bezel filter. The method first extracts bright and dark feature maps of different scales from the infrared and visible light images respectively, and generates two base images. Then, it uses two different strategies to fuse the significant (bright and dark) features in the infrared and visible light images with the base images respectively. Finally, it superimposes the fused bright and dark feature maps with the base images to generate a fused image of infrared and visible light images.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps: S1: Bright and Dark Feature Extraction from Infrared and Visible Images Based on Multi-Scale Bézier Filters The infrared image and the corresponding visible light image are input after grayscale processing. The infrared image and the visible light image are filtered stepwise using Bezier filters of different scales. At the same time, the bright and dark feature maps at the corresponding scales are extracted from the filtered infrared image and the visible light image respectively. The finally filtered infrared image and the visible light image are used as the base images of the input infrared image and the visible light image respectively. S2: Fusion of bright and dark feature maps The bright and dark feature maps of infrared and visible light images at each scale are fused using a pixel-wise maximum value strategy. The scale correlation coefficient is used to enhance the initial fused bright and dark feature maps at each scale. The enhanced multi-scale bright and dark features are then superimposed to obtain a fused bright feature map and a fused dark feature map. S3: Basic Image Fusion The average image of the two base images is obtained, and the residual bright and dark features in the two base images other than the average image are fused using the pixel-wise maximum value strategy. Finally, the average image and the fused residual bright and dark feature images are merged to obtain the fused base image. S4: Infrared and Visible Image Fusion The fused bright feature map, dark feature map, and fused base image are superimposed to generate a fused image of infrared and visible light images.

[0006] Furthermore, the specific process of step S1 is as follows: S11: First, adjust the size of the original image to (3km+1)×(3lm+1), then decompose the input image into k×l image blocks of size (3m+1)×(3m+1), where k and l are positive integers, and m is the radius of the local region where the minimum and maximum gray values ​​are searched. The original image is the input infrared image and visible light image. S12: Perform Bezier filtering on each image block and adjust the filtered image to the original image size; S13: Extracting brightness feature maps at multiple scales from the filtered infrared image Dark feature map And extract brightness feature maps at multiple scales from the filtered visible light image. Dark feature map Simultaneously, the infrared and visible light images obtained from the last filtering are used as the base images for the infrared and visible light images, respectively. and .

[0007] Furthermore, in step S1, the Bezier filter is a local interpolation operator performed on an image patch, which can be expressed as:

[0008] In the formula, The location of the interpolation point is indicated by an interpolation ratio between 0 and 1. M is a constant interpolation coefficient matrix, which is related to the location; P is a gray value matrix of 4×4 control points uniformly distributed on the image block to be filtered; and Q is the filtered image block.

[0009] Furthermore, in step S12, during Bezel filtering, the image patch is first reconstructed using a 4×4 control point local minimum gray value matrix to remove bright spots, and then the image patch is reconstructed using an updated 4×4 control point local maximum gray value matrix to fill black holes.

[0010] Furthermore, in step S12, the Bessel filtering function using the local minimum gray value matrix is ​​defined as follows: The Bézier filter function using the local maximum gray value matrix is ​​defined as follows: The process of smoothing image patches using a single-scale Bezel filter can be represented as:

[0011] In the formula, B and B f These represent the original image block and the filtered image block, respectively. Compare the original image block B with the filtered image block B f The salient features of the input image patch are obtained by subtraction, where (BB) f The positive values ​​of ) correspond to bright features in the image, and the negative values ​​correspond to dark features in the image; Then, the Bézier filter is extended to multiple scales by iteratively applying the Bézier filter, as shown in the following equation:

[0012] In the formula, i represents the current scale number of the Bezier filter, which increases from 1 to n; B i B0 represents the filtered image block at the i-th scale, and B0 represents the original image block B. Simultaneously, bright and dark features at different scales are extracted from the difference image of every two consecutive filtered image blocks, as shown in the following formula:

[0013]

[0014] In the formula, F i,b and F i,d These represent the bright and dark feature maps extracted from image block B at the i-th scale, respectively.

[0015] Furthermore, in step S13, the image patch obtained from the last filtering is used as the base image of the original image patch, as shown in the following formula:

[0016] In the formula, B base This represents the base image of the original image block B.

[0017] Furthermore, in step S2, a pixel-by-pixel maximum value strategy is used to fuse the bright and dark feature maps of the infrared image and the visible light image: .

[0018] Furthermore, in step S2, the scale correlation coefficient is used to enhance the bright and dark feature maps at each scale of the initial fusion. That is, each extracted feature map at each scale is multiplied by a scale-related weight coefficient to enhance the fused bright and dark feature maps. Then, the enhanced multi-scale bright features and multi-scale dark features are superimposed to obtain a fused bright feature map and a fused dark feature map.

[0019] In the formula, w i w represents the weight coefficients of the bright and dark feature maps at the i-th scale. i =ζ (n-i) ζ is a constant not less than 1, and n is the number of scales.

[0020] Furthermore, the specific process of step S3 is as follows: S31: Decompose the base images of the infrared image and the visible light image into three parts respectively:

[0021] In the formula, and They represent and The average image; and They represent and Residual bright features in the middle; and They represent and Residual dark features in the middle; S32: Will and The three parts obtained from the decomposition are then fused together:

[0022] In the formula, , and These represent the fused average image, the residual bright feature map, and the residual dark feature map, respectively. S33: Overlay , and Determine the base image for fusion: .

[0023] Furthermore, in step S4, the base image, bright feature map, and dark feature map are superimposed and fused to generate a fused image of the infrared and visible light images: .

[0024] Compared with the prior art, the present invention has the following advantages: a multi-scale Bezier filter is designed and the Bezier filter is used to perform multi-scale filtering on infrared and visible light images. It can effectively extract significant dark and bright features from infrared and visible light images and generate an information-rich fused image. It is not only superior to advanced transform domain image fusion methods, but also superior to the latest deep learning-based image fusion methods. It can be widely used in various security monitoring systems. Attached Figure Description

[0025] Figure 1 This is a flowchart of the infrared and visible light image fusion method using a multi-scale Bezel filter in Embodiment 1 of the present invention. Figure 2 This is a demonstration example of the multi-scale Bezier filter in an infrared image according to Embodiment 1 of the present invention; Figure 3 (a) is an infrared image selected from the TNO dataset in Embodiment 2 of the present invention; Figure 3(b) is a visible light image selected from the TNO dataset in Embodiment 2 of the present invention (and...). Figure 3 (a) is a pair); Figure 3 (c) is the image fusion method based on guided filtering in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 3 (d) is the image fusion method based on hybrid multi-scale decomposition in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 3 (e) is the image fusion method of Laplacian pyramid and sparse representation in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 3 (f) is the image fusion method based on adversarial neural networks in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 3 (g) is the general image fusion method based on deep learning in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 3 (h) is the method of the present invention in Embodiment 2 of the present invention. Figure 3 (a) and Figure 3 (b) is the fusion result diagram; Figure 4 (a) is an infrared image selected from the TNO dataset in Embodiment 2 of the present invention; Figure 4 (b) is a visible light image selected from the TNO dataset in Embodiment 2 of the present invention (and...). Figure 4 (a) is a pair); Figure 4 (c) is the image fusion method based on guided filtering in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram; Figure 4 (d) is the image fusion method based on hybrid multi-scale decomposition in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram; Figure 4 (e) is the image fusion method of Laplacian pyramid and sparse representation in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram; Figure 4 (f) is the image fusion method based on adversarial neural networks in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram; Figure 4 (g) is the general image fusion method based on deep learning in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram; Figure 4 (h) is the method of the present invention in Embodiment 2 of the present invention. Figure 4 (a) and Figure 4 (b) is the fusion result diagram. Detailed Implementation

[0026] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0027] Example 1 like Figure 1 As shown in this embodiment, an infrared and visible light image fusion method utilizing a multi-scale Bezel filter is provided. This method mainly includes four steps: First, infrared and visible light images are progressively filtered using Bezier filters of different scales. Simultaneously, bright and dark feature maps at corresponding scales are extracted from the filtered infrared and visible light images, and the final filtered infrared and visible light images are used as their base images. Next, a pixel-wise maximum value strategy is used to fuse the bright and dark feature maps of the infrared and visible light images at each scale. The scale correlation coefficient is then used to enhance the initially fused bright and dark feature maps at each scale. The enhanced multi-scale bright and dark features are then superimposed to obtain a fused bright feature map and a fused dark feature map. Subsequently, the average image of the two base images is calculated, and the average of the two base images is taken to obtain the fused average image. Simultaneously, a pixel-wise maximum value strategy is used to fuse the residual bright and dark features from the two base images, excluding the average image. Finally, the fused average image, residual bright and dark feature maps are superimposed to obtain the fused base image. Finally, the fused bright and dark feature maps and the base image are superimposed to obtain the fused image generated by this method.

[0028] The multi-scale Bezel filter in this invention will be described below.

[0029] Generally speaking, a natural image contains both bright and dark features, such as Figure 1Bright pedestrians in mid-infrared images and a darker corner of a roof in visible light images. Smoothing an image and subtracting it from the original image is a common method for extracting bright and dark features. Ideally, the smoothed image should smooth out bright spots and fill in black holes in the original image, thus effectively extracting bright and dark features by subtracting the original and filtered images.

[0030] In this embodiment, an excellent Bézier filter is designed. This filter not only solves the problem that existing Bézier filters cannot be used to extract dark features in images, but also can be extended to extract image features at multiple scales. The specific construction method is as follows: The Bezier filter is essentially a local interpolation operator performed on an image patch, which can be represented as: (1) In the formula, The location of the interpolation point is indicated by an interpolation ratio between 0 and 1. M is a constant interpolation coefficient matrix, which is related to the location; P is a gray value matrix of 4×4 control points uniformly distributed on the image block to be filtered; and Q is the filtered image block.

[0031] Specifically , , , .

[0032] In this embodiment, image patches of different sizes (i.e., (3m+1)×(3m+1)) are reconstructed using 4×4 control points. Here, m determines the smoothing scale of the Bezier filter and satisfies m=2i-1, where i represents the current scale of the Bezier filter. Therefore, the Bezier filter can interpolate the gray value matrix of 4×4 control points into image patches of size (6i-2)×(6i-2), and u=v=[0,1 / (6i-1),...,(6i-2) / (6i-1),1]. To effectively smooth the input image, this method first reconstructs the image patch using the local minimum gray value matrix of 4×4 control points to remove bright spots, and then reconstructs the image patch using the updated local maximum gray value matrix of 4×4 control points to fill black holes. The reconstruction of local minima and local maxima using Bezier interpolation in sequence constitutes a single-scale Bezier filter. Furthermore, in this method, the radius of the local region searching for the minimum and maximum gray values ​​is set to m.

[0033] In this embodiment, the Bezier filter function that utilizes the local minimum gray value is defined as... The Bessel filter function that utilizes the local maximum gray value is defined as follows: The process of smoothing an image patch using a single-scale Bezel filter can be represented as: (2) In the formula, B and B f These represent the original image block and the filtered image block, respectively.

[0034] Then, the original image block B and the filtered image block B are combined. f The salient features of the input image patch can be obtained by subtracting the features, where (BB) f The positive values ​​of ) correspond to the bright features in the image, and the negative values ​​correspond to the dark features in the image. Figure 2 An example of extracting infrared image features using a Bezier filter is presented. The demonstration results show that the Bezier filter designed in this embodiment can effectively extract bright and dark features from the original image.

[0035] Subsequently, the Bézier filter is extended to multiple scales by iteratively applying the Bézier filter, as shown in Equation (3); at the same time, bright feature maps and dark feature maps of different scales can be extracted from the difference image of every two consecutive filtered image blocks, as shown in Equations (4) and (5).

[0036] (3) In the formula, i represents the number of Bezel filter scales, increasing from 1 to n. i Let B0 represent the filtered image block at the i-th scale. Specifically, B0 represents the original image block B. (4) (5) In the formula, F i,b and F i,d These represent the bright and dark feature maps extracted from image block B at the i-th scale, respectively.

[0037] Finally, the filtered image patch at the nth scale (i.e., the last scale) is used as the base image of the original image patch, as shown in formula (6): (6) In the formula, B base This represents the base image of the original image block B.

[0038] The infrared and visible light image fusion method using a multi-scale Bezel filter in this embodiment will be further explained below.

[0039] The Bessel filter of this invention was described above using the filtering of a single image block as an example. The method for filtering the entire image can be summarized as follows: First, the size of the original image is adjusted to (3m+1)×(3m+1). Then, the input image is decomposed into k×l image blocks of size (3m+1)×(3m+1). Each image block is then subjected to Bessel filtering. Finally, the filtered image is adjusted to the size of the original image. Furthermore, the brightness and darkness feature maps of the original image can be extracted by subtracting the original image from the filtered image. Based on the above process, brightness feature maps at multiple scales can be extracted from the infrared image. ) and dark feature map ( ), and extracting multi-scale brightness feature maps from visible light images ( ) and dark feature map ( Simultaneously generating two base images: an infrared image and a visible light image (i.e., and Then, the high-frequency feature maps and low-frequency base images of the infrared and visible light images are fused according to two different fusion rules.

[0040] For high-frequency bright and dark features, this embodiment directly adopts a pixel-by-pixel maximum value strategy to fuse the bright and dark feature maps of infrared and visible light images: (7) Similar to other feature extractors, the Bessel filter proposed in this invention cannot extract all bright and dark detail features from the original image. Therefore, this invention multiplies the extracted feature maps at each scale by a scale-related weight to enhance the fused bright and dark feature maps. Subsequently, the enhanced multi-scale bright features and multi-scale dark features are superimposed to obtain a fused bright feature map and a fused dark feature map. (8) In the formula, w i Let w represent the weight coefficients of the bright and dark feature maps at the i-th scale, where n is the number of scales. Generally, lower-scale feature maps usually correspond to richer texture details and therefore should be assigned larger weights. In this embodiment, we set w... i =ζ (n-i) , where ζ is a constant not less than 1.

[0041] For low-frequency base images, residual bright and dark features may still exist. Therefore, this embodiment further decomposes the base images of infrared and visible light images into three parts respectively: (9) In the formula, and They represent and The average image; and They represent and Residual bright features in the middle; and They represent and The remaining dark features.

[0042] Then, the present invention will and The three parts obtained from the decomposition are then fused together: (10) In the formula, , and These represent the fused average image, residual bright feature map, and residual dark feature map, respectively. Then, they are overlaid. , and The base image for fusion can be obtained: (11) Finally, a fused image of the infrared and visible light images is generated by overlaying and fusing the base image, bright feature map, and dark feature map: (12) Example 2 The method in Example 1 involves two unknown parameters: the number of scales, n, and the cardinality of the enhancement coefficients, ζ. In this example, we use a grid search method to find the optimal set of n (from 1 to 6, with a step size of 1) and ζ (from 1 to 1.2, with a step size of 0.01) that maximizes the average NSSIM value of all fused images. Based on the above method, the optimal set of n and ζ is found to be 6 and 1.05, respectively.

[0043] To verify the effectiveness of the infrared and visible light image fusion method proposed in this invention, its performance was compared with five advanced image fusion methods on a commonly used infrared and visible light image dataset. Detailed experimental setup, results, and conclusions are as follows: 1. Experimental setup The experimental setup in this embodiment is as follows: First, 20 commonly used pairs of infrared and visible light images from the TNO dataset are selected as the test set; second, five advanced infrared and visible light image fusion methods are selected as comparison methods, including guided filtering-based image fusion (GFF), hybrid multiscale decomposition-based image fusion (HMSD), Laplacian pyramid and sparse representation-based image fusion (LPSR), adversarial neural network-based image fusion (FusionGAN), and deep learning-based general image fusion (U2Fusion).

[0044] Then, this embodiment evaluated the six image fusion methods using both qualitative and quantitative methods. Specifically, the qualitative evaluation method relied on the human eye to compare the visual effects of the fused images generated by different image fusion methods. In the quantitative evaluation method, we selected six quantitative indicators to quantify the objective quality of the fused images from different methods. These six quantitative indicators are entropy (E), spatial frequency (SF), mean absolute gradient (AG), standard deviation (STD), visual information fidelity (VIFF), and a novel structural similarity measure (NSSIM). The higher the value of these six indicators, the better the performance of the corresponding image fusion method.

[0045] 2. Qualitative assessment results Figure 3 and Figure 4 Two comparative examples of six image fusion methods are provided. Figure 3 These are infrared and visible light images acquired during daylight conditions. The optimal fused image should primarily combine significant bright features at different scales in the infrared image with rich bright and dark features in the visible light image. Figure 3 It can be seen that GFF failed to integrate the brightest bunker areas from the infrared images into its fused image. HMSD and LPSR did not incorporate most of the bright features from the infrared images into their fused images (see...). Figure 3 (d) and 3(e) are the bunker areas within the rectangles. FusionGAN's fused image loses much of the texture information from the visible light image, resulting in the worst visual effect among all fused images. U2Fusion effectively integrates significant texture features from both infrared and visible light images into its fused image, but its fused image has lower contrast compared to GFF, HMSD, LPSR, and the method of this invention. Finally, the fused image of the method of this invention effectively integrates bright features at different scales in the infrared image and rich details in the visible light image, exhibiting the best visual effect among all fused images.

[0046] exist Figure 4In this case, the sky region in the visible light image is overexposed. Ideally, the fusion of infrared and visible light images would integrate the salient bright features of the infrared image (including features around the sky region) with the texture details of the non-sky regions in the visible light image. Figure 4 It can be seen that GFF failed to incorporate the brighter pedestrian areas in the infrared image into its fused image (see...). Figure 4 (c)). HMSD and LPSR fused infrared and visible light images well in most areas, but failed to fuse brighter treetop areas (see Figure 4 Regions within the rectangles (d) and (e) are integrated into their fused image. Figure 3 (f) Similar, Figure 4 (c) This shows that FusionGAN failed to transfer the rich texture information from infrared and visible light images to its fused image, resulting in lower contrast compared to the source images and other methods. Figure 4 (g) It can be seen that U2Fusion can integrate most of the significant information from infrared and visible light images into its fused image, but the contrast of its fused image is slightly lower than that of the source image. Finally, from Figure 4 (h) It can be seen that the fused image of the method of the present invention not only integrates the bright branches of the infrared image well, but also achieves good contrast, thus presenting the best visual effect.

[0047] These two comparative examples demonstrate that the method of this invention can effectively fuse salient features of infrared and visible light images into a single image, outperforming current advanced image fusion methods. Next, we will further utilize quantitative evaluation methods to verify the advantages of the method of this invention.

[0048] 3. Quantitative assessment results Since qualitative assessment relies too heavily on subjective observation and has the disadvantages of inaccuracy and being time-consuming and laborious, we further adopted six quantitative indicators to objectively compare the performance of different methods.

[0049] First, quantitative metrics for the fusion results of all image fusion methods on the image datasets used were calculated, and then the average metric values ​​of all methods are listed in Table 1. As can be seen from Table 1, the method of this invention achieves optimal results in five out of the six metrics. Specifically, the method of this invention achieves the largest E, SF, and AG values ​​and the second largest STD value, indicating that the fused image generated by the method of this invention has more texture detail information compared to the fused images of the other five methods. Furthermore, the method of this invention achieves the largest VIFF and NSSIM values, indicating that the fused image generated by the method of this invention extracts more texture structure information from the source image compared to the fused images of the other five methods.

[0050] Table 1. Quantitative evaluation results of different image fusion methods

[0051] Note: In each row of data, bold and underlined values ​​represent the best and second-best performance, respectively.

[0052] In summary, the qualitative and quantitative evaluation results verify that the method of the present invention can effectively fuse infrared and visible light images, and in most cases outperforms current advanced image fusion methods. Furthermore, the method of the present invention requires approximately 0.50 seconds to fuse a pair of infrared and visible light images without optimization; further improvements in efficiency can be made by utilizing parallel computing techniques or deploying the method on a graphics processing unit.

[0053] 4. Conclusion This invention designs a highly efficient multi-scale Bezel filter, which can extract bright and dark features at different scales from infrared and visible light images and generate a smooth base image for each source image. Based on the proposed multi-scale Bezel filter, this invention further proposes a method for infrared and visible light image fusion. This image fusion method can integrate significant bright and dark features from infrared and visible light images into an information-rich fused image. Extensive experimental results show that the infrared and visible light image fusion method proposed in this invention significantly outperforms five existing advanced image fusion methods (including two deep learning-based methods) both qualitatively and quantitatively. Overall, the method of this invention has great application potential in civilian and military security monitoring systems.

[0054] In summary, the infrared and visible light image fusion method using multi-scale Bézier filters described in the above embodiments designs a multi-scale Bézier filter and uses it to perform multi-scale filtering on infrared and visible light images. This effectively extracts significant bright and dark features from the infrared and visible light images and generates an information-rich fused image. This method is not only superior to advanced transform domain image fusion methods but also even superior to the latest deep learning-based image fusion methods. It can be widely applied to various security monitoring systems.

[0055] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for fusing infrared and visible light images using multi-scale Bézier filters, characterized in that, Includes the following steps: S1: Extraction of bright and dark features from infrared and visible light images based on multi-scale Bézier filters; The infrared image and the corresponding visible light image are input, and the infrared image and the visible light image are filtered stepwise using Bezier filters of different scales. At the same time, the bright and dark feature maps at the corresponding scales are extracted from the filtered infrared image and the visible light image respectively, and the finally filtered infrared image and the visible light image are used as the base images of the input infrared image and the visible light image respectively. S2: Fusion of bright and dark feature maps The bright and dark feature maps of infrared and visible light images at each scale are fused using a pixel-wise maximum value strategy. The scale correlation coefficient is used to enhance the initial fused bright and dark feature maps at each scale. The enhanced multi-scale bright and dark features are then superimposed to obtain a fused bright feature map and a fused dark feature map. S3: Basic Image Fusion The average image of the two base images is calculated separately. Then, the average image of the two base images is averaged to obtain the fused average image. At the same time, the residual bright and dark features in the two base images other than the average image are fused using the pixel-wise maximum value strategy. Finally, the fused average image and the residual bright and dark feature images are superimposed to obtain the fused base image. S4: Infrared and Visible Image Fusion The fused bright feature map, dark feature map and fused base image are superimposed to generate a fused image of infrared and visible light images; In step S1, the Bezier filter is a local interpolation operator performed on an image patch, denoted as: ； In the formula, The location of the interpolation point is indicated by an interpolation ratio between 0 and 1. M is a constant interpolation coefficient matrix; P is a gray value matrix of 4×4 control points uniformly distributed on the image block to be filtered; Q is the filtered image block. In step S2, the scale correlation coefficient is used to enhance the bright and dark feature maps at each scale of the initial fusion. Specifically, this includes multiplying each extracted feature map at each scale by a scale-related weight to enhance the fused bright and dark feature maps, and then superimposing the enhanced multi-scale bright features and multi-scale dark features to obtain a fused bright feature map and a fused dark feature map. ； In the formula, This represents the weight coefficients of the bright and dark feature maps at the i-th scale. , is a constant not less than 1, and n is the number of scales; The specific process of step S3 is as follows: S31: Decompose the base images of the infrared image and the visible light image into three parts respectively: ； In the formula, and They represent and The average image; and They represent and Residual bright features in the middle; and They represent and Residual dark features in the middle; and These are the base images for infrared and visible light images, respectively; S32: Will and The three parts obtained from the decomposition are then fused together: ； In the formula, , and These represent the fused average image, the residual bright feature map, and the residual dark feature map, respectively. S33: Overlay , and Determine the base image for fusion: 。 2. The infrared and visible light image fusion method using a multi-scale Bezel filter according to claim 1, characterized in that: The specific process of step S1 is as follows: S11: First, adjust the size of the original image to (3km+1)×(3lm+1), then decompose the input image into k×l image blocks of size (3m+1)×(3m+1), where k and l are positive integers, m is the radius of the local region where the minimum and maximum gray values ​​are searched, and the original image is the input infrared image and visible light image; S12: Perform Bezier filtering on each image block and adjust the filtered image to the original image size; S13: Extracting brightness feature maps at multiple scales from the filtered infrared image Dark feature map And extract brightness feature maps at multiple scales from the filtered visible light image. Dark feature map Meanwhile, the infrared image and visible light image obtained from the last filtering are used as the base images for the infrared image and visible light image, respectively.

3. The infrared and visible light image fusion method using a multi-scale Bezel filter according to claim 2, characterized in that: In step S12, during Bezel filtering, the image block is first reconstructed using a 4×4 control point local minimum gray value matrix to remove bright spots, and then the image block is reconstructed using an updated 4×4 control point local maximum gray value matrix to fill black holes.

4. The infrared and visible light image fusion method using a multi-scale Bezel filter according to claim 3, characterized in that: In step S12, the Bessel filtering function using the local minimum gray value matrix is ​​defined as follows: The Bézier filter function using the local maximum gray value matrix is ​​defined as follows: The process of smoothing an image patch using a single-scale Bezier filter can be represented as follows: ； In the formula, B and These represent the original image block and the filtered image block, respectively. Compare the original image block B with the filtered image block The salient features of the input image patch are obtained by subtraction, where (B- The positive values ​​of ) correspond to bright features in the image, and the negative values ​​correspond to dark features in the image; Then, the Bézier filter is extended to multiple scales by iteratively applying the Bézier filter, as shown in the following equation: ； In the formula, i represents the current scale number of the Bezier filter, which increases from 1 to n; B i B0 represents the filtered image block at the i-th scale, and B0 represents the original image block B. Simultaneously, bright and dark features at different scales are extracted from the difference image of every two consecutive filtered image blocks, as shown in the following formula: ； ； In the formula, and These represent the bright and dark feature maps extracted from image block B at the i-th scale, respectively.

5. The infrared and visible light image fusion method using a multi-scale Bezel filter according to claim 4, characterized in that: In step S13, the image patch obtained from the last filtering is used as the base image of the original image patch, as shown in the following formula: ； In the formula, This represents the base image of the original image block B.

6. The infrared and visible light image fusion method using a multi-scale Bezel filter according to claim 5, characterized in that: In step S2, a pixel-by-pixel maximum value strategy is used to fuse the bright and dark feature maps of the infrared image and the visible light image: 。 7. The infrared and visible light image fusion method using a multi-scale Bézier filter according to claim 6, characterized in that: In step S4, the base image, bright feature map, and dark feature map are superimposed and fused to generate a fused image of the infrared and visible light images: 。

Images