Infrared and visible image fusion method based on image enhancement and weight optimization

Abstract

The application relates to the technical field of image processing, and discloses an infrared and visible light image fusion method based on image enhancement and weight optimization. A new weight map construction method is adopted, important features of input images can be more fully extracted, and source images are enhanced before weighted fusion, so as to solve the defect that weighted fusion is prone to cause insufficient details and features; in addition, optimization of the weight coefficient makes the method more robust compared with a traditional weighted fusion method. The fusion image generated by the method has good feature and detail performance and does not produce artifacts.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for fusing infrared and visible light images based on image enhancement and weight optimization. Background Technology

[0002] Infrared sensors can effectively capture high-temperature targets, obtaining usable images even under obstructed or low-light conditions. However, infrared images suffer from incomplete scene details and poor clarity. Unlike infrared images, visible light images are typically rich in detail, but their imaging quality is greatly affected by lighting conditions, requiring a high level of environmental sensitivity. In harsh environments, such as those with insufficient light or smoke interference, it is necessary to use both infrared and visible light sensors simultaneously to capture images—that is, to obtain multi-source images—to obtain more complete information about the target scene. However, multi-source images require more storage space and more time to review. Infrared and visible light image fusion can overcome the imaging deficiencies of infrared sensors and the environmental limitations of visible light sensors, generating images containing more effective information.

[0003] In the field of pixel-level image fusion, deep learning-based methods and multi-scale transformation-based methods are currently the two most commonly used approaches. Deep learning-based methods suffer from long training times and a lack of real reference images. Multi-scale transformation-based methods include pyramid, multi-scale geometric analysis, and wavelet methods. Typically, these methods first decompose the source image into different scales to better handle details and features, then process, fuse, and inversely transform these scales using predefined rules to obtain the fused image. While MST-based methods generally preserve image details well, they are computationally complex and may produce artifacts near edges. In traditional multi-scale transformation-based fusion methods, the base layer uses an "average" rule for fusion, which can lead to contrast loss, especially when handling multi-source image fusion tasks. To achieve reasonable utilization of information from different source images, researchers have proposed some optimization-based fusion methods, transforming the fusion problem into an optimization problem. These optimization-based methods may result in an overly smooth fused image during iteration, lacking good detail and feature performance.

[0004] Therefore, this application proposes an infrared and visible light image fusion method based on image enhancement and weight optimization to solve the above-mentioned technical problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of existing technologies, this invention provides an infrared and visible light image fusion method based on image enhancement and weight optimization. It employs a novel weight map construction method, which can more fully extract important features from the input image. The source image is enhanced before weighted fusion to address the deficiency in detail and feature retention often caused by weighted fusion. Furthermore, optimization of the weight coefficients makes this method more robust than traditional weighted fusion methods. The fused image generated by this method exhibits excellent feature and detail representation and does not produce artifacts.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: an infrared and visible light image fusion method based on image enhancement and weight optimization, comprising the following steps:

[0007] S1. Obtain an infrared image of the target scene using an infrared sensor, and obtain a visible light image of the target scene using a visible light sensor;

[0008] S2. Generate weighted graphs for infrared and visible light images:

[0009] 1) Generate infrared image weight map: Obtain weight map B from the visible light image through improved grayscale inversion, and average the infrared image itself and weight map B to obtain the infrared image weight map.

[0010] 2) Generate visible light image weight map: Obtain weight map A from the infrared image through improved grayscale inversion, and average the visible light image itself and weight map A to obtain the visible light image weight map;

[0011] S3, Image Enhancement:

[0012] 1) Enhance the contrast of infrared images;

[0013] 2) Enhance the contrast and brightness of visible light images;

[0014] S4. Use the dandelion optimizer to determine the weight coefficients of the visible light image and the infrared image respectively, and perform weight optimization. Generate a fused image through weighted fusion.

[0015] Furthermore, the contrast of the infrared image in step S3 is enhanced by using effective image-guided filtering.

[0016] Furthermore, the contrast and brightness of the visible light image in step S3 are adaptively enhanced by using a multiphase algorithm based on retina theory combined with a dandelion optimizer.

[0017] In summary, this invention has the following beneficial effects: It employs a novel weighted graph construction method, which can more fully extract important features from the input image; it enhances the source image before weighted fusion to address the shortcomings of weighted fusion that easily lead to insufficient detail and features; furthermore, the optimization of the weight coefficients makes this method more robust than traditional weighted fusion methods. The fused image generated by this method has good feature and detail representation and does not produce artifacts. Attached Figure Description

[0018] Figure 1 This is a flowchart of the infrared and visible light image fusion method based on image enhancement and weight optimization in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating the construction process of the infrared and visible light image weight map in an embodiment of the present invention;

[0020] Figure 3 This is a graph showing the experimental results on the TNO dataset in this embodiment of the invention. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1-3 The present invention will be described in further detail below.

[0022] Example 1: An infrared and visible light image fusion method based on image enhancement and weight optimization, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0023] S1. Obtain an infrared image of the target scene using an infrared sensor, and obtain a visible light image of the target scene using a visible light sensor;

[0024] S2. Generate weighted graphs for infrared and visible light images:

[0025] 1) Generate infrared image weight map: Obtain weight map B from the visible light image through improved grayscale inversion, and average the infrared image itself and weight map B to obtain the infrared image weight map.

[0026] 2) Generate visible light image weight map: Obtain weight map A from the infrared image through improved grayscale inversion, and average the visible light image itself and weight map A to obtain the visible light image weight map;

[0027] S3, Image Enhancement:

[0028] 1) Enhance the contrast of the infrared image by using effective image steering filtering;

[0029] 2) Enhance the contrast and brightness of visible light images, wherein the contrast and brightness of visible light images are adaptively enhanced by using a multiphase algorithm based on retina theory combined with a dandelion optimizer.

[0030] S4. Use the dandelion optimizer to determine the weight coefficients of the visible light image and the infrared image respectively, and perform weight optimization. Generate a fused image through weighted fusion.

[0031] like Figure 3 The figure shown is a partial experimental result of this method on the TNO dataset. Figure 3 (a) is an infrared image. Figure 3 (b) is a visible light image. Figure 3 (c) is the fused image. Observation shows that... Figure 3 In the infrared image, there is clear pedestrian and traffic information, while the visible light image has shop signs and window outlines as well as other details. The fused image retains the features and details of both images, clearly showing shop signs and pedestrian information, and does not produce artifacts.

[0032] Example 2: An infrared and visible light image fusion algorithm based on image enhancement and weight optimization, which consists of three parts: weight map construction, image enhancement, and weight optimization.

[0033] (a) Weight graph construction

[0034] In most cases, objects and backgrounds in infrared images can be clearly distinguished, and important features are well represented. Therefore, this application considers the infrared image itself as part of its weight map. The weight map of an infrared image is a graph that combines the infrared image itself with... Figure 2 The weighted map of an infrared image is obtained by averaging the weighted map B. The weighted map of an infrared image is generated by the following formula:

[0035] W B =max(I VI )-I VI (1)

[0036]

[0037] Among them, W B yes Figure 2 In the weighted graph B, max(I) VI W is a matrix of the same size as the infrared image, composed of the maximum values ​​of the infrared image. IR It is the weighted map of the infrared image, I IR with I VI These represent infrared images and visible light images, respectively.

[0038] W BIt is obtained by modifying the grayscale inversion of a visible light image, thus eliminating important features of the visible light image. Therefore, the average I IR and W B The resulting weighted map can preserve the features of infrared images well without weakening the features of visible light images.

[0039] Similar to infrared image weight maps, visible light image weight maps consist of visible light images and... Figure 2 The weighted graph A is averaged in the graph, and the specific formula is as follows:

[0040] W A =max(I IR )-I IR (3)

[0041]

[0042] Where max(I) IR W is a matrix of the same size as the visible light image, composed of the maximum values ​​of the visible light image. VI It is a weighted map of a visible light image.

[0043] The weighted graph construction method proposed in this application not only effectively connects infrared and visible light images, enabling them to retain the feature information of the input image to the greatest extent, but also makes it less prone to artifacts.

[0044] (b) Image enhancement

[0045] After weighted fusion, the features and details of the two images are weakened to some extent. To preserve as many features and details as possible from the input images in the fused image, we perform different image enhancements on the two images before fusion. For the infrared image, Effective Image Guided Filtering (EGIF) is used to enhance contrast and make image features more prominent. Because EGIF can adaptively enhance image details, the EGIF parameters used in this method are the same as the default parameters.

[0046] For visible light images, since the visible light image in the fusion of infrared and visible light images is often captured at night or in low-light environments, its brightness or contrast is usually insufficient. Therefore, this application uses a multiphase algorithm based on retinal theory (RBMP) to enhance the brightness and contrast of visible light images. When using the RBMP algorithm to enhance images, an input parameter γ is needed to control the enhancement effect. The larger the γ value, the smaller the brightness enhancement and the larger the contrast enhancement; conversely, the smaller the γ value, the larger the brightness enhancement and the smaller the contrast enhancement. To achieve adaptive image enhancement, the image entropy (EN) is used as the objective function, and a suitable γ value is obtained through the dandelion optimizer (DO) to maximize the entropy of the enhanced image and achieve a balance between brightness and contrast enhancement in the visible light image. The method for calculating image entropy is as follows:

[0047]

[0048] Where P g EN represents the probability of each gray level, where G is the number of gray levels set to 256. The larger EN is, the more information the image contains, and the better the enhancement effect.

[0049] (c) Weight optimization

[0050] After obtaining the weighted mapping and enhanced images, a weighted average can be used to obtain the fused image to minimize information loss. Pixel-weighted averaging provides good robustness for the fusion algorithm; however, if one of the input images is of poor quality, the fused image will not have a good visual effect. To address this issue, an optimization method is used to generate different weight coefficients for the two enhanced images. The final fused image can be obtained from the following formula:

[0051] I F =W1·W IR E IR +W2·W VI E VI (6)

[0052] Where W1 and W2 are the weighting coefficients for the infrared image and the visible light image, respectively, E IR and E VI These are enhanced infrared images and enhanced visible light images, respectively.

[0053] Similar to visible light image enhancement, we use DO (Distributed Object) to perform weight optimization, finding suitable W1 and W2. Due to the lack of a real reference image, a metric is needed to judge the quality of the fused image. This application introduces structural similarity as an evaluation metric. Structural Similarity (SSIM) is a commonly used evaluation metric that reflects the similarity between two images, expressed as follows:

[0054] SSIM(I1,I2)=[l(I1,I2)] α [c(I1,I2)] β [s(I1,I2)] γ (7)

[0055] Where l(,), c(,), and s(,) represent brightness similarity, contrast similarity, and structural similarity, respectively, and I1 and I2 are two input images.

[0056] When using SSIM as the objective function, if the brightness difference between two images is large, the fused image will always be biased towards the brighter image, resulting in a lack of dark details. Therefore, this application removes the brightness similarity portion of SSIM, and SSIM is rewritten as follows:

[0057]

[0058] in and These are the standard deviations of image I1 and image I2, respectively. It is the covariance of the two images, and C is a given constant.

[0059] Removing the brightness similarity component might result in weighting coefficients that are too large or too small, leading to excessively large or small brightness in the fused image. Therefore, a constraint is added to W1 and W2 so that their sum is 2. From formula (1-4), we can obtain W... IR and W VI The sum is max(I) IR ) and max(I VI The mean of the weight maps of the two images is such that when both W1 and W2 are 1, the maximum value of the sum of the weight maps of the two images is close to 1, which means that the brightness of the fused image will be close to that of the input image.

[0060] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A method for fusing infrared and visible light images based on image enhancement and weight optimization, characterized in that, Includes the following steps: S1. Obtain an infrared image of the target scene using an infrared sensor, and obtain a visible light image of the target scene using a visible light sensor; S2. Generate weighted graphs for infrared and visible light images: 1) Generate infrared image weight map: Obtain weight map B from the visible light image through improved grayscale inversion, and average the infrared image itself and weight map B to obtain the infrared image weight map; The infrared image weight map is generated by the following formula: （1） （2） Among them, W B It is weighted graph B, max(I) VI W is a matrix of the same size as the infrared image, composed of the maximum values ​​of the infrared image. IR It is the weighted map of the infrared image, I IR with I VI These represent infrared images and visible light images, respectively. 2) Generate visible light image weight map: Obtain weight map A from the infrared image through improved grayscale inversion, and average the visible light image itself and weight map A to obtain the visible light image weight map; The visible light image weight map is generated by the following formula: （3） （4） Among them, W A It is a weighted graph A, max(I) IR W is a matrix of the same size as the visible light image, composed of the maximum values ​​of the visible light image. VI It is a weighted map of a visible light image; S3, Image Enhancement: 1) Enhance the contrast of infrared images; 2) Enhance the contrast and brightness of visible light images; S4. Use the dandelion optimizer to determine the weight coefficients of the visible light image and the infrared image respectively and perform weight optimization. During the optimization of the weight coefficients, set constraints according to the infrared image weight map and the visible light image weight map generated in step S2. Then, the infrared image and the visible light image enhanced in step S3 are weighted and fused according to the optimized weight coefficients to generate a fused image. The fused image can be obtained by the following formula: （6） Where W1 and W2 are the weighting coefficients for the infrared image and the visible light image, respectively, E IR and E VI These are enhanced infrared images and enhanced visible light images, respectively.

2. The infrared and visible light image fusion method based on image enhancement and weight optimization according to claim 1, characterized in that, The contrast of the infrared image in step S3 is enhanced by using effective image-guided filtering.

3. The infrared and visible light image fusion method based on image enhancement and weight optimization according to claim 1, characterized in that, The contrast and brightness of the visible light image in step S3 are adaptively enhanced by using a multiphase algorithm based on retina theory combined with the dandelion optimizer.

Images