A method for fusion of nighttime visible light and thermal infrared images

Abstract

The present invention relates to a method for fusing nighttime visible light and thermal infrared images, and belongs to the field of image processing. Based on the image layering theory, the present invention uses a combination strategy of weighted guided filtering and Gaussian filtering to decompose nighttime visible light images and thermal infrared images into basic, edge and texture layers respectively; then a layered complementary fusion method based on visual brightness perception characteristics is proposed to fuse and reconstruct the basic layer, texture layer and edge layer of nighttime visible light images and thermal infrared images. This method not only retains the natural color and texture details of the visible light image, but also fuses the thermal radiation information of the thermal infrared image, significantly improving the saliency of the target and the overall quality of the image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a method for fusing nighttime visible light and thermal infrared images. Background Art

[0002] At night or in low-light environments, visible light imaging devices are limited by light intensity, and the images they capture are often accompanied by a lot of noise, with blurred details and difficulty in accurately reflecting the target scene. In contrast, thermal infrared imaging technology uses infrared radiation emitted by objects to form images, which is not restricted by lighting conditions and can capture the thermal information of targets even in completely dark environments. It has unique advantages in nighttime target detection. However, thermal infrared images have low spatial resolution and lack detailed information, especially texture and color, which limits their effectiveness in certain application scenarios. In view of the respective characteristics and limitations of visible light images and thermal infrared images, image fusion technology has emerged, aiming to integrate the complementary information of the two to generate a fused image that simultaneously displays the color and texture details of visible light and the thermal information of thermal infrared, thereby providing more complete and clearer target scene information.

[0003] The nighttime visible light and thermal infrared image fusion technology has shown broad application prospects in many fields, including nighttime monitoring, target detection, military reconnaissance, and security monitoring. Through fusion, not only can the salience and visibility of targets in shadows or low illumination in visible light images be enhanced, but the fused image can also be used to improve the performance of subsequent detection and recognition tasks.

[0004] However, the challenges faced in the fusion process cannot be ignored. First, visible light and thermal infrared images are based on different physical signals and imaging principles, and their data characteristics are significantly different. How to effectively use the complementarity of these physical signals in fusion is a major problem. Secondly, the night imaging environment is complex and changeable, and the image quality of the two is affected differently by external factors. How to balance the advantages of the two in fusion to avoid information loss and visual degradation is another key issue.

[0005] Traditional image fusion methods mostly focus on image data features rather than the significance of the transformation from physical signals to image features, which results in the fused image not being able to fully retain important information of human visual perception, such as edges and colors. In addition, some fusion methods are deficient in processing low-contrast information, resulting in unnatural visual effects of the fused image, or are too complex in parameter adjustment, requiring re-adjustment for different scenes, reducing practicality and flexibility. Although deep learning technology has made some progress in the field of image fusion in recent years, such as the use of convolutional neural networks and generative adversarial networks, which have improved the details and realism of fused images, there are still problems such as difficulty in multimodal data registration, unstable training, and lack of real data guidance.

[0006] In response to the above problems, researchers are constantly exploring new fusion strategies, such as methods based on multi-scale decomposition, using saliency models to calculate weights, and combining deep learning frameworks to optimize the fusion process. However, these methods still need to be strengthened in processing the complementarity of color features of visible light images, especially in night environments. How to retain the true color information in the fused image, avoid detail loss, and improve the saliency of the target area is a key issue that needs to be urgently addressed in current research. Summary of the invention

[0007] In view of this, in order to at least partially solve the above technical problems, the present invention provides a method for fusing visible light and thermal infrared images at night based on the visual brightness perception characteristics, aiming to make full use of the chromaticity and brightness information of visible light images and the thermal information of thermal infrared images, overcome the limitation of single image mode lacking information in night environment, and thus design an image fusion technology that can effectively fuse the two at night.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A method for fusing nighttime visible light and thermal infrared images, comprising:

[0010] Decompose the nighttime visible light image and thermal infrared image respectively to obtain the corresponding basic layer, texture layer and edge layer;

[0011] The basic layer, texture layer and edge layer of the nighttime visible light image are reconstructed with the basic layer, texture layer and edge layer of the thermal infrared image in turn according to the fusion weights of each layer to obtain a fused image.

[0012] Preferably, the process of decomposing the image includes:

[0013] The image is subjected to weighted guided filtering and Gaussian low-pass filtering in sequence to obtain the basic layer of the image;

[0014] The difference between the original image and the weighted guided filtering result is used as the texture layer;

[0015] And the difference between the weighted guided filtering result and the image basic layer is used as the edge layer.

[0016] Preferably, before decomposing the nighttime visible light image, brightness adjustment and denoising are performed with the corresponding color saliency map as a guide; the steps include:

[0017] Normalize the color saliency map;

[0018] The normalized color saliency map is brightness adjusted using a weighted guided filter function to obtain a filtered brightness map;

[0019] The filtered brightness image is denoised using a global adaptive logarithmic mapping adjustment method.

[0020] Preferably, the filtered brightness image is denoised according to the following formula:

[0021]

[0022] In the formula, Adjust the brightness value after processing for preliminary noise reduction. Indicates the maximum value of the input image brightness, represents the logarithmic mean of the brightness values ​​of the input image; and

[0023]

[0024] In the formula, is the size of the image, A constant that prevents the brightness value from being zero.

[0025] Preferably, the step of acquiring the color saliency map of the nighttime visible light image comprises:

[0026] Convert nighttime visible light images from RGB color space to HSV color space;

[0027] The color saliency map is calculated according to the following formula:

[0028]

[0029] In the formula, represents the calculated color saliency map, where S and V represent the saturation and brightness of the visible light image converted to the HSV color space, respectively.

[0030] Preferably, the step of obtaining the base layer fusion weight includes:

[0031] Obtain the color saliency map of the nighttime visible light image, perform weighted guided filtering and Gaussian low-pass filtering in sequence, and obtain the basic layer of the color saliency map;

[0032] According to the basic layer of the color saliency map and the basic layer of the thermal infrared image, the basic layer fusion weight is determined by using a global adaptive logarithmic mapping adjustment method.

[0033] Preferably, the base layer fusion weight is determined using a global adaptive logarithmic mapping adjustment method such as the following formula;

[0034]

[0035] In the formula, Complementary feature map of the basic layer of thermal infrared image, ALTM global is a global adaptive logarithmic mapping adjustment method, is the base layer fusion weight.

[0036] Preferably, the step of obtaining the edge layer fusion weight includes:

[0037] Using the Sobel operator gradient function, the gradient feature maps of the edge layer of the nighttime visible light image and the edge layer of the thermal infrared image are calculated and normalized.

[0038] The normalized nighttime visible light image gradient feature map and the thermal infrared image gradient feature map are evaluated for mutual correlation coefficients to obtain a mutual correlation feature map;

[0039] Obtain a color saliency map of the nighttime visible light image, and combine it with the cross-correlation feature map to determine the edge layer fusion weight according to the following formula;

[0040]

[0041] In the formula, Indicates the degree of information loss in the visible light image. Represents the cross-correlation feature map of each band image of visible light and thermal infrared image, Represents the color saliency map of nighttime visible light images, is the fusion weight corresponding to the edge layer.

[0042] Preferably, the step of acquiring the texture layer fusion weight includes:

[0043] Obtain the difference between the nighttime visible light image texture layer and the thermal infrared image texture layer, and perform normalization;

[0044] According to the normalized difference map, the texture layer fusion weight is determined according to the following formula;

[0045]

[0046] In the formula, is the normalized difference map, Add weights to the texture layer. Represents a global adaptive logarithmic mapping adjustment method.

[0047] Preferably, the base layer, texture layer and edge layer of the nighttime visible light image and the base layer, texture layer and edge layer of the thermal infrared image are reconstructed according to the following formulas;

[0048]

[0049] Where F represents the reconstructed fused image, represents the base layer of a visible light image, represents the basic layer fusion weight, represents the base layer of thermal infrared images, represents the edge layer fusion weight, represents the edge layer of the visible light image, Represents the edge layer of the thermal infrared image, represents the texture layer fusion weight, A texture layer representing a visible light image, Texture layer representing a thermal infrared image.

[0050] It can be seen from the above technical solution that the present invention discloses a method for fusing nighttime visible light and thermal infrared images, which generates a synthetic image with the advantages of both images by fusing the rich texture and color information of the visible light image and the thermal radiation distribution characteristics of the thermal infrared image, thereby significantly improving the recognizability of nighttime scenes.

[0051] In the fusion process, the present invention uses a multi-level complementary fusion strategy based on visual perception. This strategy fully utilizes the advantages of visible light images in color and texture, and combines the strengths of thermal infrared images in target saliency, ensuring that the fused image achieves the best balance in detail preservation and target highlighting.

[0052] Furthermore, the present invention designs a color saliency feature guide map based on brightness and saturation differences to target the complex characteristics of nighttime illumination changes. This not only enhances visible light information in low-brightness areas, improves image contrast and detail clarity, but also reduces noise impact, significantly improving the quality of nighttime visible light images. When fusing information from different frequency layers, a fusion strategy that mimics the human eye's visual mechanism is adopted, that is, in the low-frequency information layer, the visible light color saliency map based on the shadow index realizes the complementary fusion of low-frequency information; in the edge layer, the color saliency feature and edge correlation are combined to evaluate the degree of loss of visible light information at night, and the complementarity of edge information is defined; in the texture layer, the optimization of texture details is achieved through global adaptive adjustment.

[0053] Compared with the prior art, this application has the following technical effects:

[0054] This application uses an image fusion method based on the differences in visual perception of the human eye under different lighting conditions and the complementary information of visible light and thermal infrared images to generate a fused image that is both clear and has rich details and color information. It can not only supplement the radiation information of thermal infrared in the dark, but also maintain the visual naturalness of the brightness area.

[0055] This application can be widely used to improve image quality in fields such as night monitoring, target detection, military reconnaissance and security monitoring, providing clearer and more accurate target scene information. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A schematic flow chart of a method for fusing nighttime visible light and thermal infrared images provided in an embodiment of the present invention.

[0058] Figure 2 An experimental comparison of different fusion methods on LLVIP data image 1 provided by an embodiment of the present invention.

[0059] Figure 3 An experimental comparison of different fusion methods on LLVIP data image 2 provided by an embodiment of the present invention.

[0060] Figure 4 An experimental comparison of different fusion methods on KAIST data image 1 provided by an embodiment of the present invention.

[0061] Figure 5 An experimental comparison of different fusion methods on KAIST data image 2 provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] This embodiment discloses a method for fusing nighttime visible light and thermal infrared images, comprising:

[0064] Decompose the nighttime visible light image and thermal infrared image respectively to obtain the corresponding basic layer, texture layer and edge layer;

[0065] The basic layer, texture layer and edge layer of the nighttime visible light image are reconstructed with the basic layer, texture layer and edge layer of the thermal infrared image in turn according to the fusion weights of each layer to obtain a fused image.

[0066] Specifically, the present application firstly analyzes the brightness and saturation of visible light images and uses color saliency map construction technology to pre-process night visible light images to enhance the brightness and color features of the images while suppressing image noise; secondly, optimizes the image brightness through global adaptive logarithmic mapping adjustment combined with weighted guided filtering and Gaussian filtering to ensure that the brightness of darker areas is significantly improved, thereby improving the visual effect of visible light images; then, based on image layering theory, the image is decomposed into low-frequency, edge and texture layers, and the combination strategy of weighted guided filtering and Gaussian filtering is used to accurately distinguish image details and edges, avoiding the generation of detail edge artifacts; finally, a layered complementary fusion method based on visual brightness perception characteristics is proposed, and the weight of the thermal distribution information of the thermal infrared image is adaptively adjusted according to the distribution of color saturation in the visible light image, ensuring that the thermal infrared information dominates in areas with low saturation, while the details of the visible light image are maintained in areas with normal saturation, thereby achieving efficient complementary fusion of visible light and thermal infrared image information.

[0067] The present invention not only retains the natural color and texture details of the visible light image, but also integrates the thermal radiation information of the thermal infrared image, significantly improving the saliency of the target and the overall quality of the image.

[0068] In an exemplary embodiment, referring to Figure 1 ,

[0069] First, before decomposing the nighttime visible light image, brightness adjustment and denoising are performed using the corresponding color saliency map as a guide; the steps include:

[0070] Normalize the color saliency map;

[0071] The normalized color saliency map is brightness adjusted using a weighted guided filter function to obtain a filtered brightness map;

[0072] The filtered brightness image is denoised using a global adaptive logarithmic mapping adjustment method.

[0073] This embodiment uses the normalized color saliency map as a guide map and performs noise reduction on the visible light image through weighted guided filtering. The weight of the filter is dynamically adjusted according to the saliency of each pixel in the color saliency map to enhance the color salient area in the image while reducing noise. The image brightness is adjusted by applying a global adaptive logarithmic mapping adjustment method to simulate the perception characteristics of the human eye in different brightness environments, so that the dark details of the image can be enhanced and the bright details can be retained.

[0074] Among them, the normalization processing and the weighted guided filtering formula used are:

[0075]

[0076] In the formula, is the normalized representation of the color saliency map, Norm is the normalization operation, is the brightness image after filtering, is the weighted guided filter function, is the parameter used by the guide filter function, and I is the brightness value of the input image. In this application, I is calculated as follows:

[0077]

[0078] Furthermore, the calculation formula of the global adaptive logarithmic mapping adjustment method applied is:

[0079]

[0080] In the formula, Adjust the brightness value after processing for preliminary noise reduction. Indicates the maximum value of the input image brightness, Represents the logarithmic mean of the brightness values ​​of the input image, which is obtained by the following formula:

[0081]

[0082] In the formula, is the size of the image, A constant that prevents the brightness value from being 0, set to 0.000001.

[0083] Optionally, in this embodiment, the step of acquiring the color saliency map of the nighttime visible light image includes:

[0084] Convert nighttime visible light images from RGB color space to HSV color space;

[0085] The color saliency map is calculated according to the following formula:

[0086]

[0087] In the formula, represents the calculated color saliency map, where S and V represent the saturation and brightness of the visible light image converted to the HSV color space, respectively.

[0088] Converting visible light images from their original RGB color space to HSV color space can more intuitively express the three basic attributes of color, namely hue (H), saturation (S), and brightness (V). By calculating the difference between saturation and brightness and normalizing them, a color saliency map (CSI) is formed, which can highlight areas in the image with rich colors or abnormal brightness, thereby providing important visual information for subsequent image processing.

[0089] In another embodiment, the image is decomposed according to the following steps:

[0090] The image is subjected to weighted guided filtering and Gaussian low-pass filtering in sequence to obtain the basic layer of the image;

[0091] The difference between the original image and the weighted guided filtering result is used as the texture layer;

[0092] And the difference between the weighted guided filtering result and the image basic layer is used as the edge layer.

[0093] The denoised visible light image and thermal infrared image are processed by weighted guided filtering to extract the texture layer in the image. Then, the basic layer and edge layer of the image are further extracted using Gaussian low-pass filtering. The weighted guided filtering and Gaussian low-pass filtering formulas are:

[0094]

[0095] In the formula, c represents the visible light image after noise reduction, is the result of weighted guided filtering, is the basic layer of the final visible light, are the parameters of the filter.

[0096] Furthermore, the same filtering process is performed on the thermal infrared image to obtain and , It is a thermal infrared single-channel image.

[0097] After obtaining the basic layer, the texture layer and edge layer are calculated by the following formula;

[0098]

[0099] In the formula, and are the texture layer and edge layer of the visible light image, and It represents the texture layer and edge layer of the thermal infrared image.

[0100] In order to further optimize the above technical solution,

[0101] In one embodiment, the base layer fusion weights are obtained by the following steps:

[0102] Obtain the color saliency map of the nighttime visible light image, perform weighted guided filtering and Gaussian low-pass filtering in sequence, and obtain the basic layer of the color saliency map;

[0103] According to the basic layer of the color saliency map and the basic layer of the thermal infrared image, the basic layer fusion weight is determined by using a global adaptive logarithmic mapping adjustment method.

[0104] This embodiment first performs the same decomposition process on the color saliency map to obtain information of different frequency layers. Then, by calculating the complementary feature map of the basic layer of the thermal infrared image and combining it with the basic layer of the color saliency map, the fusion weight of the low-frequency layer is calculated, and finally the basic layer of the visible light image is fused with the basic layer of the thermal infrared image to form a fused low-frequency basic layer.

[0105] Specifically, suppose the color saliency map is subjected to the same decomposition process, and the different layers of the color saliency map are obtained as follows: and , where csi is the color saliency map, is the weighted guided filtering result of the color saliency map, is the basic layer of the color saliency map obtained by Gaussian low-pass filtering.

[0106] Furthermore, the complementary feature map of the basic layer of the thermal infrared image is calculated and combined with the basic layer of the color saliency map. The fusion weight of the low-frequency layer is calculated by the global adaptive logarithmic mapping adjustment method. The weight calculation formula is:

[0107]

[0108] In the formula, Complementary feature map of the basic layer of thermal infrared image, ALTM global is a global adaptive logarithmic mapping adjustment method, is the base layer fusion weight.

[0109] In one embodiment, the edge layer fusion weight is obtained by the following steps:

[0110] (1) Using the Sobel operator gradient function, the gradient feature maps of the edge layer of the nighttime visible light image and the edge layer of the thermal infrared image are calculated and normalized respectively. The calculation formula is:

[0111]

[0112] In the formula, represents the Sobel operator gradient calculation function, is the normalized gradient feature map of each band of visible light, calculated in the same way , is the normalized gradient feature map of the thermal infrared image;

[0113] (2) Evaluate the cross-correlation coefficient of the normalized nighttime visible light image gradient feature map and the thermal infrared image gradient feature map to obtain a cross-correlation feature map;

[0114] In this embodiment, the normalized cross-correlation coefficient is used to evaluate the correlation between the edge gradient feature maps of the two images, and the calculation formula is:

[0115]

[0116] In the formula, Represents the cross-correlation feature map of each band image of visible light and thermal infrared image, with a value between -1 and 1

[0117] (3) Obtain the color saliency map of the nighttime visible light image, and determine the edge layer fusion weight in combination with the cross-correlation feature map. The present invention uses the color saliency feature map corresponding to the edge layer to evaluate the degree of information loss in the visible light image and calculate the edge layer fusion weight. The calculation formula is:

[0118]

[0119] In the formula, Indicates the degree of information loss in the visible light image. Represents the cross-correlation feature map of each band image of visible light and thermal infrared image, Represents the color saliency map of nighttime visible light images, is the fusion weight corresponding to the edge layer.

[0120] In one embodiment, the texture layer fusion weights are obtained by the following steps:

[0121] First, the difference between the nighttime visible light image texture layer and the thermal infrared image texture layer is obtained and normalized; the calculation formula for the normalized difference map is:

[0122]

[0123] Secondly, according to the normalized difference map, the texture layer fusion weight is determined by the following formula using the global adaptive logarithmic mapping adjustment method;

[0124]

[0125] In the formula, is the normalized difference map, Add weights to the texture layer. Represents a global adaptive logarithmic mapping adjustment method.

[0126] In another embodiment, according to the above-obtained basic layer fusion weight, edge layer fusion weight and texture layer fusion weight, the basic layer, edge layer and texture layer of the denoised visible light and thermal infrared images are reconstructed to obtain a final visible light three-band fusion image.

[0127] The reconstruction formula is as follows;

[0128]

[0129] Where F represents the reconstructed fused image, represents the base layer of a visible light image, represents the basic layer fusion weight, represents the base layer of thermal infrared images, represents the edge layer fusion weight, represents the edge layer of the visible light image, Represents the edge layer of the thermal infrared image, represents the texture layer fusion weight, A texture layer representing a visible light image, Texture layer representing a thermal infrared image.

[0130] In order to fully verify the effectiveness and practicality of the method of the present invention, in the embodiments of the present invention, the proposed nighttime visible light and thermal infrared image fusion method guided by the visual brightness perception characteristics was extensively tested and evaluated. In the test, the method of the present invention was compared with a variety of existing advanced image fusion technologies, including Representation-learning guided Fusion (LRR), Hybrid Multi-scale Decomposition (Hybrid), Gradient Transfer Fusion (GTF), A End-to-end Residual Fusion (RFN) and SeAfusion. These comparative experiments cover two main data sets: LLVIP data set and KAIST multispectral data set, which contain nighttime images with different lighting conditions and signal-to-noise ratios, and can fully simulate the diverse scenes in practical applications.

[0131] Figures 2 to 5 The image fusion effects of the method of the present invention and other comparative methods in different scenarios are demonstrated. Specifically, Figure 2 An experimental comparison of different fusion methods on LLVIP data image 1 provided by an embodiment of the present invention; Figure 3 An experimental comparison of different fusion methods on LLVIP data image 2 provided by an embodiment of the present invention; Figure 4 An experimental comparison of different fusion methods on KAIST data image 1 provided by an embodiment of the present invention; Figure 5 An experimental comparison of different fusion methods on KAIST data image 2 provided by an embodiment of the present invention.

[0132] Among them, Figure 1-Figure 5In the figure, (a) represents a visible light image, (b) represents a thermal infrared image, (c) represents an LRR result, (d) represents a Hybrid result, (e) represents a GTF result, (f) represents an RFN result, (g) represents a SeAfusion result, and (h) represents the result of the method of the present invention.

[0133] Through intuitive visual effect comparison, it can be seen that the method of the present invention has significant advantages in maintaining image brightness and color naturalness. Especially in low illumination and high noise environments, the method of the present invention can effectively enhance the brightness and color characteristics of the image, while suppressing noise, making the fused image clearer and more natural.

[0134] In order to quantitatively evaluate the performance of each fusion method, this embodiment uses multiple objective evaluation indicators, including information entropy (IE), standard deviation (SD), mutual information (MI), structural similarity (SSIM), edge information transfer (QAB / F) and visual information fidelity (VIF). In addition, in order to evaluate the accuracy of color features, color difference (CD) is also used as an evaluation indicator.

[0135] Tables 1 and 2 summarize the test results of various fusion methods on the LLVIP dataset and KAIST dataset on the above evaluation indicators.

[0136] Table 1

[0137]

[0138] Table 2

[0139]

[0140] It can be seen from the data that the method of the present invention shows excellent performance in multiple evaluation indicators, especially in VIF and CD indicators, the scores of the method of the present invention are significantly higher than those of other comparison methods, which shows that the method of the present invention has obvious advantages in maintaining the naturalness and color accuracy of the image.

[0141] Based on the analysis results of subjective evaluation and objective indicators, it can be concluded that the nighttime visible light and thermal infrared image fusion method of the present invention performs well in improving image brightness, enhancing color features, suppressing noise, and maintaining image naturalness. It can effectively overcome the challenges of nighttime imaging environment and provide a high-quality image fusion solution for applications such as nighttime monitoring and target detection.

[0142] From the description of the above embodiments, those skilled in the art can know that: in order to overcome the problems existing in the existing nighttime visible light and thermal infrared image fusion technology, such as color distortion, loss of detail information, and unnatural phenomena of the fused image, the present invention proposes an innovative nighttime visible light and thermal infrared image fusion method guided by visual brightness perception characteristics. The method is based on the visual brightness perception characteristics of the human eye under different lighting conditions, and makes full use of the complementarity of the color texture information of the visible light image and the thermal radiation information of the thermal infrared image.

[0143] In the implementation of the method, the color saliency map is first used to evaluate and enhance the color information in the visible light image. Through specific weighted guided filtering and global adaptive logarithmic mapping adjustment, the brightness and color characteristics of the image are effectively improved, while the influence of noise is reduced. Then, through image decomposition technology, the visible light and thermal infrared images are decomposed into different frequency layers, including low-frequency layer, edge layer and texture layer, providing structured image data for subsequent fusion.

[0144] Furthermore, the method of the present invention adopts weight calculation based on visual brightness perception characteristics in low-frequency layer fusion, which optimizes the fusion effect of the basic layer of visible light and thermal infrared images. In the edge layer complementary mapping, the complementary relationship between the edge information of nighttime visible light and thermal infrared images is accurately fitted by calculating the cross-correlation feature map and using the color saliency map to evaluate the degree of information loss. Finally, in the texture layer enhancement, the texture details are effectively enhanced by calculating the texture difference and applying the global adaptive logarithmic mapping adjustment.

[0145] Through experimental results and analysis, the effectiveness of the proposed method on different data sets is verified, and its superiority in improving the quality of nighttime visible light and thermal infrared image fusion is demonstrated. The proposed method not only improves the naturalness and visual appeal of the fused image, but also enhances the usability of the image in practical applications, such as nighttime monitoring, target detection and other scenarios.

[0146] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0147] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fusing nighttime visible light and thermal infrared images, characterized in that: Decomposing the nighttime visible light image and the thermal infrared image respectively to obtain the corresponding basic layer, texture layer and edge layer; before decomposing the nighttime visible light image, brightness adjustment and denoising are performed with the corresponding color saliency map as a guide; the steps include: Normalize the color saliency map; The normalized color saliency map is brightness adjusted using a weighted guided filter function to obtain a filtered brightness map; The filtered brightness image is denoised using a global adaptive logarithmic mapping adjustment method; The steps of obtaining the color saliency map of the nighttime visible light image include: Convert nighttime visible light images from RGB color space to HSV color space; The color saliency map is calculated according to the following formula: In the formula, I csi represents the calculated color saliency map, where S and V represent the saturation and brightness of the visible light image converted to the HSV color space; The basic layer, texture layer and edge layer of the nighttime visible light image are reconstructed with the basic layer, texture layer and edge layer of the thermal infrared image in turn according to the fusion weights of each layer to obtain a fused image.

2. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: The process of decomposing an image includes: The image is subjected to weighted guided filtering and Gaussian low-pass filtering in sequence to obtain the basic layer of the image; The difference between the original image and the weighted guided filtering result is used as the texture layer; And the difference between the weighted guided filtering result and the image basic layer is used as the edge layer.

3. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: De-noise the filtered brightness image according to the following formula; In the formula, I c (x, y) is the brightness value after the initial noise reduction adjustment process, L w (x, y) is the brightness image after filtering, ALTM global is a global adaptive logarithmic mapping adjustment method, L wmax Indicates the maximum value of the input image brightness, L′ w represents the logarithmic mean of the brightness values ​​of the input image; and Where m×n is the size of the image, and σ is a constant that prevents the brightness value from being zero.

4. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: The steps for obtaining the basic layer fusion weights include: Obtain the color saliency map of the nighttime visible light image, normalize it, and then perform weighted guided filtering and Gaussian low-pass filtering in sequence to obtain the basic layer of the color saliency map; According to the basic layer of the color saliency map and the basic layer of the thermal infrared image, the basic layer fusion weight is determined by using a global adaptive logarithmic mapping adjustment method.

5. The method for fusing nighttime visible light and thermal infrared images according to claim 4, characterized in that: The basic layer fusion weight is determined by using the global adaptive logarithmic mapping adjustment method as shown in the following formula; In the formula, is the complementary feature map of the basic layer of the thermal infrared image, is the basic layer of the color saliency map obtained by Gaussian low-pass filtering, ALTMglobal is the global adaptive logarithmic mapping adjustment method, is the base layer fusion weight.

6. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: The steps for obtaining edge layer fusion weights include: Using the Sobel operator gradient function, the gradient feature maps of the edge layer of the nighttime visible light image and the edge layer of the thermal infrared image are calculated and normalized respectively. The normalized nighttime visible light image gradient feature map and the thermal infrared image gradient feature map are evaluated for mutual correlation coefficients to obtain a mutual correlation feature map; Obtain a color saliency map of the nighttime visible light image, and combine it with the cross-correlation feature map to determine the edge layer fusion weight according to the following formula; In the formula, Indicates the degree of information loss in the visible light image, nccmapc represents the cross-correlation feature map of each band image of the visible light and the thermal infrared image, I csi Represents the color saliency map of nighttime visible light images, is the fusion weight corresponding to the edge layer, and tanh() represents a nonlinear activation function, which is used to convert Compressed to the interval [-1, 1].

7. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: The steps for obtaining the texture layer fusion weight include: Obtain the difference between the nighttime visible light image texture layer and the thermal infrared image texture layer, and perform normalization; According to the normalized difference map, the texture layer fusion weight is determined according to the following formula; In the formula, Dif c is the normalized difference map, Enhance weights for texture layers, ALTM global () represents the global adaptive logarithmic mapping adjustment method.

8. The method for fusing nighttime visible light and thermal infrared images according to claim 1, characterized in that: The basic layer, texture layer and edge layer of the nighttime visible light image and the basic layer, texture layer and edge layer of the thermal infrared image are reconstructed according to the following formulas; Where F represents the reconstructed fused image, represents the base layer of a visible light image, represents the basic layer fusion weight, represents the basic layer of thermal infrared imagery, represents the edge layer fusion weight, represents the edge layer of the visible light image, Represents the edge layer of the thermal infrared image, represents the texture layer fusion weight, A texture layer representing a visible light image, Texture layer representing a thermal infrared image.

Images