Brightness compensation and texture enhancement infrared and visible light image fusion method

Abstract

The invention relates to the technical field of image processing, and discloses a brightness compensation and texture enhancement infrared and visible light image fusion method. According to image fusion, multiple images in the same scene are fused, and an image with larger information amount is generated. The invention provides a fusion method based on brightness compensation and detail enhancement in order to solve the problems that in the fusion process of an infrared image and a visible light image, the brightness of an infrared target is damaged, and image details are poor. Firstly, an image is divided into a plurality of detail layers and a base layer by using rolling guide filtering; in order to better reflect image features and illumination intensity, the base layer is divided into a significant feature layer and a brightness layer. In order to further improve fusion details, a fusion rule combining intensity details and gradient details is used. The infrared salient region adopts an intensity detail layer, and the background region adopts a gradient detail layer. And finally, superposing the detail layer on the base layer to generate a final fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method for fusion of infrared and visible light images with brightness compensation and texture enhancement. Background Technology

[0002] With the continuous development of digital image processing technology, infrared and visible light image fusion technology is also constantly evolving. Infrared images reflect the radiation information of a scene, while visible images reflect the reflected light information. Fusing and complementing the two allows us to see both surface reflection information, such as texture and color, and the scene's radiation information, enabling observation from multiple dimensions. Currently, commonly used infrared and visible light image fusion methods are based on multi-scale approaches. This involves decomposing the image into a base layer and a detail layer, applying different fusion rules to each layer. Typically, the base layer uses the average value, while the detail layer uses values ​​with large gradients. This can lead to problems such as reduced brightness of infrared targets and blurred textures, thus requiring the design of reasonable fusion rules. For example, Chinese Patent 2019106888777 discloses "An Image Fusion Method Based on Brightness Adaptation and Saliency Detection," with authorization announcement number CN110490914B. This image fusion method achieves separate processing of global targets and details by classifying image brightness and decomposing the image to be fused based on rolling guided filtering. It uses brightness weights and saliency maps to fuse the base layer image and the least squares method to fuse the detail layer image, thereby improving the image fusion effect. Summary of the Invention

[0003] To address the issues of diminished infrared target brightness and blurred visible background texture in fused images, this invention proposes a novel infrared and visible light image fusion method that differs from existing techniques in its brightness compensation and texture enhancement. A brightness layer is introduced to compensate for the brightness of the fused image, while a detail layer combining intensity and gradient weights is introduced to enhance texture.

[0004] The infrared and visible light image fusion method for brightness compensation and texture enhancement described in this invention includes the following steps: A method for fusion of infrared and visible light images with brightness compensation and texture enhancement, characterized by comprising the following steps: 1) Input the registered infrared and visible light 8-bit digital images; 2) The infrared image is decomposed into a set of infrared image detail layers and an infrared image base layer by performing a rolling guided filter; the visible light image is decomposed into a set of visible light image detail layers and a visible light image base layer by performing a rolling guided filter. 3) The infrared image base layer and the visible light image base layer are fused to obtain a fused base layer; 4) Extract the intensity detail layer and gradient detail layer from each infrared image detail layer in a set of infrared image detail layers to obtain the infrared intensity detail layer and infrared gradient detail layer of each infrared image detail layer; extract the intensity detail layer and gradient detail layer from each visible light image detail layer in a set of visible light image detail layers to obtain the visible intensity detail layer and visible gradient detail layer of a set of visible light image detail layers; fuse the infrared intensity detail layer with the corresponding visible intensity detail layer to generate a fused intensity detail layer, fuse the infrared gradient detail layer with the corresponding visible gradient detail layer to generate a fused gradient detail layer, and weight the fused intensity detail layer and the fused gradient detail layer to generate a fused detail layer; 5) The final fused image is obtained by overlaying several fusion detail layers on the fusion base layer.

[0005] Preferably, the method for decomposing an infrared or visible light image into multiple detail layers and a base layer using rolling guided filtering is as follows: Let the rolling guided filter operator be... Where I is the source image, σs is the standard deviation used as a scale parameter, σr is the range used to control the weights, taking a value of 0.01–0.1, and T is the number of iterations, taking a value of 3–5; the source image is decomposed into multiple detail layers and a base layer using the following formula: , , , , In the formula: It is the output image of the j-th layer filter. ; It is the j-th detail layer, and N is the number of decomposed layers; the source image is an infrared image or a visible light image.

[0006] Preferably, the method for fusing the infrared image base layer and the visible light image base layer is as follows: Calculate the grayscale saliency V of the infrared image IR The feature extraction weight map W of the infrared image is calculated using the following formula, based on the grayscale saliency map Vvis of the visible light image and the grayscale saliency map Vvis of the visible light image. IR W IR =0.5+0.5*(V IR -Vvis) The feature extraction weight map W of the visible light image is calculated using the following formula. VIS WVIS =1-W IR Weight map W extracted from infrared images IR Feature extraction weight map W of visible light image VIS Features are extracted from the base layer of the infrared image and the base layer of the visible light image respectively to generate a fused feature layer S(i,j); The fused brightness layer is calculated using a local energy strategy as the brightness extraction function. The fusion feature layer and the fusion brightness layer are fused to obtain the fusion base layer.

[0007] Preferably, the grayscale saliency V of the infrared image IR The grayscale saliency map Vvis of the visible light image is calculated using the following method. a. Histogram accumulation: For each gray value i in the source image, calculate the squared difference between it and all other gray values, multiply it by its frequency, and accumulate them to obtain the significance score of that gray value, which is its feature significance value s[i]. The formula is as follows: Where f[j] is the frequency of gray value j, i.e. the value of the histogram; b. Feature significance value normalization: The calculated feature significance value s[i] is normalized to the range of 0-255 to obtain the normalized feature significance value S[i], as shown in the following formula: Among them, s min and s max These are the minimum and maximum values ​​of s[i], respectively; Assign a saliency value to each pixel: For each pixel in the source image, assign a corresponding normalized feature saliency value S[i] based on its grayscale value i, and obtain the grayscale saliency map V of the infrared image. IR And the grayscale saliency map Vvis of the visible light image; The feature extraction from the infrared image base layer and the visible light image base layer, and the generation of the fused feature layer S(i,j) are calculated according to the following formula. In the above formula, Represents the infrared base layer. Represents the visible base layer.

[0008] Preferably, the method for calculating the fused luminance layer using a local energy strategy as a luminance extraction function is as follows: The infrared and visible base layers are divided into multiple local regions, resulting in multiple infrared base layer local regions and multiple visible base layer local regions. Local brightness is calculated using the sum of squared pixel grayscale values ​​within either the infrared or visible base layer local regions. The local brightness of the infrared base layer local region is compared with that of the corresponding visible base layer local region. The grayscale value of the corresponding pixel in the fused brightness layer is selected as the base layer grayscale value of the pixel with the larger local brightness, resulting in the fused brightness layer as shown in the following formula. ; The fusion base layer is obtained by fusing the fusion feature layer and the fusion brightness layer according to the following formula. .

[0009] Preferably, in step 4), a weighted mapping rule combining intensity features and gradient features is used to merge detail layers, wherein the intensity features utilize local intensity energy as an activity level metric, i.e., DLI, to obtain salient structure from a brightness perspective; DLI is defined as: in Intensity weighting diagram Calculate according to the following formula, The blending strength detail layer is represented as follows: Using the local average gradient energy (DLG) as another measure of activity level in the spatial domain, rich detailed structural information contained in the detail layers is extracted; the local average gradient energy (DLG) is calculated according to the following formula. Gradient weight graph Calculate according to the following formula, The fused gradient detail layer is represented as Each blended detail layer image Represented as .

[0010] The advantages of this invention compared to the prior art are: This invention improves the saliency of infrared targets in fused images by performing brightness compensation and enhances the naturalness of visible background textures.

[0011] This invention introduces an energy factor to extract a brightness layer from the base layer, compensating for infrared brightness and making infrared targets more prominent in the fused image.

[0012] This invention provides a new detail fusion rule that combines intensity and gradient weights, resulting in more prominent textures in the fused detail layers. Attached Figure Description

[0013] Figure 1 This is a flowchart of one embodiment of the present invention.

[0014] Figure 2 This is a comparison image between the image generated by the fusion method of this invention and the image generated by the fusion method of the prior art. Detailed Implementation

[0015] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0016] This invention provides a method for fusion of infrared and visible light images with brightness compensation and texture enhancement, used to achieve image fusion with both brightness compensation and texture enhancement. The process of this method is as follows: Figure 1 As shown, it includes the following steps: 1) Input the registered infrared and visible light 8-bit digital images; 2) The infrared image is decomposed into a set of infrared image detail layers and an infrared image base layer by performing a rolling guided filter; the visible light image is decomposed into a set of visible light image detail layers and a visible light image base layer by performing a rolling guided filter. In this step, the method for decomposing an infrared or visible light image into multiple detail layers and a base layer using rolling guided filtering is as follows: Let the rolling guided filter operator be... Where I is the source image, σs is the standard deviation used as a scale parameter, σr is the range used to control the weights, taking a value of 0.01–0.1, and T is the number of iterations, taking a value of 3–5; the source image is decomposed into multiple detail layers and a base layer using the following formula: , , , , In the formula: It is the output image of the j-th layer filter. . It is the j-th detail layer, and N is the number of decomposed layers; the source image is an infrared image or a visible light image.

[0017] The infrared image base layer and the visible light image base layer are fused to obtain a fused base layer; First, calculate the grayscale saliency V of the infrared image according to the following steps. IR And the grayscale saliency map Vvis of the visible light image, a. Histogram accumulation: For each gray value i in the source image, calculate the squared difference between it and all other gray values, multiply it by its frequency, and accumulate them to obtain the significance score of that gray value, which is its feature significance value s[i]. The formula is as follows: Where f[j] is the frequency of gray value j, i.e. the value of the histogram; b. Feature significance value normalization: The calculated feature significance value s[i] is normalized to the range of 0-255 to obtain the normalized feature significance value S[i], as shown in the following formula: Among them, s min and s max These are the minimum and maximum values ​​of s[i], respectively; Assign a saliency value to each pixel: For each pixel in the source image, assign a corresponding normalized feature saliency value S[i] based on its grayscale value i, and obtain the grayscale saliency map V of the infrared image. IR And the grayscale saliency map Vvis of the visible light image; Subsequently, the feature extraction weight map W of the infrared image is calculated using the following formula. IR W IR =0.5+0.5*(V IR -Vvis) The feature extraction weight map W of the visible light image is calculated using the following formula. VIS W VIS =1-W IR Weight map W extracted from infrared images IR Feature extraction weight map W of visible light image VIS Features are extracted from the base layer of the infrared image and the base layer of the visible light image respectively to generate a fused feature layer S(i,j); The fused feature layer S(i,j) is calculated according to the following formula. In the above formula, Represents the infrared base layer. Represents the visible base layer.

[0018] Subsequently, the fused brightness layer was calculated using a local energy strategy as the brightness extraction function; The infrared base layer and the visible base layer are divided into multiple local regions. For example, the infrared brightness layer or the visible brightness layer can be divided into 2*2, 3*3, or more local regions, resulting in multiple local regions for the infrared brightness layer and multiple local regions for the visible brightness layer. The local brightness is calculated using the sum of squared pixel grayscale values ​​within the local regions of the infrared base layer or the visible base layer. The local brightness of the local regions of the infrared base layer and the corresponding local regions of the visible base layer are compared. The grayscale value of the corresponding pixel in the fused brightness layer is selected as the base layer grayscale value of the pixel with the larger local brightness, resulting in the fused brightness layer as shown in the following formula. ; The fusion feature layer and the fusion brightness layer are fused to obtain the fusion base layer; the fusion base layer is obtained according to the following formula. .

[0019] For each infrared image detail layer in a set of infrared image detail layers, extract its intensity detail layer and gradient detail layer to obtain the infrared intensity detail layer and infrared gradient detail layer of each infrared image detail layer; for each visible light image detail layer in a set of visible light image detail layers, extract its intensity detail layer and gradient detail layer to obtain the visible intensity detail layer and visible gradient detail layer of the set of visible light image detail layers; fuse the infrared intensity detail layer with the corresponding visible intensity detail layer to generate a fused intensity detail layer, fuse the infrared gradient detail layer with the corresponding visible gradient detail layer to generate a fused gradient detail layer, and weight the fused intensity detail layer and the fused gradient detail layer to generate a fused detail layer; In this step, a weighted mapping rule combining intensity features and gradient features is used to merge detail layers. The intensity features utilize local intensity energy as a measure of activity level, i.e., DLI, to obtain salient structure from a brightness perspective; DLI is defined as: in Intensity weighting diagram Calculate according to the following formula, The blending strength detail layer is represented as follows: Using the local average gradient energy (DLG) as another measure of activity level in the spatial domain, rich detailed structural information contained in the detail layers is extracted; the local average gradient energy (DLG) is calculated according to the following formula. Gradient weight graph Calculate according to the following formula, The fused gradient detail layer is represented as Each blended detail layer image Represented as .

[0020] 5) The final fused image is obtained by overlaying several fusion detail layers on the fusion base layer.

[0021] The invention will now be described in detail using W*H resolution infrared and visible 8-bit images.

[0022] 1) Input the registered infrared and visible light 8-bit digital images; 2) The infrared and visible light images are decomposed into multiple detail layers and one base layer by rolling guided filtering, resulting in multiple infrared detail layers, one infrared base layer, multiple visible detail layers, and one visible base layer; The Rolling Guided Filter (RGF) has edge-preserving and scale-aware properties, therefore it is used to decompose multi-source images. Let the Rolling Guided Filter operator be... Where I is the input image, σs is the standard deviation used as a scale parameter, and σr is used to control the range of weights. In this embodiment, σr = 0.05 is taken. Of course, in other embodiments of the present invention, σr = 0.01 or σr = 0.1 can also be taken. T is the number of iterations. Here, T = 4 is taken. In other implementations, T = 3 or T = 5 may also be taken.

[0023] The source image used for infrared or visible light images can be decomposed into multiple scales using the following formula: In the formula: It is the output image of the j-th layer filter. . This is the j-th detail layer. N is the number of layers to be decomposed. Using the above formula, the visible light image and the infrared image are decomposed into corresponding detail layers and base layers, respectively, that is, multiple infrared detail layers, one infrared base layer, multiple visible detail layers, and one visible base layer.

[0024] The infrared base layer and the visible base layer are fused; this step first calculates the grayscale saliency V of the infrared image. IR And the grayscale saliency map Vvis of the visible image, The grayscale saliency V of the infrared image IR The grayscale saliency map Vvis of the visible image is calculated using the following method. Histogram accumulation: For each gray value i in the image, calculate its feature significance value s[i], as shown in the following formula: Where f[j] is the frequency of gray value j, i.e., the value of the histogram. This formula calculates the squared difference between each gray value and all other gray values, multiplies it by its respective frequency, and sums them to obtain the significance score of that gray value.

[0025] b. Feature significance value normalization: The calculated feature significance value s[i] is normalized to the range of 0-255, as shown in the following formula: Among them, s min and s max These are the minimum and maximum values ​​of s[i], respectively.

[0026] Assign a saliency value to each pixel: For each pixel in the image, assign a corresponding saliency value s[i] based on its grayscale value i to obtain a saliency map.

[0027] Then, the infrared feature extraction weight map W is generated according to the following formula. IR , W IR =0.5+0.5*(V IR -Vvis) Generate the visible light feature extraction weight map according to the following formula. W VIS =1-W IR Features are extracted from the infrared and visible base layers using saliency maps to generate a fused feature layer S(i,j); remember and This represents the base layer image after filtering the source image. The saliency map is multiplied and added to the filtered image to obtain the saliency feature layer, as shown in the following formula: Local energy is used as the brightness extraction function. Local energy refers to the sum of the squares of the gray values ​​of all pixels in a specified local region. The magnitude of the local energy reflects the light intensity in a local area of ​​the image. The larger the local energy value, the greater the light intensity in the current region. Therefore, local energy is used as the brightness extraction function to preserve light intensity. Local energy is defined by the following formula: Local brightness is calculated using pixels within a 3 × 3 region. The blended brightness depends on the larger local brightness between the two images, as shown below. Therefore, the fusion base layer can be represented as .

[0028] 4) Extract the intensity detail layer and gradient detail layer from each detail layer, and merge them to generate a new detail layer; The structural features of the detail layers typically reveal the salient edges and contours of an object. To further enhance detail, a weighted mapping rule combining intensity and gradient features is used to merge the detail layers. The infrared region primarily reflects the target's brightness, therefore an intensity detail layer is employed. The background region reflects detailed texture and is displayed by a gradient detail layer. Local intensity energy (DLI) is used as a measure of activity level to extract salient structure from the brightness aspect. DLI is defined as: Intensity weighting diagram Calculate according to the formula. The intensity detail layer is represented as: Local average gradient energy (DLG), expressed as the sum of squared differences of the average Laplacian operator, is used as another measure of activity level in the spatial domain to extract rich detailed structural information contained in the detail layers. Based on gradient weight graph Calculate according to the formula. Gradient detail layer is represented as Finally, the image of each detail layer is obtained using the following formula. Merge detail layers according to formula 5) Utilize the integration of grassroots and blending detail layers The final fused image is obtained. .

[0029] in, It is a fusion base layer. It is a layer of blending details.

[0030] The table below shows the objective evaluation scoring table for the fusion method of this invention compared to DLF, GTF, and MGF. MI SF SD CC This invention 2.021601739 13.6064472 37.0627911 0.062251798 DLF 1.552983824 7.903977032 25.71935014 0.040558159 GTF 1.090376955 13.06924343 30.74409502 0.041047905 MGF 1.361429325 12.96679873 34.69093723 0.054304766 DLF refers to the existing deep learning fusion method, GTF to the gradient transfer fusion method, and MGF to the multi-scale guided filtering fusion method. Mutual information (MI), spatial frequency (SF), standard deviation (SD), and contrast (CC) are used to measure the fusion effect. Mutual information represents the degree of transfer between the fused image and the source image. Spatial frequency and standard deviation measure the richness of information in the fused image. Contrast represents the visual effect of the fused image. It can be seen that the fusion method of this invention scores higher than the other three methods in all four metrics.

[0031] In addition, such as Figure 2 As shown, image (a) is an infrared image, image (b) is a light-emitting image, image (c) is an image generated using a deep learning fusion method, image (d) is an image generated using a gradient transfer fusion method, image (e) is an image generated using a multi-scale guided filtering fusion method, and image (f) is an image generated using the fusion method of this invention. Comparing images (a)-(f), it can be seen that the image generated by the fusion method of this invention has a brighter target and a more natural background texture.

Claims

1. A method for fusion of infrared and visible light images with brightness compensation and texture enhancement, characterized in that, Includes the following steps: 1) Input the registered infrared and visible light 8-bit digital images; 2) The infrared image is decomposed into a set of infrared image detail layers and an infrared image base layer by performing a rolling guided filter; the visible light image is decomposed into a set of visible light image detail layers and a visible light image base layer by performing a rolling guided filter. 3) The infrared image base layer and the visible light image base layer are fused to obtain a fused base layer; 4) Extract the intensity detail layer and gradient detail layer from each infrared image detail layer in a set of infrared image detail layers to obtain the infrared intensity detail layer and infrared gradient detail layer of each infrared image detail layer; extract the intensity detail layer and gradient detail layer from each visible light image detail layer in a set of visible light image detail layers to obtain the visible intensity detail layer and visible gradient detail layer of a set of visible light image detail layers; fuse the infrared intensity detail layer with the corresponding visible intensity detail layer to generate a fused intensity detail layer, fuse the infrared gradient detail layer with the corresponding visible gradient detail layer to generate a fused gradient detail layer, and weight the fused intensity detail layer and the fused gradient detail layer to generate a fused detail layer; 5) The final fused image is obtained by overlaying several fusion detail layers on the fusion base layer.

2. The infrared and visible light image fusion method for brightness compensation and texture enhancement according to claim 1, characterized in that, The method for decomposing an infrared or visible light image into multiple detail layers and a base layer using rolling guided filtering is as follows: Let the rolling guided filter operator be... Where I is the source image, σs is the standard deviation used as a scale parameter, σr is the range used to control the weights, taking a value of 0.01–0.1, and T is the number of iterations, taking a value of 3–5; the source image is decomposed into multiple detail layers and a base layer using the following formula: ， ， ， ， In the formula: It is the output image of the j-th layer filter. ; It is the j-th detail layer, and N is the number of decomposed layers; the source image is an infrared image or a visible light image.

3. The infrared and visible light image fusion method for brightness compensation and texture enhancement according to claim 1 or 2, characterized in that, The method for fusing the infrared image base layer and the visible light image base layer is as follows: Calculate the grayscale saliency V of the infrared image IR The feature extraction weight map W of the infrared image is calculated using the following formula, based on the grayscale saliency map Vvis of the visible light image and the grayscale saliency map Vvis of the visible light image. IR IN IR =0.5+0.5*(V IR -Vvis) The feature extraction weight map W of the visible light image is calculated using the following formula. VIS IN VIS =1-W IR Weight map W extracted from infrared images IR Feature extraction weight map W of visible light image VIS Features are extracted from the base layer of the infrared image and the base layer of the visible light image respectively to generate a fused feature layer S(i,j); The fused brightness layer is calculated using a local energy strategy as the brightness extraction function. The fusion feature layer and the fusion brightness layer are fused to obtain the fusion base layer.

4. The infrared and visible light image fusion method for brightness compensation and texture enhancement according to claim 3, characterized in that, The grayscale saliency V of the infrared image IR The grayscale saliency map Vvis of the visible light image is calculated using the following method. a. Histogram accumulation: For each gray value i in the source image, calculate the squared difference between it and all other gray values, multiply it by its frequency, and sum them to obtain the significance score of that gray value, which is its eigenvalue. s[i], the formula is as follows: Where f[j] is the frequency of gray value j, i.e. the value of the histogram; b. Feature significance value normalization: The calculated feature significance value s[i] is normalized to the range of 0-255 to obtain the normalized feature significance value S[i], as shown in the following formula: Among them, s min and s max These are the minimum and maximum values ​​of s[i], respectively; Assign a saliency value to each pixel: For each pixel in the source image, assign a corresponding normalized feature saliency value S[i] based on its grayscale value i, and obtain the grayscale saliency map V of the infrared image. IR And the grayscale saliency map Vvis of the visible light image; The feature extraction from the infrared image base layer and the visible light image base layer, and the generation of the fused feature layer S(i,j) are calculated according to the following formula. In the above formula, Represents the infrared base layer. Represents the visible base layer.

5. The infrared and visible light image fusion method for brightness compensation and texture enhancement according to claim 3, characterized in that, The method for calculating the fused brightness layer using a local energy strategy as a brightness extraction function is as follows: The infrared and visible base layers are divided into multiple local regions, resulting in multiple infrared base layer local regions and multiple visible base layer local regions. Local brightness is calculated using the sum of squared pixel grayscale values ​​within either the infrared or visible base layer local regions. The local brightness of the infrared base layer local region is compared with that of the corresponding visible base layer local region. The grayscale value of the corresponding pixel in the fused brightness layer is selected as the base layer grayscale value of the pixel with the larger local brightness, resulting in the fused brightness layer as shown in the following formula. ； The fusion base layer is obtained by fusing the fusion feature layer and the fusion brightness layer according to the following formula. 。 6. The infrared and visible light image fusion method for brightness compensation and texture enhancement according to claim 1 or 2, characterized in that, In step 4), a weighted mapping rule combining intensity features and gradient features is used to merge detail layers. The intensity features utilize local intensity energy as a measure of activity level, i.e., DLI, to obtain salient structure from a brightness perspective. DLI is defined as: in Intensity weighting diagram Calculate according to the following formula, The blending strength detail layer is represented as follows: Using the local average gradient energy (DLG) as another measure of activity level in the spatial domain, rich detailed structural information contained in the detail layers is extracted; the local average gradient energy (DLG) is calculated according to the following formula. Gradient weight graph Calculate according to the following formula, The fused gradient detail layer is represented as Each blended detail layer image Represented as 。

Images