Infrared and visible light image fusion method based on adaptive gamma transform decomposition

Abstract

The invention provides an infrared and visible light image fusion method based on self-adaptive gamma transform decomposition, which comprises the following steps: firstly, inputting an infrared and visible light image, and designing a self-adaptive gamma transform decomposition strategy to decompose a source image into a series of background and target layers; secondly, denoising the infrared background layer by adopting an improved Gaussian filter, and obtaining an infrared and visible light background layer fusion weight according to texture complexity; secondly, designing a non-subsampled pyramid to respectively extract global and local visual saliency maps of the infrared and visible light target layers, and further fusing the target layers through visual saliency features; and finally, adding the fused background layer and the target layer to obtain a fused image. According to the method, complementary information contained in the infrared image and the visible light image can be fully fused, the fused image with rich information is obtained, and the defect of a single type of image is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information fusion, and in particular to an infrared and visible light image fusion method based on adaptive gamma transform decomposition. Background Art

[0002] Due to the wavelength and technical limitations of imaging equipment, images captured by a single type of sensor are difficult to effectively and comprehensively describe the scene. Therefore, image fusion technology is gradually emerging, and its purpose is to generate a fused image by combining the complementary information of the source images to enhance the understanding of the scene. Among them, infrared and visible light image fusion technology is most widely used in the engineering field. Among them, infrared sensors can capture the thermal radiation information of the scene, so they are not easily affected by noise interference, but lack texture information. In contrast, visible light sensors record the reflected light of the scene, thereby producing visible light images that contain rich texture information but are susceptible to extreme environments. By combining the complementary information of infrared and visible light images, a fused image with important targets and rich detail information can be obtained. Therefore, infrared and visible light image fusion technology is widely used in military surveillance, forest fire prevention, target rescue and other fields.

[0003] In recent years, scholars have proposed many infrared and visible light image fusion methods. Generally, the fusion process of these methods can be expressed as: F = R (E (I 1 ,I 2 ,...,I n ), where F is the fusion result, F represents the complementary information extraction operation, and I n Represents the source image, and R represents the fusion rule. Therefore, in the field of infrared and visible light image fusion, two key tasks are to accurately extract complementary information and formulate effective fusion rules. In order to effectively realize infrared and visible light image fusion, one of the most popular solutions is to first decompose the source image into different parts, and then extract background and target features. Then, design specific fusion rules to integrate the extracted features. Finally, reconstruct the fused image. This method not only eliminates the influence of other information on the feature extraction and fusion process, but also simplifies the complexity and difficulty of image fusion.

[0004] However, existing fusion methods always use multi-scale transformation tools or edge-preserving filters to decompose the source image into background and target parts, which is essentially based on the intensity difference between adjacent pixels. Unfortunately, due to the similarity in the intensity difference characteristics between high-contrast textures in the background and prominent edges in the target part, this method always decomposes a lot of details into the target part, which may cause blurred texture information in the fused image. In addition, although many fusion methods focus on designing fusion rules for background information, the balanced ratio of infrared and visible background parts has not been effectively formulated. Not only that, compared with the background fusion rules, less attention is paid to the fused target part. Most infrared and visible image fusion methods simply combine the target parts together through an averaging rule, which may result in low contrast in the fused image. Summary of the invention

[0005] According to the technical problems raised above, a method for fusion of infrared and visible light images based on adaptive gamma transform decomposition is provided. The present invention first designs an adaptive curve decomposition method to obtain a series of background and target parts from the source image. In addition, the infrared background usually contains a lot of noise and invalid information, while the background of the visible light image has rich texture information. Therefore, when fusing the background part, it is necessary to reduce the infrared noise and retain more visible detail information. Secondly, the present invention proposes an infrared texture clustering filter to remove the noise of the infrared background part, and then introduces a balance factor to obtain the fused background part. Finally, since the target feature is an object of great interest to the human visual system, a non-subsampled visual saliency model is designed to merge the target part, which can effectively avoid the contrast loss in the fused image.

[0006] The technical means adopted by the present invention are as follows:

[0007] A method for fusion of infrared and visible light images based on adaptive gamma transform decomposition, comprising:

[0008] S1. Input infrared and visible light images, and design an adaptive gamma transform decomposition strategy to decompose the source image into a series of background and target layers;

[0009] S2, using an improved Gaussian filter to denoise the infrared background layer, and obtaining the background layer fusion weight according to the texture complexity;

[0010] S3. Design a non-subsampled pyramid to extract the global and local visual saliency maps of the infrared and visible light target layers respectively, and use the saliency maps to fuse the target layers.

[0011] S4. Add the fused background layer and target layer to obtain a fused image.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. Normalize the infrared and visible light images to 0-1. The expression is as follows:

[0014] D n =Nor(I n )

[0015] Among them, I n Represents the source image; Nor() represents the normalization operation; D n represents the normalized image, n∈{1,2};

[0016] S12. The normalized image is stretched using gamma change and decomposed into a series of background and target layers. The expression is as follows:

[0017]

[0018] in, Represents the background layer, represents the target layer, represents the exponential factor;

[0019] S13, using image variance and grayscale mean to adaptively calculate the exponential factor, the expression is as follows:

[0020]

[0021] Among them, var() calculates the image variance and μ() calculates the image mean.

[0022] Further, step S2 specifically includes:

[0023] S21, select the maximum and minimum values ​​of the infrared background part as the initial cluster center, and calculate the distance between each pixel and the initial cluster center. The calculation formula is as follows:

[0024]

[0025] Among them, E represents the distance from each pixel to the cluster center, Ck represents the cluster center, and F 1 l (x,y) represents the infrared background layer;

[0026] S22, compare the distance between each pixel and the Kth cluster center, and divide it into cluster centers with close distances to form initial clusters; then, calculate the mean of pixels in the same cluster, and compare it with the K cluster centers calculated in the previous step. If the cluster center is different from the mean, use the mean as the cluster center, and repeatedly use the distance calculation formula between each pixel and the initial cluster center to divide the clusters until it no longer changes;

[0027] S23. After dividing each pixel into different clusters, an improved Gaussian filter model is designed to remove noise in the infrared background layer. The expression is as follows:

[0028]

[0029] in, represents the denoised image, (p,q) represents the neighborhood pixels with (x,y) as the center of the filter window, σ represents the filter parameter, and Represent the cluster center values ​​of the center point and neighborhood pixels respectively;

[0030] S24. Calculate the texture complexity of the denoised image and the visible light background layer. The expression is as follows:

[0031]

[0032] Among them, M and N represent the image size, BF 1 l and Represent the texture complexity of the denoised image and the visible light background layer respectively;

[0033] S25. According to the determined texture complexity, the fused background layer is obtained. The calculation formula is as follows:

[0034]

[0035] in, represents the fused background layer, α l and β l Represent the infrared and visible light fusion coefficients respectively.

[0036] Furthermore, step S3 specifically includes:

[0037] S31. Construct a non-subsampled Gaussian pyramid. The calculation formula is as follows:

[0038]

[0039] Among them, G(x,y) represents a Gaussian filter, In the above formula, Represents the input image for the next filtering;

[0040] S32. Use the upper part of the non-subsampled Gaussian pyramid to obtain the local visual saliency map of the target layer. The calculation formula is as follows:

[0041]

[0042] Among them, DB n (x,y) represents the local visual saliency map;

[0043] S33, use the lower half of the non-subsampled Gaussian pyramid to obtain the global visual saliency map of the target layer, and the calculation formula is as follows:

[0044]

[0045] Among them, DQ n (x,y) represents the local visual saliency map;

[0046] S34, adding the local and global visual salient images to obtain a visual salient image, and the calculation formula is as follows:

[0047] SM n (x,y)=DB n (x,y)+DQ n (x,y)

[0048] Among them, SM n (x,y) represents the visual saliency map;

[0049] S35. Obtain the target layer fusion result by comparing the visual saliency map. The calculation formula is as follows:

[0050]

[0051] Further, step S4 specifically includes:

[0052] The fused image is obtained by using the fused background and target layers. The calculation formula is as follows:

[0053]

[0054] Among them, Fusion(x,y) represents the fusion result.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] 1. The present invention provides an infrared and visible light image fusion method based on adaptive gamma transform decomposition, which can fully fuse the complementary information contained in the infrared and visible light images to obtain a fused image with rich information, thus making up for the shortcomings of a single type of image.

[0057] 2. The present invention designs an adaptive curve decomposition method to obtain a series of background and target parts from the source image. In addition, the infrared background usually contains a lot of noise and invalid information, while the background of the visible light image has rich texture information. Therefore, when fusing the background part, it is necessary to reduce the infrared noise and retain more visible detail information.

[0058] 3. The present invention proposes an infrared texture clustering filter to remove the noise of the infrared background part, and then introduces a balance factor to obtain the fused background part.

[0059] 4. The present invention provides an infrared and visible light image fusion method based on adaptive gamma transform decomposition. Since the target features are objects of great interest to the human visual system, a non-subsampled visual saliency model is designed to merge the target parts, which can effectively avoid contrast loss in the fused image.

[0060] Based on the above reasons, the present invention can be widely promoted in the fields of multimodal information fusion and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 The figure is a flow chart of the method of the present invention.

[0063] Figure 2 Infrared and visible light cameras provided by embodiments of the present invention

[0064] Figure 3 This is an infrared image provided by an embodiment of the present invention.

[0065] Figure 4 A visible light image provided by an embodiment of the present invention.

[0066] Figure 5 This is a fused image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] like Figure 1 As shown, the present invention provides an infrared and visible light image fusion method based on adaptive gamma transform decomposition, comprising:

[0070] S1. Use Figure 2 The infrared and visible light cameras shown in the figure take pictures respectively. Figure 3 The infrared image shown and Figure 4 As shown in the visible light image, the infrared and visible light images are input, and an adaptive gamma transform decomposition strategy is designed to decompose the source image into a series of background and target layers;

[0071] S2, using an improved Gaussian filter to denoise the infrared background layer, and obtaining the background layer fusion weight according to the texture complexity;

[0072] S3. Design a non-subsampled pyramid to extract the global and local visual saliency maps of the infrared and visible light target layers respectively, and use the saliency maps to fuse the target layers.

[0073] S4. Add the fused background layer and target layer to obtain a fused image.

[0074] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0075] S11. Normalize the infrared and visible light images to 0-1. The expression is as follows:

[0076] D n =Nor(I n )

[0077] Among them, I n Represents the source image; Nor() represents the normalization operation; D n represents the normalized image, n∈{1,2};

[0078] S12. The normalized image is stretched using gamma change and decomposed into a series of background and target layers. The expression is as follows:

[0079]

[0080] in, Represents the background layer, represents the target layer, represents the exponential factor;

[0081] S13, using image variance and grayscale mean to adaptively calculate the exponential factor, the expression is as follows:

[0082]

[0083] Among them, var() calculates the image variance and μ() calculates the image mean.

[0084] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0085] S21, select the maximum and minimum values ​​of the infrared background part as the initial cluster center, and calculate the distance between each pixel and the initial cluster center. The calculation formula is as follows:

[0086]

[0087] Among them, E represents the distance from each pixel to the cluster center, Ck represents the cluster center, and F 1 l (x,y) represents the infrared background layer;

[0088] S22, compare the distance between each pixel and the Kth cluster center, and divide it into cluster centers with close distances to form initial clusters; then, calculate the mean of pixels in the same cluster, and compare it with the K cluster centers calculated in the previous step. If the cluster center is different from the mean, use the mean as the cluster center, and repeatedly use the distance calculation formula between each pixel and the initial cluster center to divide the clusters until it no longer changes;

[0089] S23. After dividing each pixel into different clusters, an improved Gaussian filter model is designed to remove noise in the infrared background layer. The expression is as follows:

[0090]

[0091] in, represents the denoised image, (p,q) represents the neighborhood pixels with (x,y) as the center of the filter window, σ represents the filter parameter, and Represent the cluster center values ​​of the center point and neighborhood pixels respectively;

[0092] S24. Calculate the texture complexity of the denoised image and the visible light background layer. The expression is as follows:

[0093]

[0094] Among them, M and N represent the image size, BF 1 l and Represent the texture complexity of the denoised image and the visible light background layer respectively;

[0095] S25. According to the determined texture complexity, the fused background layer is obtained. The calculation formula is as follows:

[0096]

[0097] in, represents the fused background layer, α l and β l Represent the infrared and visible light fusion coefficients respectively.

[0098] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0099] S31. Construct a non-subsampled Gaussian pyramid. The calculation formula is as follows:

[0100]

[0101] Among them, G(x,y) represents a Gaussian filter, In the above formula, Represents the input image for the next filtering;

[0102] S32. Use the upper part of the non-subsampled Gaussian pyramid to obtain the local visual saliency map of the target layer. The calculation formula is as follows:

[0103]

[0104] Among them, DB n (x,y) represents the local visual saliency map;

[0105] S33, use the lower half of the non-subsampled Gaussian pyramid to obtain the global visual saliency map of the target layer, and the calculation formula is as follows:

[0106]

[0107] Among them, DQ n (x,y) represents the local visual saliency map;

[0108] S34, adding the local and global visual salient images to obtain a visual salient image, and the calculation formula is as follows:

[0109] SM n (x,y)=DB n (x,y)+DQ n (x,y)

[0110] Among them, SM n (x,y) represents the visual saliency map;

[0111] S35. Obtain the target layer fusion result by comparing the visual saliency map. The calculation formula is as follows:

[0112]

[0113] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes:

[0114] The fused background and target layers are used to obtain Figure 5 The fused image shown is calculated as follows:

[0115]

[0116] Among them, Fusion(x,y) represents the fusion result.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on adaptive gamma transform decomposition, characterized in that: include: S1. Input infrared and visible light images, and design an adaptive gamma transform decomposition strategy to decompose the source image into a series of background and target layers; S2, using an improved Gaussian filter to denoise the infrared background layer, and obtaining the background layer fusion weight according to the texture complexity; S3. Design a non-subsampled pyramid to extract the global and local visual saliency maps of the infrared and visible light target layers respectively, and use the saliency maps to fuse the target layers. S4. Add the fused background layer and target layer to obtain a fused image.

2. The infrared and visible light image fusion method based on adaptive gamma transform decomposition according to claim 1, characterized in that: Step S1 specifically includes: S11. Normalize the infrared and visible light images to 0-1. The expression is as follows: D n =Nor(I n ) Among them, I n Represents the source image; Nor() represents the normalization operation; D n represents the normalized image, n∈{1,2}; S12. The normalized image is stretched using gamma change and decomposed into a series of background and target layers. The expression is as follows: in, Represents the background layer, represents the target layer, represents the exponential factor; S13, using image variance and grayscale mean to adaptively calculate the exponential factor, the expression is as follows: Among them, var() calculates the image variance and μ() calculates the image mean.

3. The infrared and visible light image fusion method based on adaptive gamma transform decomposition according to claim 1, characterized in that: Step S2 specifically includes: S21, select the maximum and minimum values ​​of the infrared background part as the initial cluster center, and calculate the distance between each pixel and the initial cluster center. The calculation formula is as follows: Among them, E represents the distance from each pixel to the cluster center, C k represents the cluster center, Indicates the infrared background layer; S22, compare the distance between each pixel and the Kth cluster center, and divide it into cluster centers with close distances to form initial clusters; then, calculate the mean of pixels in the same cluster, and compare it with the K cluster centers calculated in the previous step. If the cluster center is different from the mean, use the mean as the cluster center, and repeatedly use the distance calculation formula between each pixel and the initial cluster center to divide the clusters until it no longer changes; S23. After dividing each pixel into different clusters, an improved Gaussian filter model is designed to remove noise in the infrared background layer. The expression is as follows: in, represents the denoised image, (p,q) represents the neighborhood pixels with (x,y) as the center of the filter window, σ represents the filter parameter, and Represent the cluster center values ​​of the center point and neighborhood pixels respectively; S24. Calculate the texture complexity of the denoised image and the visible light background layer. The expression is as follows: Among them, M and N represent the image size, and Represent the texture complexity of the denoised image and the visible light background layer respectively; S25. According to the determined texture complexity, the fused background layer is obtained. The calculation formula is as follows: in, represents the fused background layer, α l and β l Represent the infrared and visible light fusion coefficients respectively.

4. The method for fusion of infrared and visible light images based on adaptive gamma transform decomposition according to claim 1, characterized in that: Step S3 specifically includes: S31. Construct a non-subsampled Gaussian pyramid. The calculation formula is as follows: Among them, G(x,y) represents a Gaussian filter, In the above formula, Represents the input image for the next filtering; S32. Use the upper part of the non-subsampled Gaussian pyramid to obtain the local visual saliency map of the target layer. The calculation formula is as follows: Among them, DB n (x,y) represents the local visual saliency map; S33, use the lower half of the non-subsampled Gaussian pyramid to obtain the global visual saliency map of the target layer, and the calculation formula is as follows: Among them, DQ n (x,y) represents the local visual saliency map; S34, adding the local and global visual salient images to obtain a visual salient image, and the calculation formula is as follows: SM n (x,y)=DB n (x,y)+DQ n (x,y) Among them, SM n (x,y) represents the visual saliency map; S35. Obtain the target layer fusion result by comparing the visual saliency map. The calculation formula is as follows:

5. The infrared and visible light image fusion method based on adaptive gamma transform decomposition according to claim 1, characterized in that: Step S4 specifically includes: The fused image is obtained by using the fused background and target layers. The calculation formula is as follows: Among them, Fusion(x,y) represents the fusion result.

Images