A method for fusing visible light and short-wave infrared images

Abstract

The application provides a visible light and short-wave infrared image fusion method, S1: obtaining a short-wave infrared source image ISWIR and a visible light source image IVIS, and taking a luminance channel image of the visible light source image IVIS as a visible light image Y to be fused; S2: obtaining weight maps WSWIR and WY of the short-wave infrared source image ISWIR and the visible light image Y according to the short-wave infrared source image ISWIR and the visible light image Y; S3: constructing an image pyramid to perform image fusion. The scheme designs a local texture and gradient contrast function LTG(I(x, y)), which can extract the texture unique in the short-wave infrared image and weak in the visible light image and can enhance the image fusion effect when the image is fused.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method for fusing visible light and shortwave infrared images. Background Technology

[0002] Image fusion refers to image processing of images of the same scene obtained from multiple sensors, combining the advantages of different sources and fusing their rich information to obtain a high-quality image with good visual effects and rich visual information. Currently, in the field of visible light and infrared image fusion, the most extensive research object is the fusion algorithm of visible light images and thermal imaging images, while research on image fusion algorithms for other infrared bands is relatively limited.

[0003] Short-wave infrared, due to its unique wavelength range and imaging principle similar to visible light, can penetrate clouds and fog to form images, and it also has unique advantages in material identification. Therefore, fusing visible light and short-wave infrared images is of certain significance. Summary of the Invention

[0004] Based on the problems existing in the prior art, the present invention aims to solve the technical problems of the prior art...

[0005] This invention provides a method for fusing visible light and shortwave infrared images, comprising:

[0006] S1: Acquire shortwave infrared source image I SWIR and visible light source image I VIS Image I of visible light source VIS Convert from RGB format to YUV format, and take the luminance channel image as the visible light image Y to be fused;

[0007] S2: Based on the shortwave infrared source image I SWIR From the visible light image Y, obtain the short-wave infrared source image I. SWIR Weighted map W of visible light image Y SWIR W Y ,include:

[0008] S21: Calculate the shortwave infrared source image I SWIR Texture richness LTG SWIR (x,y) and visible light source image I VIS Texture richness LTG VIS (x,y);

[0009] S22: Determine the shortwave infrared source image I SWIR The initial weights S(x, y) and adjustment factors H(x, y);

[0010] S23: Transfer the shortwave infrared source image I SWIRThe shortwave infrared source image I is obtained by multiplying the initial weights S(x,y) and the adjustment factor H(x,y). SWIR The initial weighted graph W' SWIR (x,y);

[0011] S24: For the initial weight map W' SWIR Normalize (x,y) to obtain the shortwave infrared source image I. SWIR Normalized weighted graph W SWIR Simultaneously, the normalized weight map W of the visible light image Y is obtained. Y ;

[0012] S3: Based on the shortwave infrared source image I SWIR Visible light image Y, normalized weighted image W SWIR and W Y Construct an image pyramid for image fusion.

[0013] According to an embodiment of the present invention, the shortwave infrared source image I is calculated. SWIR and visible light source image I VIS The formula for the richness of texture is:

[0014]

[0015] Where α is a coefficient used to adjust the ratio of image texture to image gradient; LC(x',y') represents the local contrast of the image, GD(x',y') represents the gradient value of the image; N(x,y) represents the local neighborhood of the image centered at (x,y), the neighborhood is set to 5×5, and the total number of pixels is N; mean is the local pixel mean of the image. It is represented as the second reciprocal of the x-direction of the image (x, y). It is represented as the second reciprocal of the image (x, y) in the y-direction;

[0016] According to one embodiment of the present invention, the proportionality coefficient α is 0.5.

[0017] According to an embodiment of the present invention, the shortwave infrared source image I SWIR The initial weighted graph W' SWIR The formula for calculating (x,y) is:

[0018] W′ SWIR (x,y)=H(x,y)*S(x,y);

[0019] For the initial weight map W' SWIR Normalize (x,y) to obtain the shortwave infrared source image I. SWIR Normalized weighted graph W SWIR The formula is:

[0020]

[0021] W Y =1-W SWIR ;

[0022] Among them, W' SWIR (x,y)∣ max With W' SWIR (x,y)∣ min W' SWIR The maximum and minimum values ​​of (x,y).

[0023] According to an embodiment of the present invention, in step S22, the shortwave infrared source image I is determined. SWIR The initial weights S(x, y) include: determining the LTG SWIR (x,y)≧LTG VIS If (x,y) holds true, then let Otherwise, S(x, y) is 0.

[0024] According to an embodiment of the present invention, in step S22, the shortwave infrared source image I is determined. SWIR The regulation factor H(x, y) includes: including: judging I SWIR (x,y)≥I VIS If (x,y) holds true, then let Otherwise, H(x, y) takes the value 1; where I SWIR (x,y) represents the shortwave infrared source image I. SWIR The pixel value at (x, y), I VIS (x,y) represents the pixel value at (x,y) in the visible light source image.

[0025] According to an embodiment of the present invention, in step S3, based on the shortwave infrared source image I SWIR Visible light image Y, normalized weighted image W SWIR and W Y Constructing an image pyramid for image fusion includes:

[0026] Constructing the shortwave infrared source image I SWIR Image pyramids of visible light image Y and visible light image Y are used to obtain short-wave infrared source image I. SWIR And the Laplace pyramid LP of the visible light image Y SWIR (n) LP Y (n) Thus, the LP of the Laplace Pyramid is obtained. SWIR (n) and LP Y (n)Detail layer image of each layer, wherein n=1, 2, 3...

[0027] Constructing the normalized weight map W SWIR and W Y Image pyramid, to obtain the normalized weight map W SWIR and W Y Gaussian pyramid GP SWIR (l) , GP Y (l) , so as to obtain the Laplacian pyramid LP SWIR (n) and LP Y (n) The normalized detail layer weight map corresponding to each layer of detail layer image, wherein l=1, 2, 3...

[0028] The Laplacian pyramid LP SWIR (n) and LP Y (n) Each layer of detail layer image is weighted and fused based on the Gaussian pyramid GP SWIR (l) and GP Y (l) Each layer of normalized detail layer weight map, to obtain the fusion result of each layer of detail layer:

[0029] Fusion_d (n) = LP Y (n) *GP Y (l) + LP SWIR (n) *GP SWIR (l) ;

[0030] The detail layer fusion result is pyramid inverse transformed to obtain the fused image.

[0031] According to an embodiment of the present application, the detail layer fusion result is pyramid transformed to obtain the fused image, comprising:

[0032] The detail layer fusion result Fusion_d (n) is inverse transformed with the base layer of the visible light image Y to obtain the fusion result:

[0033] The fusion result is taken as the luminance channel F Y , and is converted into RGB space together with the color channels U, V of the visible light source image I VIS , so as to obtain the final fusion result.

[0034] The beneficial effects of the present application are:

[0035] The visible light and short wave infrared image fusion method provided by the embodiment of the present application designs a local texture and gradient contrast function LTG(I(x, y)) for representing the texture richness of the short wave infrared source image I SWIR and the visible light source image I VIS , and respectively determines the initial weight S(x, y) and the adjustment factor H(x, y) according to the size of LTG(I(x, y)) and the image pixel value I(x, y), so as to obtain the normalized weight maps of the short wave infrared source image I SWIR and the visible light image Y, which can extract the unique texture in the short wave infrared image and the weak texture in the visible light image, and can enhance the image fusion effect. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 is a step flowchart of the visible light and short wave infrared image fusion method provided by the present application;

[0038] Figure 2 is a result schematic diagram of the visible light and short wave infrared image fusion method provided by the embodiment of the present application in the image processing process. DETAILED DESCRIPTION

[0039] The descriptions of the following embodiments are with reference to the additional drawings to illustrate specific embodiments that can be used to implement the present application. The directions mentioned in the present application, such as [up], [down], [front], [back], [left], [right], [inward], [outward], [side] and the like, are only with reference to the directions of the additional drawings. Therefore, the directions used are used to illustrate and understand the present application, but not to limit the present application. In the drawings, similar structures are denoted by the same reference numerals.

[0040] The present application provides a visible light and short wave infrared image fusion method, the step flowchart of which is shown as Figure 1

[0041] As shown in Figure 2 is a result schematic diagram of the image in the image fusion process of the embodiment of the present application. First, the short wave infrared source image I SWIR and the visible light source image I VIS are obtained, and the visible light source image I​VIS After converting from RGB to YUV format, the visible light source image I is extracted. VIS The brightness channel (Y channel) image is used as the visible light image Y to be fused;

[0042] Secondly, to address the unique details in short-wave infrared images that are lost in visible light images due to rain, fog, and other conditions, a function called Local Texture and Gradient (LTG) was designed to represent the details in the short-wave infrared source image I. SWIR and visible light source image I VIS The formula for calculating the richness of texture is as follows:

[0043]

[0044] Where α is a coefficient used to adjust the ratio of image texture to image gradient; LC(x',y') represents the local contrast of the image, GD(x',y') represents the gradient value of the image; N(x,y) represents the local neighborhood of the image centered at (x,y), the neighborhood is set to 5×5, and the total number of pixels is N; mean is the local pixel mean of the image. It is represented as the second reciprocal of the x-direction of the image (x, y). It is represented as the second reciprocal of the image (x, y) in the y-direction;

[0045] Then, the shortwave infrared source image I is determined using the LTG function. SEIR The initial weights are given by the following formula:

[0046]

[0047] Meanwhile, due to the imaging differences between short-wave infrared images and visible light images, the short-wave infrared source image I also needs to be considered during detail layer fusion. SEIR and visible light source image I VIS Differences in brightness. Specifically, when the short-wave infrared source image I... SWIR The brightness is greater than that of the visible light source image I VIS When the brightness is high, detail layers with high brightness will affect the visual effect, and their weight should be appropriately reduced; when the short-wave infrared source image I SEIR The brightness is less than that of the visible light source image I VIS When the brightness is high, it is generally a short-wave infrared source image I. SEIR The shadowed areas should have a larger weight to increase the realism of the image. Therefore, this scheme sets an adjustment factor H(x,y) to adjust the weight based on the shortwave infrared source image I. SEIR and visible light source image I VIS Brightness determination of shortwave infrared source image I during image fusion SWIRThe formula for determining the adjustment factor H(x,y) based on the weights is as follows:

[0048]

[0049] Among them, I SWIR (x,y) represents the shortwave infrared source image I. SWIR The pixel value at (x, y), I VIS (x,y) represents the pixel value at (x,y) in the visible light source image.

[0050] Multiplying the initial weight S(x,y) by the adjustment factor H(x,y) yields the shortwave infrared source image I. SWIR The initial weighted graph W' SWIR (x,y), the specific formula is:

[0051] W′ SWIR (x,y)=H(x,y)*S(x,y);

[0052] After normalization, the shortwave infrared source image I is obtained. SWIR Normalized weighted graph W SWIR The formula for the normalized weighted graph is:

[0053]

[0054] Among them, the normalized weighted graph W SWIR The range of pixels in (x,y) is (0,1), W' SWIR (x,y)∣ max With W' SWIR (x,y)∣ min W' SWIR The maximum and minimum values ​​of (x,y);

[0055] Therefore, the normalized weight map of the visible light image Y can be obtained as follows:

[0056] W Y =1-W SWIR .

[0057] Next, based on the shortwave infrared source image I SWIR Visible light image Y, normalized weighted image W SWIR and W Y An image pyramid is constructed for image fusion, and the specific process is as follows:

[0058] First, construct the shortwave infrared source image I. SWIR Image pyramids of visible light image Y and visible light image Y are used to obtain short-wave infrared source image I. SWIR And the Laplace pyramid LP of the visible light image Y SWIR (n), LP Y (n) , thereby obtaining the Laplacian pyramid LP SWIR (n) and LP Y (n) the detail layer image of each layer, wherein n = 1, 2, 3,...

[0059] Then: construct the normalized weight map W SWIR and the image pyramid of W Y , to obtain the Gaussian pyramid GP SWIR of the normalized weight map W Y and W SWIR (l) , GP Y (l) , thereby obtaining the Laplacian pyramid LP SWIR (n) and LP Y (n) the normalized detail layer weight map corresponding to the detail layer image of each layer, wherein l = 1, 2, 3,...

[0060] Next: the Laplacian pyramid LP SWIR (n) and LP Y (n) the detail layer image of each layer is weighted and fused based on the Gaussian pyramid GP SWIR (l) and GP Y (l) the normalized detail layer weight map of each layer, to obtain the fusion result of each layer of the detail layer:

[0061] Fusion_d (n) = LP Y (n) * GP Y (l) + LP SWIR (n) * GP SWIR (l) .

[0062] Finally, the detail layer fusion result is inversely transformed by the pyramid to obtain the fused image. Specifically, the fusion result of the detail layer is inversely transformed with the base layer of the visible light image Y to obtain the fusion result F (n) , and the specific formula is as follows:

[0063]

[0064] F (n) = F (l) + Fusion_d (n),

[0065] Specifically, in the embodiment of the present application, the Gaussian pyramid GP SWIR (l) and GP Y (l) are all 4 layers, i.e. l=4; and l=n+1, i.e. n=3. Then the fusion result of the embodiment is:

[0066]

[0067] F (n) =F (n+1) +Fusion_d (n) , n=3, 2, 1.

[0068] The final F (1) is taken as the luminance channel E Y , and is converted into RGB space together with the color channels U, V of the visible light source image I VIS , thereby obtaining the final fusion result Fusion, and the conversion formula is:

[0069]

[0070] In summary, the visible light and short-wave infrared image fusion method provided by the embodiment of the present application designs a local texture and gradient contrast function LTG(I(x, y)) for representing the texture richness of the short-wave infrared source image I SWIR and the visible light source image I VIS , and respectively determines the initial weight S(x, y) and the adjustment factor H(x, y) according to the size of LTG(I(x, y)) and the image pixel value I(x, y), thereby obtaining the normalized weight maps of the short-wave infrared source image I SWIR and the visible light image Y, which can extract the texture unique to the short-wave infrared image and weak in the visible light image, and can enhance the image fusion effect.

[0071] It should be noted that, although the present application is disclosed as above with specific embodiments, the above embodiments are not used to limit the present application, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application is defined by the scope of the claims.

Claims

1. A method for fusing visible light and shortwave infrared images, characterized in that, include: S1: Acquire shortwave infrared source image and visible light source images Image of visible light source Convert from RGB format to YUV format, and take the luminance channel image as the visible light image Y to be fused; S2: Based on the shortwave infrared source image Combined with the visible light image Y, a short-wave infrared source image is obtained. Normalized weighted map of visible light image Y , ,include: S21: Calculate the shortwave infrared source image Texture richness and visible light source images Texture richness ; S22: Based on shortwave infrared source image Texture richness and visible light source images Texture richness Determine the shortwave infrared source image The initial weights S(x, y) are based on shortwave infrared source images. Visible light source images The pixel values ​​determine the adjustment factor H(x, y); S23: Image of shortwave infrared source The shortwave infrared source image is obtained by multiplying the initial weight S(x,y) and the adjustment factor H(x,y). Initial weight graph ; S24: For the initial weight graph After normalization, the shortwave infrared source image is obtained. Normalized weights graph Simultaneously, the normalized weight map of the visible light image Y is obtained. ; S3: Based on the shortwave infrared source image Visible light image Y, normalized weight map and Construct an image pyramid for image fusion; Calculate shortwave infrared source images and visible light source images The formula for the richness of texture is: ， ， ， in, It is a coefficient used to adjust the ratio of image texture to image gradient; Represents the local contrast of an image. Represents the gradient value of the image; represents the local neighborhood of the image centered at (x, y), with the neighborhood set to 5×5 and the total number of pixels being N; mean is the local pixel mean of the image; It is represented as the second derivative of the image in the x-direction. express The second derivative in the x-direction, It is represented as the second derivative of the image in the y-direction. express The second derivative in the y-direction.

2. The visible light and shortwave infrared image fusion method according to claim 1, characterized in that, The proportionality coefficient The value is 0.

5.

3. The visible light and shortwave infrared image fusion method according to claim 1, characterized in that, The shortwave infrared source image Initial weight graph The calculation formula is: ； For the initial weight graph After normalization, the shortwave infrared source image is obtained. Normalized weights graph The formula is: ， ； in, and They are respectively The maximum and minimum values.

4. The visible light and shortwave infrared image fusion method according to claim 1, characterized in that, In step S22, the shortwave infrared source image is determined. The initial weights S(x, y) include: determining ≧ Is it true? If it is true, then let... ;otherwise, Take 0.

5. The visible light and shortwave infrared image fusion method according to claim 1, characterized in that, In step S22, the shortwave infrared source image is determined. The regulation factor H(x, y) includes: including: judgment Is it true? If it is true, then let... Otherwise, H(x, y) is 1; where, Image of shortwave infrared source The pixel value at (x, y) Image I of a visible light source vis The pixel value at (x, y).

6. The visible light and shortwave infrared image fusion method according to claim 1, characterized in that, In step S3, based on the shortwave infrared source image Visible light image Y, normalized weight map and Constructing an image pyramid for image fusion includes: Constructing the shortwave infrared source image Image pyramids of visible light image Y and visible light image Y are used to obtain short-wave infrared source images respectively. Laplace's pyramid and visible light image Y , Thus, the Pyramid of Laplace was obtained. and The detail layer image for each layer, where n=1,2,3...; Constructing a normalized weight graph and The image pyramid is used to obtain normalized weight maps respectively. and Gauss Pyramid , Thus, we obtain the connection with the Pyramid of Laplace. and The normalized detail layer weight map corresponding to each detail layer image, where l=1,2,3...; The Pyramid of Laplace and The detail layer images of each layer are based on Gaussian pyramids. and The normalized detail layer weight maps of each layer are weighted and fused to obtain the fusion result of each detail layer: ； The fused detail layer results are subjected to inverse pyramid transformation to obtain the fused image.

7. The visible light and shortwave infrared image fusion method according to claim 6, characterized in that, The detail layer fusion result is subjected to inverse pyramid transformation to obtain the fused image, including: The result of blending the detail layers The fusion result is obtained by performing an inverse transform on the base layer of the visible light image Y: The fusion result is used as the luminance channel. and with visible light source image The color channels U and V are converted together into the RGB space to obtain the final fusion result.

Images