Visible light and short wave infrared image fusion method

By calculating the texture richness and weight map of short-wave infrared and visible light images, the image pyramid is constructed for weighted fusion, which solves the problem of poor fusion effect of short-wave infrared and visible light images in the existing technology, and achieves a better visual information fusion effect.

CN119067862AActive Publication Date: 2024-12-03HUAZHONG UNIV OF SCI & TECH +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410703892.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-03
Publication Date
2024-12-03
Estimated Expiration
2044-06-03

AI Technical Summary

Technical Problem

In the prior art, there are few researches on the fusion of visible light and infrared images, especially the fusion algorithm of short-wave infrared images and visible light images has not been fully explored, resulting in poor fusion effect.

Method used

By acquiring short-wave infrared and visible light source images, calculating texture richness and weight maps, building an image pyramid for fusion, using local texture and gradient comparison functions and adjustment factors to determine the weight, and normalizing the process to weight the fused image detail layer.

Benefits of technology

Effectively extract unique textures in short-wave infrared images, enhance image fusion effect, and improve the richness and quality of visual information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067862A_ABST
    Figure CN119067862A_ABST
Patent Text Reader

Abstract

The invention provides a visible light and short-wave infrared image fusion method, which comprises the following steps: S1, acquiring a short-wave infrared source image ISWIR and a visible light source image IVIS, and taking a brightness channel image of the visible light source image IVIS as a visible light image Y to be fused; s2, according to the short-wave infrared source image ISWIR and the visible light image Y, obtaining weight maps WSWIR and WY of the short-wave infrared source image ISWIR and the visible light image Y; and S3, constructing an image pyramid to perform image fusion. According to the scheme, a local texture and gradient contrast function LTG (I (x, y)) is designed, specific textures in a short-wave infrared image and weak textures in a visible light image can be extracted during image fusion, and the image fusion effect can be enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a method for fusing visible light and short-wave infrared images. Background Art

[0002] Image fusion refers to the image processing of images of the same scene obtained by multi-source sensors, combining the respective advantages of images from different sources, fusing their respective rich information, and obtaining high-quality images with good visual effects and rich visual information. Currently, in the field of visible light and infrared image fusion, the most widely studied object is the fusion algorithm of visible light images and thermal imaging images, while there are fewer studies on image fusion algorithms for other infrared bands.

[0003] Due to its special wavelength range and imaging principles similar to visible light, short-wave infrared can penetrate clouds and fog to form images, and it also has unique advantages in material identification. Therefore, it is of certain significance to fuse visible light and short-wave infrared images. Summary of the invention

[0004] Based on the above problems existing in the prior art, the present invention aims to solve the technical problems in the prior art.

[0005] The present invention provides a visible light and short-wave infrared image fusion method, comprising:

[0006] S1: Get shortwave infrared source image I SWIR and visible light source image I VIS , the visible light source image I VIS Convert from RGB format to YUV format, and take the brightness channel image as the visible light image Y to be fused;

[0007] S2: Based on the short-wave infrared source image I SWIR and visible light image Y to obtain the short-wave infrared source image I SWIR and the weight map W of the visible light image Y SWIR , W Y ,include:

[0008] S21: Calculate shortwave infrared source image I SWIR Texture richness of LTG SWIR (x, y) and visible light source image I VIS Texture richness of LTG VIS (x,y);

[0009] S22: Determine the short-wave infrared source image I SWIR The initial weight S(x, y) and adjustment factor H(x, y);

[0010] S23: shortwave infrared source image I SWIRThe initial weight S(x, y) and the adjustment factor H(x, y) are multiplied to obtain the short-wave infrared source image I SWIR The initial weight graph W' SWIR (x,y);

[0011] S24: the initial weight map W' SWIR (x, y) is normalized to obtain the short-wave infrared source image I SWIR The normalized weight graph W SWIR , and obtain the normalized weight map W of the visible light image Y Y ;

[0012] S3: Based on the short-wave infrared source image I SWIR , visible light image Y, normalized weight map W SWIR and W Y Construct an image pyramid for image fusion.

[0013] According to an embodiment of the present invention, the short-wave infrared source image I is calculated. SWIR and visible light source image I VIS The formula for the texture richness is:

[0014]

[0015] Among them, α is a coefficient used to adjust the ratio of image texture to image gradient; LC(x',y') represents the local contrast of the image, GD(x',y') represents the gradient value of the image; N(x,y) represents the local neighborhood of the image centered at (x, y), the neighborhood is set to 5×5, and the total number of pixels is N; mean is the local pixel mean of the image; Expressed as the second-order inverse of the x direction of the image (x, y), Expressed as the second-order inverse of the y direction of the image (x, y);

[0016] According to an embodiment of the present invention, the proportionality coefficient α is set to 0.5.

[0017] According to an embodiment of the present invention, the short-wave infrared source image I SWIR The initial weight graph W' SWIR The calculation formula for (x,y) is:

[0018] W′ SWIR (x,y)=H(x,y)*S(x,y);

[0019] For the initial weight graph W' SWIR (x, y) is normalized to obtain the short-wave infrared source image I SWIR The normalized weight graph W SWIR The formula is:

[0020]

[0021] W Y =1-W SWIR ;

[0022] Among them, W' SWIR (x,y)| max With W' SWIR (x,y)| min W' SWIR The maximum and minimum values ​​of (x,y).

[0023] According to an embodiment of the present invention, in step S22, the short-wave infrared source image I is determined. SWIR The initial weight S(x, y) includes: judging LTG SWIR (x,y)≧LTG VIS (x,y) is true, if true, then let Otherwise, S(x, y) is 0.

[0024] According to an embodiment of the present invention, in step S22, the short-wave infrared source image I is determined. SWIR The adjustment factor H(x, y) includes: including: judging I SWIR (x,y)≥I VIS (x,y) is true, if true, then let Otherwise, H(x, y) takes 1; where I SWIR (x, y) is the short-wave infrared source image I SWIR The pixel value at (x, y), I VIS (x, y) is the pixel value at (x, y) in the visible light source image.

[0025] According to an embodiment of the present invention, in step S3, according to the short-wave infrared source image I SWIR , visible light image Y, normalized weight map W SWIR and W Y Build an image pyramid for image fusion, including:

[0026] Construct the shortwave infrared source image I SWIR and the image pyramid of the visible light image Y to obtain the short-wave infrared source image I SWIR And the Laplacian pyramid LP of the visible light image Y SWIR (n) LP Y (n) , thus obtaining the Laplace pyramid LP SWIR (n) and LP Y (n)Detail layer images of each layer, where n = 1, 2, 3, ...;

[0027] Construct normalized weight graph W SWIR and W Y To obtain the normalized weight map W SWIR and W Y Gaussian Pyramid GP SWIR (l) , GP Y (l) , thus getting the same as the Laplace pyramid LP SWIR (n) and LP Y (n) The normalized detail layer weight map corresponding to each detail layer image, where l = 1, 2, 3, ...;

[0028] The Laplace pyramid LP SWIR (n) and LP Y (n) Each layer of detail layer image is based on Gaussian pyramid GP SWIR (l) and GP Y (l) The normalized detail layer weight map of each layer is weighted fused to obtain the fusion result of each detail layer:

[0029] Fusion_d (n) =LP Y (n) *GP Y (l) +LP SWIR (n) *GP SWIR (l) ;

[0030] The detail layer fusion result is subjected to an inverse pyramid transform to obtain the fused image.

[0031] According to an embodiment of the present invention, a pyramid transformation is performed on the detail layer fusion result to obtain a fused image, including:

[0032] The fusion result of the detail layer is Fusion_d (n) Perform an inverse transformation with the base layer of the visible light image Y to obtain the fusion result:

[0033] The fusion result is used as the brightness channel F Y , and the visible light source image I VIS The color channels U and V are transformed into RGB space together to obtain the final fusion result.

[0034] The beneficial effects of the present invention are:

[0035] A visible light and short-wave infrared image fusion method provided in an embodiment of the present invention designs a local texture and gradient contrast function LTG(I(x, y)) for representing a short-wave infrared source image I SWIR and visible light source image I VIS The texture richness of the image is determined by the initial weight S(x, y) and the adjustment factor H(x, y) according to the size of LTG(I(x, y)) and the image pixel value I(x, y), so as to obtain the short-wave infrared source image I SWIR The normalized weight map of the visible light image Y can extract the texture that is unique to the shortwave infrared image but weak in the visible light image during image fusion, which can enhance the image fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 The present invention provides a schematic flow chart of the steps of a method for fusing visible light and short-wave infrared images;

[0038] Figure 2 It is a schematic diagram of the result of an image processing method of a visible light and short-wave infrared image fusion method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following descriptions of the embodiments are made with reference to the attached drawings to illustrate specific embodiments in which the present invention may be implemented. The directional terms mentioned in the present invention, such as [up], [down], [front], [back], [left], [right], [inside], [outside], [side], etc., are only with reference to the directions of the attached drawings. Therefore, the directional terms used are used to illustrate and understand the present invention, but not to limit the present invention. In the drawings, units with similar structures are represented by the same reference numerals.

[0040] The present invention provides a method for fusing visible light and short-wave infrared images, the steps of which are as follows: Figure 1 As shown,

[0041] like Figure 2 FIG. 1 is a schematic diagram of the image result in the image fusion process according to an embodiment of the present invention. First, a short-wave infrared source image I is obtained. SWIR and visible light source image I VIS , and the visible light source image IVIS After converting from RGB format to YUV format, take the visible light source image I VIS The brightness channel (Y channel) image is used as the visible light image Y to be fused;

[0042] Secondly, a local texture and gradient function LTG (Local texture and gradient) is designed to represent the shortwave infrared source image I in order to obtain the details that are unique to the shortwave infrared image but are missing in the visible light image due to rain and fog. SWIR and visible light source image I VIS The texture richness is calculated as follows:

[0043]

[0044] Among them, α is a coefficient used to adjust the ratio of image texture to image gradient; LC(x',y') represents the local contrast of the image, GD(x',y') represents the gradient value of the image; N(x,y) represents the local neighborhood of the image centered at (x,y), the neighborhood is set to 5×5, and the total number of pixels is N; mean is the local pixel mean of the image; Expressed as the second-order inverse of the x direction of the image (x, y), Expressed as the second-order inverse of the y direction of the image (x, y);

[0045] Then, the LTG function is used to determine the shortwave infrared source image I SEIR The initial weight of is as follows:

[0046]

[0047] At the same time, due to the imaging differences between shortwave infrared images and visible light images, the shortwave infrared source image I must also be considered when fusion of the detail layer SEIR and visible light source image I VIS Specifically, when the short-wave infrared source image I SWIR The brightness is greater than the visible light source image I VIS When the brightness of the short-wave infrared source image I SEIR The brightness is less than the visible light source image I VIS When the brightness is high, it is usually a short-wave infrared source image I SEIR The shadow area of ​​​​the image should have a larger weight to increase the realism of the image. Therefore, in this scheme, an adjustment factor H(x,y) is set to adjust the image according to the short-wave infrared source image I SEIR and visible light source image I VIS The brightness of the image determines the short-wave infrared source image I SWIRThe weight of the adjustment factor H(x,y) is determined as follows:

[0048]

[0049] Among them, I SWIR (x, y) is the short-wave infrared source image I SWIR The pixel value at (x, y), I VIS (x, y) is the pixel value at (x, y) in the visible light source image.

[0050] Multiply the initial weight S(x,y) by the adjustment factor H(x,y) to get the shortwave infrared source image I SWIR The initial weight graph W' SWIR (x,y), the specific formula is:

[0051] W′ SWIR (x,y)=H(x,y)*S(x,y);

[0052] After normalization, the shortwave infrared source image I is obtained. SWIR The normalized weight graph W SWIR , the formula for normalized weight graph is:

[0053]

[0054] Among them, the normalized weight map W SWIR The range of pixels in (x, y) is (0, 1), W' SWIR (x,y)| max With W' SWIR (x,y)| min W' SWIR The maximum and minimum values ​​of (x,y);

[0055] Thus, the normalized weight map of the visible light image Y can be obtained as follows:

[0056] W Y =1-W SWIR .

[0057] Next, according to the shortwave infrared source image I SWIR , visible light image Y, normalized weight map W SWIR and W Y Construct an image pyramid for image fusion. The specific process is as follows:

[0058] First, construct the short-wave infrared source image I SWIR and the image pyramid of the visible light image Y to obtain the short-wave infrared source image I SWIR and the Laplacian pyramid LP of the visible light image Y SWIR (n)LP Y (n) , thus obtaining the Laplace pyramid LP SWIR (n) and LP Y (n) Detail layer images of each layer, where n = 1, 2, 3, ...;

[0059] Then: construct the normalized weight graph W SWIR and W Y To obtain the normalized weight map W SWIR and W Y Gaussian Pyramid GP SWIR (l) , GP Y (l) , thus getting the same as the Laplace pyramid LP SWIR (n) and LP Y (n) The normalized detail layer weight map corresponding to each detail layer image, where l = 1, 2, 3, ...;

[0060] Next: Laplace pyramid LP SWIR (n) and LP Y (n) Each layer of detail layer image is based on Gaussian pyramid GP SWIR (l) and GP Y (l) The normalized detail layer weight map of each layer is weighted fused to obtain the fusion result of each detail layer:

[0061] Fusion_d (n) =LP Y (n) *GP Y (l) +LP SWIR (n) *GP SWIR (l) .

[0062] Finally, the fusion result of the detail layer is subjected to an inverse pyramid transform to obtain the fused image. Specifically, the fusion result of the detail layer is first inversely transformed with the base layer of the visible light image Y to obtain the fusion result F (n) , the specific formula is as follows:

[0063]

[0064] F (n) =F (l) +Fusion_d (n),

[0065] Specifically, in the embodiment of the present invention, the Gaussian pyramid GP SWIR (l) and GP Y (l) There are 4 layers, i.e., l=4; and l=n+1, i.e., n=3. Then the fusion result of this embodiment is:

[0066]

[0067] F (n) =F (n+1) +Fusion_d (n) ,n=3,2,1;

[0068] The final F (1) As the brightness channel E Y , and the visible light source image I VIS The color channels U and V are converted into RGB space together to obtain the final fusion result Fusion. The conversion formula is:

[0069]

[0070] In summary, a visible light and short-wave infrared image fusion method provided in an embodiment of the present invention designs a local texture and gradient contrast function LTG(I(x, y)) for representing the short-wave infrared source image I SWIR and visible light source image I VIS The texture richness of the image is determined by the initial weight S(x, y) and the adjustment factor H(x, y) according to the size of LTG(I(x, y)) and the image pixel value I(x, y), so as to obtain the short-wave infrared source image I SWIR The normalized weight map of the visible light image Y can extract the texture that is unique to the shortwave infrared image but weak in the visible light image during image fusion, which can enhance the image fusion effect.

[0071] It should be noted that although the present invention is disclosed as above by specific embodiments, the above embodiments are not intended to limit the present invention. Ordinary technicians in this field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope defined in the claims.

Claims

1. A visible light and short-wave infrared image fusion method, It is characterized in that include: S1: Get shortwave infrared source image I SWIR and visible light source image I VIS , the visible light source image I VIS Convert from RGB format to YUV format, and take the brightness channel image as the visible light image Y to be fused; S2: Based on the short-wave infrared source image I SWIR and visible light image Y to obtain the short-wave infrared source image I SWIR and the weight map W of the visible light image Y SWIR , W Y ,include: S21: Calculate shortwave infrared source image I SWIR Texture richness of LTG SWIR (x, y) and visible light source image I VIS Texture richness of LTG VIS (x,y); S22: Determine the short-wave infrared source image I SWIR The initial weight S(x, y) and adjustment factor H(x, y); S23: shortwave infrared source image I SWIR The initial weight S(x, y) and the adjustment factor H(x, y) are multiplied to obtain the short-wave infrared source image J SWIR The initial weight graph W' SWIR (x,y); S24: the initial weight map W' SWIR (x, y) is normalized to obtain the short-wave infrared source image I SWIR The normalized weight graph W SWIR , and obtain the normalized weight map W of the visible light image Y Y ; S3: Based on the short-wave infrared source image I SWIR , visible light image Y, normalized weight map W SWIR and W Y Construct an image pyramid for image fusion.

2. A visible light and short-wave infrared image fusion method according to claim 1, It is characterized in that Calculate shortwave infrared source image I SWIR and visible light source image I VIS The formula for the texture richness is: Among them, α is a coefficient used to adjust the ratio of image texture to image gradient; LC(x',y') represents the local contrast of the image, GD(x',y') represents the gradient value of the image; N(x,y) represents the local neighborhood of the image centered at (x, y), the neighborhood is set to 5×5, and the total number of pixels is N; mean is the local pixel mean of the image; Expressed as the second-order inverse of the x direction of the image (x, y), It is expressed as the second-order inverse of the y direction of the image (x, y).

3. A visible light and short-wave infrared image fusion method according to claim 2, It is characterized in that The proportionality coefficient α is set to 0.

5.

4. The method for fusing visible light and short-wave infrared images according to claim 2, It is characterized in that The shortwave infrared source image I SWIR The initial weight graph W' SWIR The calculation formula for (x,y) is: W’ SWIR (x,y)=H(x,y)*S(x,y); For the initial weight graph W' SWIR (x, y) is normalized to obtain the short-wave infrared source image I SWIR The normalized weight graph W SWIR The formula is: IN Y =1-W SWIR ; Among them, W' SWIR (x,y)| max With W' SWIR (x,y)|| min W' SWIR The maximum and minimum values ​​of (x,y).

5. The method for fusing visible light and short-wave infrared images according to claim 1, It is characterized in that In step S22, the short-wave infrared source image I is determined SWIR The initial weight S(x, y) includes: judging LTG SWIR (x,y)≧LTG VIS (x,y) is true, if true, then let Otherwise, S(x, y) is 0.

6. The method for fusing visible light and short-wave infrared images according to claim 1, It is characterized in that In step S22, the short-wave infrared source image I is determined SWIR The adjustment factor H(x, y) includes: including: judging I SWIR (x,y)≥I VIS (x,y) is true, if true, then let Otherwise, H(x, y) takes 1; where I SWIR (x, y) is the short-wave infrared source image I SWIR The pixel value at (x, y), I VIS (x, y) is the pixel value at (x, y) in the visible light source image.

7. The method for fusing visible light and short-wave infrared images according to claim 1, It is characterized in that In step S3, according to the short-wave infrared source image I SWIR , visible light image Y, normalized weight map W SWIR and W Y Build an image pyramid for image fusion, including: Construct the shortwave infrared source image I SWIR and the image pyramid of the visible light image Y to obtain the short-wave infrared source image I SWIR and the Laplacian pyramid LP of the visible light image Y SWIR (n) LP Y (n) , thus obtaining the Laplace pyramid LP SWIR (n) and LP Y (n) Detail layer images of each layer, where n=1,2,3...; Construct normalized weight graph W SWIR and W Y To obtain the normalized weight map W SWIR and W Y Gaussian Pyramid GP SWIR (l) , GP Y (l) , thus obtaining the Laplacian pyramid LP SWIR (n) and LP Y (n) The normalized detail layer weight map corresponding to each detail layer image, where l = 1, 2, 3, ...; The Laplace pyramid LP SWIR (n) and LP Y (n) Each layer of detail layer image is based on Gaussian pyramid GP SWIR (l) and GP Y (l) The normalized detail layer weight map of each layer is weighted fused to obtain the fusion result of each detail layer: Fusion_d (n) =LP Y (n) *GP Y (l) +LP SWIR (n) *GP SWIR (l) ; The detail layer fusion result is subjected to an inverse pyramid transform to obtain the fused image.

8. A visible light and short-wave infrared image fusion method according to claim 7, It is characterized in that The detail layer fusion result is subjected to pyramid transformation to obtain the fused image, including: The fusion result of the detail layer is Fusion_d (n) Perform an inverse transformation with the base layer of the visible light image Y to obtain the fusion result: The fusion result is used as the brightness channel F Y , and the visible light source image I VIS The color channels U and V are transformed into RGB space together to obtain the final fusion result.

Citation Information

Patent Citations

  • Defogging method based on image fusion of millimeter wave and visible light

    CN108921803A

  • Multi-exposure infrared image fusion method

    CN110189281A

  • Image fusion processing method and device

    CN111369486A

  • Infrared and visible light image fusion method based on phase consistency and target enhancement

    CN111462028A

  • Image fusion method and device, medium and electronic equipment

    CN112767289A