Image enhancement method and system based on fusion of optical image and infrared image

Abstract

The invention provides an image enhancement method and system based on fusion of an optical image and an infrared image, and belongs to the technical field of electric power engineering. The method comprises the following steps: extracting a brightness channel of an infrared image and a brightness channel of an optical image in an HSL color gamut, determining a pixel adaptive weight according to the brightness channel of the infrared image, obtaining an adaptive brightness channel based on the pixel adaptive weight and the brightness channel of the optical image, and outputting the adaptive brightness channel. According to the brightness channel, the overall weight and the self-adaptive brightness channel of the infrared image, an enhanced infrared image brightness channel is obtained, the enhanced infrared image brightness channel and a chrominance channel and a saturation channel before infrared image enhancement are combined, an infrared image in an enhanced HSL color gamut is obtained, the infrared image is converted into an RGB color gamut, and the infrared image in the enhanced HSL color gamut is obtained. An enhanced infrared image is obtained; according to the method, excessive enhancement of the background area of the infrared image is avoided, and the overall visual effect after fusion is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power engineering, and particularly to an image enhancement method and system based on the fusion of optical images and infrared images. Background Art

[0002] The statements in this part merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] In the detection of power equipment and the inspection of transmission lines, infrared images can promptly detect abnormalities of equipment, such as damage to poles and towers, aging of insulators, short circuits of lines, and potential fire risks, etc., which helps to ensure the safe and stable operation of power. However, since infrared radiation is only related to the temperature of an object and has nothing to do with the shape, structure or details of the object, infrared images are often relatively blurred and cannot display the structural and detailed information of the target object, which is not conducive to the accurate positioning of abnormalities. Optical images can provide rich information about the surface color, texture, shape and structure of an object. Fusing optical images with infrared images can improve the quality and recognition accuracy of infrared images, enhance the contrast and clarity of infrared images, improve the quality of the monitoring screen, better identify targets and detect abnormalities.

[0004] The inventors found that the existing optical image and infrared image fusion strategies have the following problems:

[0005] (1) When fusing optical images and infrared images, it is necessary to first register the optical image and the infrared image. The prior art generally uses feature point extraction methods for image matching. Commonly used feature points include SIFT, SURF, and ORB, etc. However, since optical images are mainly affected by lighting conditions, the image quality at night drops significantly, and infrared images are mainly sensitive to temperature. The image differences between the two are large. In addition, factors such as the moving accuracy of the pan-tilt, external disturbances, and image zoom will also have an adverse impact on image registration. The accuracy and stability of the prior art in image alignment are poor, and extremely high image quality and a stable environment are required for alignment, which limits the application scenarios;

[0006] (2) When fusing the aligned images, the prior art uses the image pyramid method to fuse infrared images and optical images. This scheme performs pyramid processing and hierarchical fusion on the images. When the image alignment accuracy is poor, it can avoid the problem of ghosting in the fused image to a certain extent. However, the fusion is processed in the RGB color gamut, and the color of the fused infrared image will be interfered by the optical image, resulting in the inability to judge the temperature of the object by color;

[0007] (3) In the prior art, the weights during fusion are the same for all pixels. However, infrared images mainly focus on objects with higher temperatures, and the background temperature is lower. The unified weights will greatly enhance the texture and detail information of the background, which will instead affect the visual perception of the infrared image, enhance the useless information on the background, interfere with the main body of the image, and reduce the consistency of the picture. Summary of the Invention

[0008] To solve the deficiencies of the prior art, the present invention provides an image enhancement method and system based on the fusion of optical images and infrared images. The fusion weights are divided into overall weights and pixel - adaptive weights, emphasizing the enhancement of foreground targets, suppressing the enhancement of background details, achieving stable and accurate alignment of optical images and infrared images, avoiding over - enhancement of the background area of infrared images, and keeping the color of the infrared image consistent before and after enhancement, thereby improving the overall visual effect after fusion.

[0009] To achieve the above - mentioned purpose, the present invention adopts the following technical solutions:

[0010] In the first aspect, the present invention provides an image enhancement method based on the fusion of optical images and infrared images.

[0011] An image enhancement method based on the fusion of optical images and infrared images includes the following processes:

[0012] Align the optical image and the infrared image to be processed, sharpen the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channels of the infrared image and the optical image in the HSL color gamut;

[0013] Determine the pixel - adaptive weights according to the luminance channel of the infrared image, and obtain the adaptive luminance channel based on the pixel - adaptive weights and the luminance channel of the optical image;

[0014] Obtain the enhanced luminance channel of the infrared image according to the luminance channel of the infrared image, the overall weights, and the adaptive luminance channel, merge the enhanced luminance channel of the infrared image and the chrominance channel and saturation channel of the infrared image before enhancement to obtain the enhanced infrared image in the HSL color gamut, and convert it to the RGB color gamut to obtain the enhanced infrared image.

[0015] In the second aspect, the present invention provides an image enhancement system based on the fusion of optical images and infrared images.

[0016] An image enhancement system based on the fusion of optical images and infrared images includes:

[0017] A luminance channel extraction unit, configured to: align an optical image and an infrared image to be processed, sharpen the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channel of the infrared image and the luminance channel of the optical image in the HSL color gamut;

[0018] A luminance channel adaptive unit, configured to: determine pixel adaptive weights according to the luminance channel of the infrared image, and obtain an adaptive luminance channel based on the pixel adaptive weights and the luminance channel of the optical image;

[0019] A fusion enhancement unit, configured to: obtain an enhanced luminance channel of the infrared image according to the luminance channel of the infrared image, an overall weight, and the adaptive luminance channel, merge the enhanced luminance channel of the infrared image and the chrominance channel and saturation channel of the infrared image before enhancement, obtain an enhanced infrared image in the HSL color gamut, and convert it to the RGB color gamut to obtain an enhanced infrared image.

[0020] In a third aspect, the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the image enhancement method based on the fusion of an optical image and an infrared image as described in the first aspect of the present invention are implemented.

[0021] In a fourth aspect, the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the image enhancement method based on the fusion of an optical image and an infrared image as described in the first aspect of the present invention are implemented.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] 1. The present invention innovatively proposes a weight adaptive method, which divides the fusion weight into an overall weight and pixel adaptive weights, emphasizes the enhancement of foreground targets, suppresses the enhancement of background details, and makes the overall visual effect after fusion better.

[0024] 2. The present invention innovatively proposes a dual-light fusion enhancement method, which uses a sharpening enhancement method to extract texture information in the visible light image, and the infrared image is fused and enhanced in the luminance channel of the HSL color gamut, ensuring the color consistency of the infrared image before and after enhancement.

[0025] 3. The present invention uses the LightGlue deep learning method to extract matching feature points, align the optical image and the infrared image, and can stably match feature points under the influence of night, zoom, pan-tilt motion error, etc. It has strong scene adaptability and high matching accuracy, further improving the visual effect after fusion.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0028] Figure 1 Schematic flowchart of the image enhancement method based on the fusion of optical image and infrared image provided in Embodiment 1 of the present invention;

[0029] Figure 2 Schematic diagram for comparing the sharpening effects provided in Embodiment 1 of the present invention;

[0030] Figure 3 Schematic flowchart of the weight adaptive fusion method provided in Embodiment 1 of the present invention;

[0031] Figure 4 Schematic diagram for comparing the effects before and after infrared image enhancement provided in Embodiment 1 of the present invention;

[0032] Figure 5 Schematic diagram of the image enhancement system based on the fusion of optical image and infrared image provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further descriptions of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0035] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0036] Embodiment 1:

[0037] As Figure 1 shown, Embodiment 1 of the present invention provides an image enhancement method based on the fusion of optical image and infrared image, including the following processes:

[0038] S1: Align the optical image and the infrared image to be processed, sharpen the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channels of the infrared image and the optical image in the HSL color gamut;

[0039] S2: Determine the pixel adaptive weight according to the luminance channel of the infrared image, and based on the pixel adaptive weight and the luminance channel of the optical image, obtain the adaptive luminance channel;

[0040] S3: According to the luminance channel of the infrared image, the overall weight, and the adaptive luminance channel, obtain the enhanced luminance channel of the infrared image, merge the enhanced luminance channel of the infrared image and the chrominance channel and saturation channel before infrared image enhancement, obtain the infrared image in the enhanced HSL color gamut, and convert it to the RGB color gamut to obtain the enhanced infrared image.

[0041] In this embodiment, HSL refers to Hue, Saturation, and Lightness. Hue is the basic attribute of color, which is the common color name, such as red, yellow, etc.; Saturation refers to the purity of color. The higher the saturation, the purer the color, and the lower it is, the grayer it gradually becomes.

[0042] The RGB color gamut is a color standard in the industrial field. It obtains various colors through the changes of the three color channels of Red, Green, and Blue and their superposition with each other. RGB represents the colors of the three channels of Red (R), Green (G), and Blue (B).

[0043] In step S1, align the optical image and the infrared image to be processed. Specifically, it includes the following process:

[0044] Collect the data sets of the optical image and the infrared image, and annotate the data sets to meet the training requirements of the LightGlue network model. Use the annotated data sets to train and test the LightGlue network model. The LightGlue network model is used to match the sparse local features between images, predict the partial matching relationship between the local feature sets extracted from the optical image and the infrared image, calculate the pairwise similarity score matrix between the points in the two images, and achieve the stable registration of the optical image and the infrared image;

[0045] It should be noted that the LightGlue network model provided in this embodiment is all implemented based on the existing network structure, and only the data set provided in this embodiment is used for training to achieve the purpose of feature point pair extraction.

[0046] Using the above trained LightGlue network model, extract the feature point pairs of the optical image and the infrared image. According to the least squares method or the RANSAC algorithm, calculate the homography matrix from the optical image to the infrared image. Map the optical image according to the homography matrix and align it with the infrared image. By training and using the LightGlue model for feature point matching between visible light images and infrared images, the problems of large differences between optical images and infrared images, poor alignment accuracy and stability can be solved, and the robustness in the face of complex scenes can be improved.

[0047] In step S1, sharpening the aligned optical image will further enhance the edge and structural information of the image. Specifically, it includes the following process:

[0048] Taking the sharpened and aligned optical image as the original image img, performing Gaussian blur on the original image img to obtain a low-frequency image, subtracting the low-frequency image from the original image to obtain a high-frequency image, and superimposing the high-frequency image and the original image according to a set ratio ratio to obtain the sharpened optical image img_en. The calculation formula of img_en is:

[0049] img_en = ratio * (img - Gaus(img)) + (1 - ratio) * img(1);

[0050] Among them, Gaus(img) represents the low-frequency image, and the sharpening effect comparison is as Figure 2 shown, Figure 2 (a) in Figure 2 is the original image,

[0051] (b) in

[0052] is the image sharpened by UMS. When the texture is not clear enough, the sharpened image can significantly enhance the detail information in the image and prepare for subsequent fusion enhancement.

[0053]

[0054]

[0055]

[0056] Extract the luminance channels of the two images after extraction and transformation for fusion enhancement. Transform them into the HSL color gamut and perform fusion on the luminance channels, which can avoid color changes in the fused image and maintain the color consistency of the image before and after enhancement.

[0057] In step S2, determine the pixel adaptive weight according to the luminance channel of the infrared image. Specifically, it includes:

[0058] Normalize the pixel values of the luminance channel to 0-1 as the enhancement weight corresponding to each pixel. Through this operation, points with higher pixel values correspond to larger enhancement weights, and points with darker backgrounds correspond to lower enhancement weights, realizing the adaptive calculation of enhancement weights according to pixel values and solving the problem of clutter and reduced visual perception in the fused image caused by over-enhancement of the background area.

[0059] In step S2, based on the pixel adaptive weight and the luminance channel of the optical image, obtain the adaptive luminance channel. Specifically, it includes: The pixel adaptive weight acts on the luminance channel of the optical image (the two are multiplied) to obtain the adaptive luminance channel for enhancement.

[0060] In step S3, as Figure 3 shown, according to the luminance channel of the infrared image, the overall weight, and the adaptive luminance channel, obtain the enhanced luminance channel of the infrared image. Specifically, it includes:

[0061] The overall weight alpha controls the fusion ratio of the optical image and the infrared image. The calculation formula is:

[0062] L = alpha * (L_ir_pix * L_opt) + (1 - alpha) * L_ir(5);

[0063] In the formula, L is the enhanced infrared luminance channel after fusion, L_ir_pix is the pixel adaptive enhancement weight, L_opt is the luminance channel of the optical image, L_ir is the original luminance channel of the infrared image. The adaptive weight can suppress the over-enhancement of the background, and the overall weight alpha can control the fusion intensity. Through the two weights, the enhancement of the main target after fusion is realized, and the background will not be cluttered, achieving a better image fusion effect. Here, the overall weight alpha is a positive real number less than 1 and can be set according to the needs of the fusion effect. The closer the overall weight alpha is set to 1, the stronger the fusion intensity of the optical image and the stronger the edge effect of the fused image.

[0064] In S3, convert to the RGB color gamut. Specifically, it includes the following process:

[0065] A color defined by the (h, s, l) values in the HSL space, with h in the value range [0, 360) indicating the hue angle, s and l representing saturation and lightness respectively in the value range [0, 1], and the corresponding (r, g, b) primary colors in the RGB space, with r, g, and b corresponding to red, green, and blue respectively also in the value range [0, 1], can be calculated as follows:

[0066] First, if s = 0, the resulting color is achromatic or gray. In this special case, r, g, and b are all equal to l. Note that the value of h is undefined in this case. When s ≠ 0, the following procedure can be used:

[0067]

[0068] p = 2 × l - q(7);

[0069]

[0070]

[0071] t G = h k (10);

[0072]

[0073] if t C < 0 → t C = t C + 1.0 for each C ∈ (R, G, B) (12);

[0074] if t C > 1 → t C = t C - 1.0 for each C ∈(R, G, B) (13);

[0075] For each color vector Color:

[0076] Color = (Color R , Color G , Color B ) = (r, g, b) (14);

[0077]

[0078] for each C∈(R, G, B)(16);

[0079] The effect comparison before and after infrared image enhancement is as Figure 4 shown, Figure 4In which, (a) is the original infrared image, Figure 4 and (b) is the enhanced infrared image.

[0080] Example 2:

[0081] As Figure 5 shown, Example 2 of the present invention provides an image enhancement system based on the fusion of an optical image and an infrared image, including:

[0082] A luminance channel extraction unit, configured to: align the optical image and the infrared image to be processed, perform sharpening processing on the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channel of the infrared image and the luminance channel of the optical image in the HSL color gamut;

[0083] A luminance channel adaptive unit, configured to: determine pixel adaptive weights according to the luminance channel of the infrared image, and obtain an adaptive luminance channel based on the pixel adaptive weights and the luminance channel of the optical image;

[0084] A fusion enhancement unit, configured to: obtain the enhanced infrared image luminance channel according to the luminance channel of the infrared image, the overall weight, and the adaptive luminance channel, merge the enhanced infrared image luminance channel and the chrominance channel and saturation channel of the infrared image before enhancement, obtain the enhanced infrared image in the HSL color gamut, and convert it to the RGB color gamut to obtain the enhanced infrared image.

[0085] For the specific implementation process of the luminance channel extraction unit, see the specific process of step S1 in Example 1. For the specific implementation process of the luminance channel adaptive unit, see the specific process of step S2 in Example 1. For the specific implementation process of the fusion enhancement unit, see the specific process of step S3 in Example 1, which will not be elaborated here.

[0086] Example 3:

[0087] Example 3 of the present invention provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the steps in the image enhancement method based on the fusion of an optical image and an infrared image as described in Example 1 of the present invention. For the detailed steps, see steps S1, S2, and S3 in Example 1, which will not be elaborated here.

[0088] Example 4:

[0089] Example 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in the image enhancement method based on the fusion of an optical image and an infrared image as described in Example 1 of the present invention. For the detailed steps, see steps S1, S2, and S3 in Example 1, which will not be elaborated here.

[0090] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image enhancement method based on the fusion of optical images and infrared images, characterized in that, It includes the following processes: Align the optical image and the infrared image to be processed, sharpen the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channels of the infrared image and the optical image in the HSL color gamut; Determine the pixel adaptive weight according to the luminance channel of the infrared image, and obtain the adaptive luminance channel based on the pixel adaptive weight and the luminance channel of the optical image; According to the luminance channel of the infrared image, the overall weight, and the adaptive luminance channel, obtain the enhanced luminance channel of the infrared image, merge the enhanced luminance channel of the infrared image and the chrominance channel and saturation channel of the infrared image before enhancement, obtain the infrared image in the enhanced HSL color gamut, and convert it to the RGB color gamut to obtain the enhanced infrared image.

2. The image enhancement method based on the fusion of optical images and infrared images according to claim 1, characterized in that, Determine the pixel adaptive weight according to the luminance channel of the infrared image, including: Normalize the pixel values of the luminance channel, and use the normalized result as the enhancement weight corresponding to each pixel, and use the obtained enhancement weight as the pixel adaptive weight.

3. The image enhancement method based on the fusion of optical images and infrared images according to claim 1, characterized in that, Obtain the adaptive luminance channel based on the pixel adaptive weight and the luminance channel of the optical image, including: Use the product of the pixel adaptive enhancement weight and the luminance channel of the optical image as the adaptive luminance channel.

4. The image enhancement method based on the fusion of optical images and infrared images according to any one of claims 1-3, characterized in that, According to the luminance channel of the infrared image, the overall weight, and the adaptive luminance channel, obtain the enhanced luminance channel of the infrared image, including: Use the product of the overall weight and the adaptive luminance channel as the first variable, use the product of the difference between 1 and the overall weight and the original luminance channel of the infrared image as the second variable, and use the sum of the first variable and the second variable as the enhanced luminance channel of the infrared image; where the overall weight is a positive real number less than 1.

5. The image enhancement method based on the fusion of optical images and infrared images according to any one of claims 1-3, characterized in that, Align the optical image and the infrared image to be processed, including: Extract the feature point pairs of the optical image and the infrared image, calculate the homography matrix from the optical image to the infrared image according to the extracted feature point pairs, and map the optical image according to the homography matrix to complete the alignment of the optical image and the infrared image.

6. The image enhancement method based on the fusion of optical images and infrared images according to any one of claims 1-3, characterized in that, Sharpen the aligned optical image, including: Use the sharpened aligned optical image as the original image, perform Gaussian blur on the original image to obtain a low-frequency image, subtract the low-frequency image from the original image to obtain a high-frequency image, and superimpose the high-frequency image and the original image according to a set ratio to obtain the sharpened optical image.

7. An image enhancement system based on the fusion of optical images and infrared images, characterized in that, It includes: A luminance channel extraction unit configured to: align the optical image and the infrared image to be processed, sharpen the aligned optical image, convert the aligned infrared image and the sharpened optical image to the HSL color gamut, and extract the luminance channels of the infrared image and the optical image in the HSL color gamut; A luminance channel adaptive unit configured to: determine the pixel adaptive weight according to the luminance channel of the infrared image, and obtain the adaptive luminance channel based on the pixel adaptive weight and the luminance channel of the optical image; The fusion enhancement unit is configured to: obtain the enhanced luminance channel of the infrared image according to the luminance channel, the overall weight, and the adaptive luminance channel of the infrared image, merge the enhanced luminance channel of the infrared image and the chrominance channel and the saturation channel of the infrared image before enhancement to obtain the infrared image in the enhanced HSL color gamut, and convert it to the RGB color gamut to obtain the enhanced infrared image.

8. The image enhancement system based on the fusion of optical images and infrared images according to claim 7, characterized in that, In the fusion enhancement unit, obtaining the enhanced luminance channel of the infrared image according to the luminance channel, the overall weight, and the adaptive luminance channel of the infrared image includes: Using the product of the overall weight and the adaptive luminance channel as the first variable, using the product of the difference between 1 and the overall weight and the original luminance channel of the infrared image as the second variable, and using the sum of the first variable and the second variable as the enhanced luminance channel of the infrared image; wherein, the overall weight is a positive real number less than 1.

9. A computer-readable storage medium, on which a program is stored, characterized in that, When the program is executed by the processor, it implements the steps in the image enhancement method based on the fusion of optical images and infrared images according to any one of claims 1-6.

10. An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the image enhancement method based on the fusion of optical images and infrared images according to any one of claims 1-6.

Images