Visible light and infrared image real-time fusion method and storage medium

Abstract

The invention discloses a visible light and infrared image real-time fusion method and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: completing the aggregation of light energy according to a visible light and infrared optical system, and obtaining a visible light image and an infrared image; obtaining field-of-view data and distorted field-of-view data through an optical system, and establishing a field-of-view distortion curve model; obtaining a corrected visible light image and a corrected infrared image from the visible light image and the infrared image through a view field distortion curve model; and establishing a fused image RGB space, and performing fusion processing on the corrected data to obtain a final fused image. According to the invention, the quality of visible light and infrared images can be improved, and the fused image can be output in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time fusion method and storage medium for visible light and infrared images. Background Art

[0002] With the rapid advancement of sensor technology, the operating wavelengths of current sensors are gradually expanding. However, individual sensors vary in their response to different environments, weather conditions, and lighting conditions, resulting in inadequate information acquisition and difficulty achieving the desired results. Therefore, image fusion plays a key role in practical applications. Infrared images can penetrate obstructions to capture targets, remain unaffected by complex environments like wind and rain, and can present target images in the form of brightness even in low-light conditions. However, their drawbacks are poor image resolution and limited detail. Visible light images can be acquired through sensors at high resolution, containing rich scene and texture details, and are less susceptible to changes in the surrounding environment. However, their drawback is that they are affected by lighting and obstructions, and cannot fully capture scene information in nighttime environments. Image fusion of infrared and visible light images can compensate for these differences, fully utilizing the texture information of the visible light image and the thermal radiation profile information of the infrared image, effectively enhancing image detail in nighttime environments. While current image fusion algorithms offer high accuracy and accurate fusion colors, they are computationally intensive and cannot achieve full real-time processing. Summary of the Invention

[0003] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to provide a method and storage medium for real-time fusion of visible light and infrared images, so as to achieve real-time fusion of visible light and infrared images while ensuring the fusion color and reduce the delay.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] The present invention discloses a method and storage medium for real-time fusion of visible light and infrared images, comprising the following steps:

[0006] 1. A method for real-time fusion of visible light and infrared images, comprising:

[0007] (1) Complete the collection of light energy through visible light and infrared optical systems;

[0008] (2) Based on the light energy collected in step (1), visible light images I are obtained through visible light and infrared imaging systems. vis and infrared image I ir ;

[0009] (3) Obtaining the field of view data and the distorted field of view data through the visible light optical system and the infrared optical system respectively, establishing a field of view distortion curve model, and transforming the visible light image I obtained in step (2) into vis and infrared image I ir Perform field distortion correction respectively to obtain the corrected visible light image I disvis and infrared image I disir ;

[0010] (4) The corrected visible light image I obtained in step (3) is respectively disvis and infrared image I disir Fusion is performed to obtain the fused image I fus ,include:

[0011]

[0012] Among them I R =(I disvis +I disir ) / 2,I G =I disir , I B =1-I disir ; and I R , I G and I B The mean of M and N represent the number of rows and columns of the image respectively, δ R , δ G and δ B Separate I R , I G and I B The standard deviation of k1, k2, k3, b1, b2, and b3 are color parameters;

[0013] I fuzR , I fuzG and I fuzB is the fused image I fuz Data in RGB space, where I fuz for:

[0014] I fuz =(I fuzR , I fuzG , I fuzB ).

[0015] Furthermore, in step (3), the field of view data x and the distorted field of view data y of the visible light optical system and the infrared optical system are obtained respectively according to the visible light optical imaging system and the optical infrared imaging system, and a field of view distortion curve model is established.

[0016] Furthermore, the field of view distortion curve model is:

[0017]

[0018] Among them, a k is the distortion curve parameter, x is the field of view data, and y is the field of view data after distortion.

[0019] Further, in step (4), the I R , I G and I B The data are R, G, and B color data in the RGB space, based on the corrected visible light image I disvis and infrared image I disir Assign values ​​to ensure that the information of visible light and infrared images is not lost.

[0020] Furthermore, in step (4), the I R , I G and I B Data, visible light and infrared images are fused in RGB space.

[0021] Furthermore, the fusion uses simple operations (such as addition, subtraction, multiplication, division, and square root operations) in calculation. The simple fusion operation saves the output time of the fused image. The fused image can be output in real time and is easy to implement in hardware. For example, it can be used on different platforms such as DSP, FPGA, ARM, etc. Among them, the FPGA fusion image with parallel processing has the shortest output time.

[0022] Furthermore, in step (4), the color parameters k1, k2, k3, b1, b2, and b3 are used to ensure the color of the visible light and infrared fusion image. It is calculated by multiple color images, usually using several images with different subject contents (such as the selected images usually have sky, house, tree, person, etc.), and extracting the RGB of the color image as I fuzR , I fuzG , I fuzB , where I fuzR , I fuzG , I fuzB Converted to grayscale images are I R , I G , I B , calculate I respectively r , I G , I B Average value and standard deviation δ R , δ G , δ B, and finally get the color parameters k1, k2, k3, b1, b2, b3.

[0023] Furthermore, k1=0.28, k2=0.27, k3=0.32, b1=0.45, b2=0.49, b3=0.51.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for real-time fusion of visible light and infrared images described in the present invention.

[0025] Beneficial effects of the present invention:

[0026] The method of the present invention does not require the acquisition of parameters of visible light and infrared cameras, nor does it require the acquisition of distorted images. By acquiring the field of view data and distorted field of view data of the visible light optical system and the infrared optical system, a field of view distortion curve model is established to perform distortion correction; the visible light image and the infrared image are converted into corrected visible light image and infrared image through the field of view distortion curve model; a fused image RGB space is established, and the corrected visible light image and infrared image are fused to obtain a final fused image. The present application performs field of view distortion correction on visible light and infrared images, and the corrected images are fused in the RGB space. The output time of the fused image is saved through a simple fusion operation, and the pseudo-color fusion effect of the image is good, and the image can be output in real time, which has a wide range of uses. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 : Flowchart of visible and infrared image fusion.

[0028] Figure 2 :Distortion model: Figure 2 Middle: (a) is the visible light field of view distortion curve model, (b) is the infrared field of view distortion curve model

[0029] Figure 3 : Distorted image and corrected image: Figure 3 Middle: (a) is the visible light distorted image, (b) is the infrared distorted image, (c) is the corrected visible light image, and (d) is the corrected infrared image

[0030] Figure 4 : fused image: Figure 4 Middle: (a) is the visible light image, (b) is the infrared image, and (c) is the fused image. DETAILED DESCRIPTION

[0031] The present invention is described in further detail below with reference to the embodiments.

[0032] Those skilled in the art will understand that the following examples are intended only to illustrate the present invention and should not be construed as encompassing the scope of the present invention. Where specific techniques or conditions are not specified in the examples, the techniques or conditions described in the literature in the art or in the product specifications were used. Materials or equipment used without manufacturer identification are commercially available conventional products.

[0033] To illustrate the specific real-time process, assume that the resolution (M×N) of the visible light image and infrared image is 1280×1024. The specific implementation steps are as follows:

[0034] Step 1: Complete the collection of light energy through visible light and infrared optical systems;

[0035] Step 2: Based on the light energy collected in step (1), visible light image I is obtained through visible light and infrared imaging systems. vis and infrared image I ir ;

[0036] Step 3: According to the field of view data and the distorted field of view data obtained by the visible light optical system and the infrared optical system respectively, a visible light field of view distortion curve model and an infrared field of view distortion curve model are established, and the visible light image I obtained in step (2) is converted into vis and infrared image I ir Perform field distortion correction respectively to obtain the corrected visible light image I disvis and infrared image I disir ,include:

[0037] Step (3.1) According to the visible light optical system and the infrared optical system, the field of view data x and the distorted field of view data y are obtained through the optical system, and the visible light field of view distortion curve model and the infrared field of view distortion curve model are respectively established:

[0038]

[0039] Visible light and infrared field of view data x and distorted field of view data y are obtained through the visible light optical system and infrared optical system respectively, and the visible light field of view distortion curve model and infrared field of view distortion curve model are established respectively through the above formula. If the visible light optical system and infrared optical system are designed, the visible light optical system field of view data x and distorted field of view data y, the infrared optical system field of view data x and distorted field of view data y are derived, and the data group is determined according to the degree of distortion. Usually, there are more than 50 data groups. The field of view distortion curve model is obtained as follows: Figure 2 (a) Figure 2 (b) shows the visible light image I vis and infrared image I irThe corrected visible light image I is obtained by respectively using the visible light field distortion curve model and the infrared field distortion curve model. disvis and infrared image I disir ,like Figure 3 shown.

[0040] Step 4: Corrected visible light image I obtained in step 3 disvis and infrared image I disir Fusion is performed to obtain the fused image I fus .

[0041] Where I in RGB space R , I G and I B The data is:

[0042] I R =(I disvis +I disir ) / 2

[0043] I G =I disir

[0044] I B =1-I disir

[0045] Calculate I separately R , I G and I B Average value and standard deviation δ R , δ G , δ B , and enter the following calculation formula to obtain the fused image:

[0046]

[0047] I fuz =(I fuzR , I fuzG , I fuzB )

[0048] Among them, k1=0.28, k2=0.27, k3=0.32, b1=0.45, b2=0.49, b3=0.51, such as Figure 4 The fused image is shown in Figure 1. The color parameters k1, k2, k3, b1, b2, and b3 are calculated from multiple color images, usually more than 100 (the selected images usually include sky, house, tree, and person), and the RGB of the color images are extracted as I fuzR , I fuzG , I fuzB , where I fuzR , IfuzG , I fuzB Converted to grayscale images are I R , I G and I B , calculate I respectively R , I G and I B Average value and and standard deviation δ R , δ G and δ B , and finally get the color parameters k1, k2, k3, b1, b2, b3.

Claims

1. A method for real-time fusion of visible light and infrared images, characterized in that: include: (1) Complete the collection of light energy through visible light and infrared optical systems; (2) Based on the light energy collected in step (1), visible light images I are obtained through visible light and infrared imaging systems. vis and infrared image I ir ; (3) Obtaining the field of view data and the distorted field of view data through the visible light optical system and the infrared optical system respectively, establishing a field of view distortion curve model, and transforming the visible light image I obtained in step (2) into vis and infrared image I ir Perform field distortion correction respectively to obtain the corrected visible light image I disvis and infrared image I disir ; (4) The corrected visible light image I obtained in step (3) is respectively disvis and infrared image I disir Fusion is performed to obtain the fused image I fus , include: Among them I R =(I disvis +I disir ) / 2,I G =I disir , I B =1-I disir ; and I R , I G and I B The mean of M and N represent the number of rows and columns of the image respectively, δ R , δ G , δ B I R , I G , I B The standard deviation of k1, k2, k3, b1, b2, and b3 are color parameters; I fuzR , I fuzG and I fuzB is the fused image I fuz Data in RGB space, where I fuz for: I fuz =(I fuzR ,I fuzG ,I fuzB )。 2. The method for real-time fusion of visible light and infrared images according to claim 1, characterized in that: In step (3), the field of view data x and the distorted field of view data y of the visible light optical system and the infrared optical system are obtained respectively according to the visible light imaging system and the infrared imaging system, and a field of view distortion curve model is established.

3. The method for real-time fusion of visible light and infrared images according to claim 2, characterized in that: The field of view distortion curve model is: Among them, a k is the distortion curve parameter, x is the field of view data, and y is the field of view data after distortion.

4. The method for real-time fusion of visible light and infrared images according to claim 1, characterized in that: In step (4), the I R , I G and I B The data are R, G, and B color data in the RGB space, based on the corrected visible light image I disvis and infrared image I disir Assign values ​​to ensure that the information of visible light and infrared images is not lost.

5. The method for real-time fusion of visible light and infrared images according to claim 1, characterized in that: In step (4), the RGB space I R , I G and I B Data, visible light and infrared images are fused in RGB space.

6. The method for real-time fusion of visible light and infrared images according to claim 5, characterized in that: The fusion operation adopts addition, subtraction, multiplication, division and square root operation, which is used to save the output time of the fused image, is easy to implement in hardware, and realizes the real-time output of the fused image.

7. The method for real-time fusion of visible light and infrared images according to any one of claims 1 to 6, characterized in that: In step (4), the color parameters k1, k2, k3, b1, b2, and b3 are used to ensure the color of the visible light and infrared fusion image.

8. The method for real-time fusion of visible light and infrared images according to claim 7, characterized in that: The color parameters k1, k2, k3, b1, b2, and b3 are obtained by extracting the RGB values ​​of the color images from a number of images with different subject contents. fuzR , I fuzG , I fuzB , where I fuzR , I fuzG , I fuzB Converted to grayscale images are I R , I G and I B , calculate I respectively R , I G , I B Average value and standard deviation δ R , δ G , δ B , and finally get the color parameters k1, k2, k3, b1, b2, b3.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for real-time fusion of visible light and infrared images as described in any one of claims 1 to 8 are implemented.

Images