Visible Light and Infrared Fusion Image Natural Color Restoration Method, Device and Storage Medium

Abstract

The present invention discloses a method, device and storage medium for natural color restoration of visible light and infrared fusion images, which relates to the technical field of image processing. S1. Obtain visible light image data and infrared image data. S2. Preprocess the images. S3. Convert the preprocessed visible light image data into a grayscale matrix, and adopt a dual-guided first-order fusion method to obtain two first-order initial fusion images. S4. Select three reference images, and convert the reference images into the YUV space respectively to obtain the values of the three channels in the YUV space. S5. Calculate the YUV components of the reference images through the channel mean and standard deviation, and then transfer them to the two first-order initial fusion images to obtain two initial natural color restoration images. S6. Perform second-order weight ratio fusion on the two initial natural color restoration images to obtain the final result. The colors of the images fused by this method have a good natural sense, the fusion and restoration results are more in line with the visual characteristics of the human eye, and there is no color spill phenomenon.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device and storage medium for restoring the natural color of visible light and infrared fusion images. Background Art

[0002] The technology of restoring the natural color of visible light and infrared fusion images is one of the important directions for the development of high-performance night vision technology at home and abroad today. This technology effectively improves people's ability to detect targets and understand scenes. The natural color restoration technology based on the human eye visual characteristics makes full use of the image information in the visible light and infrared bands, which can improve the target recognition speed and accuracy of observers by 30% - 60%, and has received extensive attention at home and abroad. At present, there are already applications of related natural color night vision equipment.

[0003] The traditional technology for restoring the natural color of visible light and infrared fusion images uses a global color transfer algorithm based on a single reference image. The hue of the restored image is greatly affected by the reference image, and it is difficult to ensure the adaptability to various scenes in practical applications. At the same time, the restored image is prone to color bleeding due to the influence of the reference image, which is not conducive to human eye visual perception. In order to achieve the natural color restoration of infrared images that conforms to the human eye visual characteristics, there is an urgent need for a method for restoring the natural color of visible light and infrared fusion images, which can improve the display effect of the image while retaining the targets in the infrared image and meet the human eye visual characteristics. Summary of the Invention

[0004] In view of the above technical problems, the purpose of the present invention is to provide a method for restoring the natural color of visible light and infrared fusion images, which can make the fusion restoration result more in line with the human eye visual characteristics and there is no color bleeding phenomenon, aiming at the defects of the existing technology.

[0005] The present invention adopts the following technical solutions: A method for restoring the natural color of visible light and infrared fusion images, comprising the following steps:

[0006] S1. Obtain visible light image data and infrared image data, and perform registration processing on them;

[0007] S2. Preprocess the visible light image data with Gaussian filtering, and preprocess the infrared image data with histogram equalization;

[0008] S3. Change the preprocessed visible light image data into a grayscale matrix, and adopt the double-guided first-order fusion method to fuse the visible light image and the infrared image in the YUV space to obtain two first-order initial fusion images;

[0009] S4. Select three reference images, and convert all the reference images into the YUV space to obtain the values of the three channels in the YUV space;

[0010] S5. Calculate the YUV components of the converted reference image through the channel mean and standard deviation, and then transfer them to the two initial first-order fusion images to obtain two initial natural-like color restoration images;

[0011] S6. Convert the two initial natural-like color restoration images to the RGB space and perform second-order weight ratio fusion to obtain the result.

[0012] Another object of the present invention is to provide a natural-like color restoration device for visible light and infrared fusion images, including a processor, and

[0013] an acquisition module for acquiring visible light image data and infrared image data,

[0014] a storage module, on which a program for natural-like color restoration of visible light and infrared fusion images that can run on the processor is stored. When the program for natural-like color restoration of visible light and infrared fusion images is executed by the processor, the steps of the above-mentioned natural-like color restoration method for visible light and infrared fusion images are realized,

[0015] an output module for outputting the calculation result.

[0016] Another object of the present invention is to provide a computer-readable storage medium, in which program codes executable by a processor are stored. The computer-readable storage medium includes multiple instructions, and the multiple instructions are configured to enable the processor to execute the above-mentioned natural-like color restoration method for visible light and infrared fusion images.

[0017] The beneficial effects of the present invention are as follows: The visible light and infrared image fusion method has better environmental adaptability than conventional algorithms. The color of the fusion image has a better natural sense, and the algorithm processing volume is small, which has little impact on the operation speed of the existing real-time hardware fusion processing algorithm. It is a natural-sense color fusion processing algorithm with strong environmental adaptability; the fusion restoration result is more in line with the visual characteristics of the human eye, and there is no color bleeding phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of the method of the present invention;

[0019] Figure 2 is a demonstration diagram of the restoration effect. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to have a clearer understanding of the technical features, objects, and beneficial effects of the present invention, the technical solutions of the present invention are described in detail below, but it should not be understood as a limitation to the implementable scope of the present invention.

[0021] As Figure 1As shown, a method for restoring the natural color of visible light and infrared fusion images includes the following steps:

[0022] S1. Obtain visible light image data and infrared image data, and perform registration processing on them;

[0023] In order to realize the fusion of visible light image data and infrared image data, first, it is necessary to obtain infrared images and visible light images from the same perspective. Among them, visible light images can be obtained by using common visible light sensors or cameras, and infrared images can be obtained by using cooled or uncooled infrared detectors. In this embodiment, the devices and methods for obtaining visible light image data and infrared image data are all common devices and methods in the art, so they will not be elaborated here.

[0024] At the same time, there are various methods for image registration. For example, region-based correlation methods, mutual information methods, Fourier transform methods, etc., feature-based fixed feature descriptor methods, pyramid methods, and physical registration methods can all be applied to this embodiment. In this embodiment, the more convenient physical registration method is selected as the registration method.

[0025] S2. Preprocess the visible light image data with Gaussian filtering, and preprocess the infrared image data with histogram equalization. When processing the visible light image data, perform weighted averaging on the entire image. After preprocessing, the value of each pixel point in the visible light image is obtained by weighted averaging of itself and other pixel values in the neighborhood, thereby removing the noise in the visible light image. At the same time, in order to enhance the contrast of the infrared image data, histogram equalization is used to preprocess the infrared image. Since Gaussian filtering and histogram equalization both belong to the prior art, the specific operation methods thereof will not be elaborated in this embodiment.

[0026] S3. Change the preprocessed visible light image data into a gray matrix, and use the double-guided first-order fusion method to fuse the visible light image and the infrared image in the YUV space to obtain two first-order initial fusion images;

[0027] Since the infrared image data itself is a grayscale matrix, in this step, only the visible light image data is transformed into a grayscale matrix. There are many methods to transform the visible light image data into a grayscale matrix, such as the classic formula: Gray = R * 0.299 + G * 0.587 + B * 0.114, or Gray = (R * 299 + G * 587 + B * 114 + 500) / 1000, or Gray = (R * 30 + G * 59 + B * 11 + 50) / 100, etc. These methods in the prior art for converting RGB images (visible light images are RGB images) into grayscale matrices can all be used in this embodiment. For the convenience of calculation, in this embodiment, the formula Gray = (R * 299 + G * 587 + B * 114 + 500) / 1000 is selected to transform the visible light image data into a grayscale matrix.

[0028] After obtaining the grayscale matrix of the visible light image data, in a double-guided first-order fusion manner, the visible light image and the infrared image are fused in the YUV space to obtain two first-order initial fusion images. The fusion is carried out through the following formula:

[0029]

[0030] In the formula, and are respectively the values of the three channels of the two first-order initial fusion images in the YUV space. V is (i, j) and IR(i, j) are respectively the original visible light image and the original infrared image. The values of a1, b1, a2, b2, a3, b3, c1, d1, c2, d2, c3, d3 range from 0 to 1 and a1 = c1 and b1 = d1. The selection of a2, b2, a3, b3 and c2, d2, c3, d3 needs to make and remain within the range of 0 to 255;

[0031] From the inventor's experience, for the weights a2, b2, a3, b3 and c2, d2, c3, d3, as long as they meet the above conditions, they can all be applied to the present invention. The only difference caused by different weight values is the color effect.

[0032] After the fusion through the above formula, two first-order initial fusion images are obtained.

[0033] S4. Select three reference images and convert all the reference images into the YUV space to obtain the values of the three channels in the YUV space;

[0034] For the reference image, its main function is to perform color transfer on the first-order initial fusion image obtained in S3. Therefore, the reference image should be a visible light image. At the same time, in order to facilitate color transfer, an image with rich colors should be selected as the reference image. For example, images of richly colored sea and sky, plants, towns, etc. can be used as reference images.

[0035] Regarding the number of reference images, theoretically, only one reference image is required to achieve subsequent color transfer. However, when using only one reference image for color transfer, the color is relatively thin and prone to transfer errors. Therefore, in this embodiment, at least two reference images are required. Theoretically, the more reference images, the better. However, the more reference images, the greater the computational amount and the slower the calculation speed. Therefore, the number of reference images is preferably 3.

[0036] After selecting the reference images, all the reference images are converted to the YUV space, and the following formula can be used for conversion:

[0037]

[0038] S5. Calculate the YUV components of the converted reference images through the channel mean and standard deviation, and then transfer them to the two first-order initial fusion images to obtain two initial pseudo-natural color restoration images;

[0039] In this step, by using the U and V channel values of the three reference images and the two first-order initial fusion images, the combined weight factors of the reference images are calculated, and then the channel mean and standard deviation after combining the three reference images are calculated. Then, the colors of the three reference images are transferred to the two first-order initial fusion images in the YUV space, and finally two initial pseudo-natural color restoration images are obtained. The specific operations are as follows:

[0040] First, calculate the difference metric of the standard deviation and mean value between the two first-order initial fusion images and the reference images through the following formula:

[0041]

[0042] In the formula, is the difference metric between the two first-order initial fusion images and all reference images; is the U space matrix of the two first-order initial fusion images; U i is the U space matrix of all reference images; is the V space matrix of the two first-order initial fusion images; V i is the V space matrix of all reference images; is the standard deviation of the U space matrix of all reference images; is the standard deviation of the V space matrix of all reference images; i is the number of the reference image, and i = 1, 2, 3; j is the number of the first-order initial fusion image, and j = 1, 2;

[0043] Then, the combination coefficients for the color transfer of the reference images are determined by the following formula:

[0044]

[0045] Subsequently, the three reference images are combined, and the standard deviation and average value of the three reference images for the YUV three channels are calculated by the following formula:

[0046]

[0047] In the formula, is the average value after the combination of the three reference images, is the standard deviation after the combination of the three reference images, is the combination coefficient in the color transfer, is the average value of the three reference images in the YUV three channels, is the standard deviation of the three reference images in the YUV three channels. i = 1, 2, 3 is the number of the reference image; j = 1, 2 is the color transfer number in the two initial fusion processes.

[0048] Finally, two initial pseudo-natural color restoration images are calculated by the following formula:

[0049]

[0050] In the formula, is the standard deviation of the YUV three channels of the two first-order initial fusion images, is the average value of the YUV three channels of the two first-order initial fusion images, Y z 、U z 、V z are the three-channel values of the two finally obtained initial pseudo-natural color restoration images, and are respectively the standard deviation and average value of the YUV three channels obtained after the combination of the three reference images, z is the number of the initial pseudo-natural color restoration image, z = 1, 2 and z = j.

[0051] S6. Convert the two initial pseudo-natural color restoration images to the RGB space and perform second-order weighted ratio fusion to obtain.

[0052] First, convert the two initial pseudo-natural color restoration images obtained in S5 to the RGB space:

[0053]

[0054] Then, the two initial natural-color restored images are fused through weight ratio, and finally a natural-color restored image is obtained. The calculation process is as follows:

[0055]

[0056] In the formula, R 0 、R 1 、R 2 are the R values of the fused image, the first initial natural-color restored image, and the second initial natural-color restored image respectively; G 0 、G 1 、G 2 are the G values of the fused image, the first initial natural-color restored image, and the second initial natural-color restored image respectively; B 0 、B 1 、B 2 are the B values of the fused image, the first initial natural-color restored image, and the second initial natural-color restored image respectively; k1 and k2 are weight ratio factors. Both k1 and k2 are in the range of 0 to 1, and k1 + k2 = 1. At the same time, through a large number of experiments by the inventor, it is found that when k1 and k2 are selected as 0.5, it has a better effect.

[0057] Finally, the picture fused by the method of this embodiment is as Figure 2 shown. As can be seen from Figure 2 , compared with the two original images, the final natural-color restored image retains the details in the original images, and at the same time is converted from the original black-and-white image into a natural-color image based on the human eye's visual characteristics; compared with the two initial natural-color restored images, the final natural-color restored image does not show the phenomenon of color bleeding, and the overall color restoration effect is better than that of the two initial natural-color restored images.

[0058] This embodiment also provides a natural-color restoration device for visible light and infrared fusion images, including,

[0059] a processor, and

[0060] an acquisition module for acquiring visible light image data and infrared image data,

[0061] a storage module, on which a program for natural-color restoration of visible light and infrared fusion images that can run on the processor is stored. When the program for natural-color restoration of visible light and infrared fusion images is executed by the processor, it realizes the steps of the above-mentioned natural-color restoration method for visible light and infrared fusion images,

[0062] an output module for outputting the calculation result.

[0063] This embodiment also provides a computer-readable storage medium, which stores program codes executable by a processor. It is characterized in that the computer-readable storage medium includes multiple instructions, and the multiple instructions are configured to enable the processor to execute the above-mentioned natural color restoration method for visible light and infrared fusion images.

[0064] As mentioned above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for restoring the natural color of visible light and infrared fusion images, characterized in that, It includes the following steps: S1. Obtain visible light image data and infrared image data, and perform registration processing on them; S2. Preprocess the visible light image data by Gaussian filtering, and preprocess the infrared image data by histogram equalization; S3. Change the preprocessed visible light image data into a grayscale matrix, and in the YUV space, fuse the visible light image and the infrared image in a double-guided first-order fusion manner to obtain two first-order initial fusion images; It includes the following sub-steps: After converting the visible light image data into a grayscale matrix, use the following formula to calculate to obtain two first-order initial fusion images: Wherein, and are the values of the three channels of two first-order initial fusion images in the YUV space, respectively, and V is (i,j) and IR(i,j) are the visible light original image and the infrared original image respectively. The value ranges of a1, b1, a2, b2, a3, b3, c1, d1, c2, d2, c3, and d3 are between 0 and 1, and a1 = c1 and b1 = d1. The selection of a2, b2, a3, b3, c2, d2, c3, and d3 needs to make and remain within the range of 0 to 255; S4. Select three reference images, and convert all the reference images into the YUV space to obtain the values of the three channels in the YUV space; S5. Calculate the channel mean and standard deviation after combining the three reference images, and then transfer the colors of the three reference images in the YUV space to the two first-order initial fusion images to finally obtain two initial natural color-like restored images; S6. Convert the two initial natural color-like restored images into the RGB space and perform second-order weight ratio fusion, which includes the following sub-steps: Convert the two initial natural color-like restored images into the RGB space, and then fuse the two initial natural color-like restored images through the following formula: Wherein, R 0 , R 1 , R 2 are the R values of the fused image, the first initial pseudo-natural color restored image, and the second initial pseudo-natural color restored image, respectively; G 0 , G 1 , G 2 are the G values of the fused image, the first initial pseudo-natural color restored image, and the second initial pseudo-natural color restored image, respectively; B 0 , B 1 , B 2 are the B values of the fused image, the first initial pseudo-natural color restored image, and the second initial pseudo-natural color restored image, respectively; k1 and k2 are weight ratio factors, both k1 and k2 are in the range of 0 to 1, and k1 + k2 = 1.

2. The method according to claim 1, characterized in that, The specific operation of S5 is: Calculate the difference metric between the two first-order initial fusion images and the reference images; In the formula, is the difference metric between two first-order initial fusion images and all reference images; is the U-space matrix of two first-order initial fusion images; U i is the U-space matrix of all reference images; is the V-space matrix of two first-order initial fusion images; V i is the V-space matrix of all reference images; is the standard deviation of the U-space matrix of all reference images; is the standard deviation of the V-space matrix of all reference images; i is the number of the reference image, and i = 1, 2, 3; j is the number of the first-order initial fusion image, and j = 1, 2; Calculate the combination coefficient for color transfer of the reference images; Calculate the standard deviation and average value of the YUV three channels of the three reference images; wherein, is the average value after combining three reference images; is the standard deviation after combining three reference images; is the combination coefficient in color transfer; is the average value of three reference images in the YUV three channels; is the standard deviation of three reference images in the YUV three channels; i = 1, 2, 3 are reference images; j = 1, 2 are the color transfer numbers of two initial fusion processes; Obtain the initial natural color-like restored images: In the formula, is the standard deviation of the YUV three channels of two first-order initial fusion images, is the average value of the YUV three channels of two first-order initial fusion images, Y z , U z , V z are the three-channel values of two finally obtained initial pseudo-natural color restoration images, and are respectively the standard deviation and the average value of the YUV three channels obtained after combining three reference images. z is the number of the initial pseudo-natural color restoration image, z = 1, 2 and z = j.

3. A natural color restoration device for visible light and infrared fusion images, characterized in that, including, a processor, and an acquisition module for obtaining visible light image data and infrared image data, a storage module on which a program for natural color restoration of visible light and infrared fusion images that can run on the processor is stored. When the program for natural color restoration of visible light and infrared fusion images is executed by the processor, it implements the steps of the method described in claim 1 or 2, an output module for outputting the calculation results.

4. A computer-readable storage medium storing program code executable by a processor, characterized in that, The computer-readable storage medium includes multiple instructions, and the multiple instructions are configured to enable the processor to execute the method for natural color restoration of visible light and infrared fusion images described in claim 1 or 2.

Images