An image fusion method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides an image fusion method, apparatus, electronic device, and storage medium. The method includes: obtaining a target visible light image and a target infrared image; determining a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image; determining a first fusion weight corresponding to each pixel point of the target infrared image based on the brightness difference feature map and a preset correspondence between the difference feature and the fusion weight; calculating a second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight; and performing image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image. By using this method, a fused image with accurate color is obtained on the premise of ensuring the quality of the fused image. The fused image has the characteristics of accurate color information and high signal-to-noise ratio.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image fusion method, apparatus, electronic device and storage medium. Background Art

[0002] The urban road checkpoint system is a system that automatically and real-time captures and records passing vehicles by installing cameras at checkpoints such as urban entrances and exits and toll stations. In order to make the captured images clearer, the urban road checkpoint system usually uses a stroboscopic light for fill light when capturing passing vehicles. Therefore, the problem of light pollution caused by stroboscopic lights is becoming more and more common. In order to avoid light pollution, how to improve the imaging quality in low-illumination scenarios has always been a hot demand in the monitoring field.

[0003] Currently, it is usually to simultaneously capture an infrared image and a visible light image of the same scene, and utilize the high signal-to-noise ratio characteristic of the infrared image to fuse the infrared image and the visible light image to obtain a fused image with higher brightness, so as to reduce the intensity of the fill light and reduce light pollution.

[0004] However, for some objects with large differences in infrared reflection characteristics, there will be certain color distortion in the fused image obtained by fusing the infrared image and the visible light image. The color distortion of the fused image can be corrected by overall restricting the weight of the brightness of the infrared image in the fused image. However, restricting the weight will result in the inability to ensure the high signal-to-noise ratio and maximum information content of the fused image, that is, the quality of the fused image will be poor. That is to say, currently, it is impossible to obtain a fused image with accurate colors on the premise of ensuring the quality of the fused image. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide an image fusion method, apparatus, electronic device and storage medium to obtain a fused image with accurate colors on the premise of ensuring the quality of the fused image.

[0006] In a first aspect, an embodiment of the present invention provides an image fusion method, including:

[0007] Obtain a target visible light image and a target infrared image;

[0008] Determine a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image; wherein, the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image;

[0009] Determine the first fusion weight corresponding to each pixel of the target infrared image based on the luminance difference feature map and the preset correspondence between the difference feature and the fusion weight; wherein, the correspondence is an inverse correlation between the difference feature and the fusion weight;

[0010] Calculate the second fusion weight corresponding to each pixel of the target visible light image according to the first fusion weight;

[0011] Perform image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image.

[0012] Optionally, after performing image fusion on the target visible light image and the target infrared image to obtain a fused image, it further includes:

[0013] Determine the preset color corresponding to each pixel according to the R, G, and B values of each pixel of the fused image;

[0014] Determine the color cast correction parameter corresponding to each pixel based on the preset color corresponding to each pixel of the fused image and the R, G, and B values of each pixel;

[0015] Determine the luminance parameter corresponding to each pixel of the fused image according to the luminance of the fused image and the luminance of the target visible light image;

[0016] Perform correction on the R, G, and B values of each pixel based on the color cast correction parameter corresponding to each pixel of the fused image, the R, G, and B values of each pixel, and the luminance parameter corresponding to each pixel to obtain a corrected fused image.

[0017] Optionally, the performing correction on the R, G, and B values of each pixel based on the color cast correction parameter corresponding to each pixel of the fused image, the R, G, and B values of each pixel, and the luminance parameter corresponding to each pixel to obtain a corrected fused image includes:

[0018] For each pixel of the fused image, calculate the correction value corresponding to each color channel of the pixel according to the gray value of each color channel of the pixel, the luminance parameter, the minimum value among the difference between the maximum gray value and the gray value of the channel, and the color cast correction parameter corresponding to the pixel;

[0019] Subtract the correction value corresponding to each color channel of the pixel from the gray value of each color channel of the pixel to obtain a corrected fused image.

[0020] Optionally, determining the brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image includes:

[0021] Perform low-pass filtering on the brightness channels of the target visible light image and the brightness channels of the target infrared image respectively to obtain a visible light brightness feature image and an infrared brightness feature image;

[0022] Calculate the absolute value of the brightness difference between the pixel points at the corresponding positions in the visible light brightness feature image and the infrared brightness feature image, and obtain an image with the absolute value as the pixel value as the brightness difference feature map.

[0023] Optionally, performing image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image includes:

[0024] Calculate the difference in pixel values of the brightness channels between each pixel point at the corresponding positions in the target infrared image and the infrared brightness feature image to obtain an infrared image detail texture layer;

[0025] Perform image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain a fused image.

[0026] Optionally, performing image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain a fused image includes:

[0027] Sum the product of the Y value of each pixel point of the target infrared image and the first fusion weight, the product of the Y value of the pixel point at the same position as this pixel point in the target visible light image and the second fusion weight, and the pixel value of the pixel point at the same position as this pixel point in the infrared image detail texture layer, and use the obtained sum value as the Y value of the pixel point at this position in the fused image. And, based on the ratio of the Y value of the pixel point at this position in the fused image to the Y value of the pixel point at this position in the target visible light image as the brightness gain, and determine the products of the U and V values of the pixel point at this position in the target visible light image and the brightness gain as the U and V values of the pixel point at this position in the fused image respectively to obtain a fused image.

[0028] Optionally, determining the first fusion weight corresponding to each pixel point of the target infrared image based on the brightness difference feature map and the corresponding relationship between the preset difference feature and the fusion weight includes:

[0029] For each pixel of the target infrared image, if the pixel value corresponding to the pixel in the brightness difference feature map is not greater than a preset difference threshold, determine that the first fusion weight corresponding to the pixel is the target weight value;

[0030] If the pixel value corresponding to the pixel in the brightness difference feature map is greater than the preset difference threshold, based on the pixel value and a preset weight function, determine the first fusion weight corresponding to the pixel; wherein, the preset weight function indicates that the first fusion weight linearly decreases with the increase of the pixel value on the basis of the target weight value.

[0031] In a second aspect, an embodiment of the present invention provides an image fusion device, including:

[0032] An image acquisition module, configured to acquire a target visible light image and a target infrared image;

[0033] A difference feature determination module, configured to determine a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image; wherein, the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image;

[0034] A first weight determination module, configured to determine the first fusion weight corresponding to each pixel of the target infrared image based on the brightness difference feature map and a preset correspondence between the difference feature and the fusion weight; wherein, the correspondence is an inverse correlation between the difference feature and the fusion weight;

[0035] A second weight determination module, configured to calculate the second fusion weight corresponding to each pixel of the target visible light image according to the first fusion weight;

[0036] An image fusion module, configured to perform image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0038] The memory is used to store a computer program;

[0039] The processor, when executing the program stored in the memory, implements the method steps described in any one of the first aspects above.

[0040] Fourthly, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method steps described in any one of the above first aspects are implemented.

[0041] Beneficial effects of the embodiment of the present invention:

[0042] By using the method provided by the embodiment of the present invention, a target visible light image and a target infrared image are obtained; according to the brightness of the target visible light image and the brightness of the target infrared image, a brightness difference feature map between the target visible light image and the target infrared image is determined; based on the brightness difference feature map and the corresponding relationship between the differential feature and the fusion weight preset, the first fusion weight corresponding to each pixel point of the target infrared image is determined; according to the first fusion weight, the second fusion weight corresponding to each pixel point of the target visible light image is calculated; based on the first fusion weight and the second fusion weight, the target visible light image and the target infrared image are subjected to image fusion to obtain a fused image. That is, the first fusion weight corresponding to each pixel point of the target infrared image can be determined through the brightness difference feature map and the corresponding relationship between the preset differential feature and the fusion weight, and the second fusion weight corresponding to each pixel point of the target visible light image is calculated according to the first fusion weight. Each pixel point at the corresponding position of the target visible light image and the target infrared image can be fused according to the first fusion weight and the second fusion weight, so that the brightness from the target infrared image and the brightness from the target visible light image in each pixel point of the obtained fused image reach the best ratio. Since the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image, and there is an inverse correlation between the differential feature and the fusion weight, for the region with a large infrared reflection characteristic difference, the value of the first fusion weight corresponding to the pixel point of the target infrared image in this region can be reduced, and the brightness of the image in this region from the infrared image in the fused image can be reduced, thereby avoiding color distortion of the image in this region in the fused image. For the region with a small infrared reflection characteristic difference, the value of the first fusion weight corresponding to the pixel point of the target infrared image in this region can be increased, and the brightness of the image in this region from the infrared image in the fused image can be increased, and the brightness of the image in this region in the fused image can be improved. That is, the method provided by the embodiment of the present invention can obtain a fused image with accurate color on the premise of ensuring the quality of the fused image. The fused image has both the color information of visible light and the advantage of high signal-to-noise ratio of the infrared light image. Therefore, by using the method provided by the embodiment of the present invention, a clear color image can be obtained in a low-light scene.

[0043] Of course, it is not necessary for any product or method implementing the present invention to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0045] Figure 1 A schematic diagram of a visible light image, an infrared image, and a fused image;

[0046] Figure 2 Another schematic diagram of a visible light image, an infrared image, and a fused image;

[0047] Figure 3 A flowchart of an image fusion method provided by an embodiment of the present invention;

[0048] Figure 4 A schematic comparison diagram of an infrared image and a visible light image;

[0049] Figure 5 Another schematic comparison diagram of an infrared image and a visible light image;

[0050] Figure 6 Another schematic comparison diagram of an infrared image and a visible light image;

[0051] Figure 7 A schematic diagram of obtaining a brightness difference feature map provided by an embodiment of the present invention;

[0052] Figure 8 A schematic diagram of differential feature mapping provided by an embodiment of the present invention;

[0053] Figure 9 A flowchart of performing color cast correction on the fused image provided by an embodiment of the present invention;

[0054] Figure 10 A schematic diagram of the corrected fused image provided by an embodiment of the present invention;

[0055] Figure 11 A schematic structural diagram of an image fusion device provided by an embodiment of the present invention;

[0056] Figure 12 Another schematic structural diagram of an image fusion device provided by an embodiment of the present invention;

[0057] Figure 13Schematic diagram of the structure of the electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the protection scope of the present invention.

[0059] Due to the increasingly common problem of light pollution caused by stroboscopic lights, in order to avoid light pollution, currently the urban road checkpoint system usually reduces the intensity of the stroboscopic lights and captures images of passing vehicles in low-illumination scenarios. However, the imaging quality in low-illumination scenarios is poor. In order to improve the imaging quality in low-illumination scenarios, currently, it is usually to simultaneously capture infrared images and visible light images of the same scene, and fuse the infrared images and visible light images to obtain a fused image with higher brightness.

[0060] Since visible light images have color information and infrared images have characteristics such as high signal-to-noise ratio, in low-illumination scenarios, infrared images and visible light images can be fused to obtain a clear color image - the fused image. However, for some objects with infrared reflective materials, the fused images obtained by fusing infrared images and visible light images often have problems of color distortion. Figure 1 Schematic diagram of a visible light image, an infrared image, and a fused image. As Figure 1 shown, Figure 1 in the visible light image 101 and the infrared image 102 are fused to obtain the fused image 103. Since the vehicle body has infrared reflective characteristics ( Figure 1 the vehicle shell color in is red), there is a problem of color distortion in the body color of the obtained fused image 103 compared with the real color.

[0061] In order to avoid the problem of color distortion in the fused image, some methods can correct the color distortion of the fused image by overall restricting the weight of the brightness of the infrared image in the fused image. Figure 2 Schematic diagram of another visible light image, infrared image, and fused image. As Figure 2 shown, Figure 2 the vehicle body has infrared reflective characteristics ( Figure 2 the vehicle shell color in is red), and Figure 2The visible light image 201 and the infrared image 202 are fused to obtain a fused image 203. When performing image fusion, the weight of the brightness of the infrared image 202 in the fused image 203 is globally restricted. Therefore, the problem of color distortion in the obtained fused image 203 is solved to a certain extent. However, since the weight of the brightness of the infrared image 202 in the fused image 203 is globally restricted, the brightness of the obtained fused image 203 is affected, and the signal-to-noise ratio and information content of the fused image 203 are also restricted, resulting in poor image quality. That is to say, there are certain color deviation problems or insufficient utilization of infrared brightness information in the fused images obtained by fusing infrared images and visible light images at present. That is, it is currently impossible to obtain a fused image with accurate colors while ensuring the quality of the fused image.

[0062] In order to obtain a fused image with accurate colors while ensuring the quality of the fused image, an embodiment of the present invention provides an image fusion method, device, electronic device, computer-readable storage medium, and computer program product.

[0063] The image fusion method provided by the embodiment of the present invention can be applied to any electronic device capable of performing image processing. For example, it can be a computer, a camera, a processing device, etc., which is not specifically limited here. The electronic device for image processing can perform processing such as filtering and image fusion on images.

[0064] Figure 3 It is a flowchart of the image fusion method provided by the embodiment of the present invention. As Figure 3 shown, the method includes:

[0065] S301, obtaining a target visible light image and a target infrared image.

[0066] S302, determining a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image.

[0067] Among them, the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image.

[0068] S303, determining a first fusion weight corresponding to each pixel point of the target infrared image based on the brightness difference feature map and a preset correspondence between the difference feature and the fusion weight.

[0069] Among them, the correspondence is an inverse correlation relationship between the difference feature and the fusion weight.

[0070] S304, calculating a second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight.

[0071] S305, based on the first fusion weight and the second fusion weight, perform image fusion on the target visible light image and the target infrared image to obtain a fused image.

[0072] Adopt the method provided by the embodiment of the present invention to obtain a target visible light image and a target infrared image; determine the brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image; based on the brightness difference feature map and the corresponding relationship between the differential feature and the fusion weight preset, determine the first fusion weight corresponding to each pixel point of the target infrared image; calculate the second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight; based on the first fusion weight and the second fusion weight, perform image fusion on the target visible light image and the target infrared image to obtain a fused image. That is, the first fusion weight corresponding to each pixel point of the target infrared image can be determined through the brightness difference feature map and the corresponding relationship between the differential feature and the fusion weight preset, the second fusion weight corresponding to each pixel point of the target visible light image can be calculated according to the first fusion weight, and each pixel point at the corresponding position of the target visible light image and the target infrared image can be fused according to the first fusion weight and the second fusion weight, so that the brightness from the target infrared image and the brightness from the target visible light image in each pixel point of the obtained fused image reach the best ratio. Since the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image, and there is an inverse correlation between the differential feature and the fusion weight, for the region with a large infrared reflection characteristic difference, the value of the first fusion weight corresponding to the pixel point of the target infrared image in this region can be reduced, reducing the brightness of the image in this region from the infrared image in the fused image, thereby avoiding color distortion of the image in this region in the fused image. For the region with a small infrared reflection characteristic difference, the value of the first fusion weight corresponding to the pixel point of the target infrared image in this region can be increased, increasing the brightness of the image in this region from the infrared image in the fused image and enhancing the brightness of the image in this region in the fused image. That is, the method provided by the embodiment of the present invention can obtain a fused image with accurate color on the premise of ensuring the quality of the fused image. The fused image has both the color information of visible light and the advantage of high signal-to-noise ratio of the infrared light image. Therefore, the method provided by the embodiment of the present invention can obtain a clear color image in a low-illumination scene.

[0073] In the embodiment of the present invention, for the same captured scene, an infrared camera and a visible light camera can be used to simultaneously capture images of the scene to obtain a target infrared image and a target visible light image.

[0074] Alternatively, the scene to be photographed can also be photographed by a dual-sensor camera to obtain an infrared image and a visible light image of the scene. The dual-sensor camera uses two starlight-level image sensors. One of the sensors captures infrared light and senses the infrared light band to obtain an infrared image, providing brightness information. The other sensor senses the visible light band to obtain a visible light image, providing environmental color information and brightness information. Specifically, the shutter gain aperture value can be preset, and the dual-sensor camera can capture the scene to be photographed to obtain a visible light image and an infrared light image. After image signal processing and image registration, the YUV data of the visible light image and the infrared light image can be obtained. Among them, YUV is a color space of an image, where "Y" represents luminance, and "U" and "V" represent chrominance and concentration.

[0075] The brightness performance of the infrared image for the same color is inconsistent and varies greatly. Figure 4 FIG. is a comparison schematic diagram of an infrared image and a visible light image. Figure 5 FIG. is another comparison schematic diagram of an infrared image and a visible light image. Figure 6 FIG. is yet another comparison schematic diagram of an infrared image and a visible light image. Figure 4 includes a visible light image 401 and an infrared image 402. Figure 5 includes a visible light image 501 and an infrared image 502. Figure 6 includes a visible light image 601 and an infrared image 602. Figure 4 、 Figure 5 and Figure 6 The bodies of the vehicles in are all red. As shown in Figure 4 、 Figure 5 and Figure 6 For the vehicles that are all red, in the infrared images 402, 502, and 602, some are bright and some are dim. Moreover, the stripes on the clothes of the person in the visible light image 601 are not visible in the infrared image 602, and the black seat belts all turn gray in the infrared image. If the characteristics of the inconsistent brightness performance of the infrared image for the same color are not distinguished, it will inevitably lead to color cast problems in the fused image.

[0076] In the embodiments of the present invention, the brightness difference feature map between the visible light image and the infrared image is used for image fusion. The brightness difference feature map between the visible light image and the infrared image can well reflect the image areas in the fused image that are prone to color cast. Therefore, the embodiments of the present invention can use the brightness difference feature map to perform color cast correction on the image areas in the fused image that are prone to color cast when fusing the images, avoiding color distortion problems such as color cast in the fused image obtained after fusion.

[0077] Figure 7A schematic diagram for obtaining a luminance difference feature map provided by an embodiment of the present invention is as follows. Figure 7 As shown in the figure, by performing a difference operation on the visible light image 701 and the infrared image 702, the corresponding luminance difference feature map 703 can be obtained. In the luminance difference feature map 703, the brighter the image region, the greater the difference in the infrared reflection characteristics of the corresponding region in the infrared image 702 and the visible light image 701, and the more likely a color cast problem occurs, such as the image region corresponding to the driver's upper garment in the luminance difference feature map 703; conversely, the darker the image region in the luminance difference feature map 703, the smaller the difference in the infrared reflection characteristics of the corresponding region in the infrared image 702 and the visible light image 701, and the less likely a color cast problem occurs, such as the image region corresponding to the steering wheel in the luminance difference feature map 703. That is to say, the infrared reflection is stronger than the visible light. In a region that is very dark in the visible light image and very bright in the infrared image, reducing the luminance of this region in the fused image can restore the true color.

[0078] In a possible implementation manner, determining the luminance difference feature map between the target visible light image and the target infrared image according to the luminance of the target visible light image and the luminance of the target infrared image may include the following steps A1 - A2:

[0079] Step A1: Perform low - pass filtering on the luminance channels of the target visible light image and the target infrared image respectively to obtain a visible light luminance feature image and an infrared luminance feature image.

[0080] In this implementation manner, a filter capable of removing noise can be selected to perform low - pass filtering on the luminance channels of the target visible light image and the target infrared image respectively. For example, a Gaussian filter with a scale of 3 can be selected to perform local smoothing and remove noise on the luminance channels of the target visible light image and the target infrared image to obtain a visible light luminance feature image and an infrared luminance feature image:

[0081]

[0082] where K(x) is a low - pass edge - preserving filter operator, such as a Gaussian filter operator, etc., F vis is the visible light luminance feature image, F nir is the infrared luminance feature image, Y vis is the target visible light image, Y nir is the target infrared image, x is the pixel point at the same position in the target visible light image and the target infrared image, represents the low - pass filtering operation.

[0083] Step A2: Calculate the absolute value of the luminance difference between the pixel points at the corresponding positions in the visible-light luminance feature image and the infrared luminance feature image, and obtain an image with the absolute value as the pixel value as the luminance difference feature map.

[0084] Specifically, the luminance difference feature map can be calculated using the following formula:

[0085] diff(x) = abs(F nir (x) - F vis (x))

[0086] where abs is the operation of taking the absolute value, diff(x) is the pixel value of the pixel point x in the luminance difference feature map, F vis (x) is the pixel value of the luminance channel of the pixel point at the same position as the pixel point x in the visible-light luminance feature image, and F nir (x) is the pixel value of the luminance channel of the pixel point at the same position as the pixel point x in the infrared luminance feature image.

[0087] In a possible implementation manner, determining the first fusion weight corresponding to each pixel point of the target infrared image based on the luminance difference feature map and the corresponding relationship between the preset difference feature and the fusion weight may include the following steps B1 - B2:

[0088] Step B1: For each pixel point of the target infrared image, if the pixel value corresponding to the pixel point in the luminance difference feature map is not greater than the preset difference threshold, determine that the first fusion weight corresponding to the pixel point is the target weight value.

[0089] The target weight value can be specifically set according to the actual image fusion scenario. Usually, it can be set to 1, that is, if the pixel value corresponding to the pixel point in the luminance difference feature map is not greater than the preset difference threshold, it means that the reflection characteristic difference between the pixel point and the corresponding pixel point in the visible-light image is not sufficient to cause a color cast problem in the fused image. Of course, the target weight value can also be set to 0.9 or 0.8, etc., which means that the reflection characteristic difference between the pixel point and the corresponding pixel point in the visible-light image will cause a slight color cast problem in the fused image. Therefore, by setting the first fusion weight to the target weight value, the color cast of the fused image can be corrected.

[0090] Step B2: If the pixel value corresponding to the pixel point in the luminance difference feature map is greater than the preset difference threshold, determine the first fusion weight corresponding to the pixel point based on the pixel value and the preset weight function.

[0091] Among them, the preset weight function indicates that the first fusion weight linearly decreases with the increase of the pixel value on the basis of the target weight value.

[0092] Since the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image. If the pixel value corresponding to the pixel point in the brightness difference feature map is greater than the preset difference threshold, it means that the reflection characteristic difference between the pixel point and the corresponding pixel point in the visible light image is too large. If the target visible light image and the target infrared image are fused, the pixel point corresponding to this pixel point in the fused image will have a relatively serious color cast problem. Therefore, the first fusion weight can linearly decrease with the increase of the pixel value on the basis of the target weight value, that is, the smaller the first fusion weight corresponding to the pixel point with a larger reflection characteristic difference between the target infrared image and the target visible light image. For the positions with large reflection characteristic differences in the target visible light image and the target infrared image, the brightness of the pixel points corresponding to the target infrared image at the corresponding positions can be reduced by reducing the first fusion weight of the pixel points at the corresponding positions of the target infrared image, so as to achieve the effect of correcting color cast.

[0093] In the embodiment of the present invention, specifically, the following formula can be used to determine the first fusion weight corresponding to each pixel point of the target infrared image based on the brightness difference feature map and the corresponding relationship between the preset difference feature and the fusion weight:

[0094]

[0095] Among them, α(x) is the first fusion weight corresponding to the pixel point x of the target infrared image, diff(x) is the pixel value corresponding to the pixel point x in the brightness difference feature map, Thresh is the preset difference threshold, and the preset difference threshold can be set according to the actual application scenario and will not be specifically limited here. b is the target weight value, and the target weight value b is usually set to 1 or 0.9, etc. S is the preset weight coefficient, and S is less than 0.

[0096] As can be seen from the above first fusion weight determination formula, for each pixel point of the target infrared image, if the pixel value diff(x) corresponding to this pixel point in the luminance difference feature map is not greater than the preset difference threshold Thresh, the first fusion weight corresponding to this pixel point is the target weight value b. If the pixel value diff(x) corresponding to this pixel point in the luminance difference feature map is greater than the preset difference threshold Thresh, since the preset weight coefficient S is less than 0, it can be determined according to the first fusion weight formula "S*(diff(x)-Thresh)+b" corresponding to "diff(x)>Thresh" that the first fusion weight α(x) linearly decreases with the increase of the pixel value diff(x) on the basis of the target weight value b.

[0097] For regions that strongly reflect infrared light and have a large reflection difference between the infrared image and the visible light image, if the luminance in the infrared image is directly used for this region during image fusion, it is easier to cause color cast and color distortion problems. Therefore, during image fusion, the use of the luminance in the infrared image for this region can be appropriately reduced.

[0098] Figure 8 A schematic diagram of differential feature mapping provided by an embodiment of the present invention is as Figure 8 shown. When the luminance difference feature diff(x) between the pixel points x at the corresponding positions in the target infrared image and the target visible light image is less than or equal to the preset difference threshold (Thresh), the first fusion weight α corresponding to the pixel point at this position in the fused image obtained by fusing the target visible light image and the target infrared image can be the highest, which can be 1; when the luminance difference feature between the pixel points at the corresponding positions in the target infrared image and the target visible light image is greater than Thresh, the first fusion weight α corresponding to the pixel point at this position in the fused image obtained by fusing the target visible light image and the target infrared image decreases with the increase of the luminance difference feature, as Figure 8 shown, and the decreasing rate can be represented by the preset weight coefficient Slope.

[0099] In a possible implementation manner, the step of performing image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image may include the following steps C1-C2:

[0100] Step C1, calculate the difference in pixel values of the luminance channels between the corresponding pixel points at the corresponding positions in the target infrared image and the infrared luminance feature image to obtain an infrared image detail texture layer.

[0101] Specifically, as shown in the following formula, the difference in pixel values of the luminance channels between the corresponding pixels in the target infrared image and the infrared luminance feature image can be calculated. Then, the difference in pixel values is multiplied by a preset strength parameter to obtain the infrared image detail texture layer:

[0102]

[0103] where nir_det_str is the preset strength parameter, which is a parameter used to control the fusion detail superposition and can be specifically set according to the actual application scenario. Y nir (x) is the pixel value of the luminance channel of pixel point x in the target infrared image. represents low-pass filtering the luminance channel of the target infrared image to obtain the infrared luminance feature image. K(x) is the low-pass edge-preserving filter operator, and D(x) is the pixel value of the pixel at the corresponding position of the obtained infrared image detail texture layer and pixel point x.

[0104] Step C2: Perform image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain the fused image.

[0105] Specifically, the performing image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain the fused image may include:

[0106] Sum the product of the Y value of each pixel point in the target infrared image and the first fusion weight, the product of the Y value of the pixel point at the same position in the target visible light image and the second fusion weight, and the pixel value of the pixel at the same position in the infrared image detail texture layer. The obtained sum value is used as the Y value of the pixel at the fused position. In addition, based on the ratio of the Y value of the pixel at the fused position to the Y value of the pixel at the same position in the target visible light image as the luminance gain, and multiply the U and V values of the pixel at the same position in the target visible light image by the luminance gain respectively to determine the U and V values of the pixel at the fused position, thereby obtaining the fused image.

[0107] In the embodiments of the present invention, in the YUV data of the target visible light image and the target infrared light image, "Y" represents brightness, and "U" and "V" represent chrominance and concentration.

[0108] In an embodiment of the present invention, calculating the second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight may include: using the difference obtained by subtracting the first fusion weight corresponding to each pixel point of the target visible light image from 1 as the second fusion weight corresponding to this pixel point.

[0109] For example, the Y channels of the target visible light image and the target infrared image can be subjected to brightness weighted fusion, and then superimposed with the infrared image detail texture layer D to obtain the Y channel of the fused image:

[0110] Y fus (x) = (1 - α(x))Y vis (x) + α(x)Y nir (x) + D(x)

[0111] Wherein, Y fus (x) is the Y value of the pixel point x in the fused image, Y vis (x) is the Y value of the pixel point at the same position as the pixel point x in the target visible light image, (1 - α(x)) is the second fusion weight corresponding to the pixel point at the same position as the pixel point x in the target visible light image, Y nir (x) is the Y value of the pixel point at the same position as the pixel point x in the target infrared image, α(x) is the first fusion weight corresponding to the pixel point at the same position as the pixel point x in the target infrared image, and D(x) is the pixel value of the pixel point at the same position as the pixel point x in the infrared image detail texture layer.

[0112] Then, the ratio of the Y value of the pixel point at this position in the fused image to the Y value of the pixel point at this position in the target visible light image can be used as the brightness gain: Y fus (x) / Y vis (x).

[0113] Then, the following formula can be used to respectively determine the U and V values of the pixel point at this position in the fused image by multiplying the U and V values of the pixel point at this position in the target visible light image by the brightness gain, so as to obtain the fused image:

[0114]

[0115] Wherein, U fus (x) is the U value of the pixel point x in the fused image, V fus (x) is the V value of the pixel point x in the fused image.

[0116] By using the method provided in the embodiment of the present invention, each pixel at the corresponding position of the target visible light image and the target infrared image can be fused according to the first fusion weight and the second fusion weight, so that the brightness from the target infrared image and the brightness from the target visible light image in each pixel of the obtained fused image reach the optimal ratio. For the area with large differences in infrared reflection characteristics, the value of the first fusion weight of the corresponding pixel of each pixel in the target infrared image in this area can be reduced, reducing the brightness of the image in this area from the infrared image in the fused image, thereby avoiding color distortion of the image in this area in the fused image. For the area with small differences in infrared reflection characteristics, the value of the first fusion weight of the corresponding pixel of each pixel in the target infrared image in this area can be increased, increasing the brightness of the image in this area from the infrared image in the fused image and enhancing the brightness of the image in this area in the fused image. That is, the method provided in the embodiment of the present invention can obtain a fused image with accurate colors on the premise of ensuring the quality of the fused image. The fused image has both the color information of visible light and the advantage of high signal-to-noise ratio of the infrared light image.

[0117] In the embodiment of the present invention, for the fused image obtained by performing image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight, if there is still a problem of color cast in some areas of the fused image, the embodiment of the present invention can further perform color cast correction on the fused image.

[0118] In a possible implementation manner, Figure 9 is a flowchart for performing color cast correction on the fused image provided by the embodiment of the present invention. As Figure 9 shown, after performing image fusion on the target visible light image and the target infrared image to obtain the fused image, it further includes:

[0119] S901, determine the preset color corresponding to each pixel according to the R, G, and B values of each pixel of the fused image.

[0120] The color of each pixel in the image is determined by the values of the three primary colors R (Red), G (Green), and B (Blue). The range of the R, G, and B values is 0 - 255, and different values of R, G, and B form different colors.

[0121] In this embodiment, for each pixel of the fused image, the preset color corresponding to the pixel can be determined according to the R, G, and B values of the pixel. Specifically, the R, G, and B values of the pixel can be sorted according to their magnitudes, and then the first difference between the maximum value and the middle value, and the second difference between the middle value and the minimum value can be calculated. By comparing the magnitudes of the first difference and the second difference, if the first difference is greater than the second difference, it indicates that the color of the pixel is closer to the color corresponding to the maximum value among the R, G, and B values, and thus the color of the pixel can be determined as the color corresponding to the maximum value among the R, G, and B values; if the first difference is less than the second difference, it indicates that the color of the pixel is closer to the color corresponding to the minimum value among the R, G, and B values, and thus the color of the pixel can be determined as the color corresponding to the minimum value among the R, G, and B values; if the first difference is equal to the second difference, the color of the pixel can be determined as the color corresponding to the minimum value among the R, G, and B values or the color corresponding to the maximum value among the R, G, and B values.

[0122] If the R value of a pixel is the largest among the R, G, and B values of the pixel, the color corresponding to the R value is red; if the R value of a pixel is the smallest among the R, G, and B values of the pixel, the color corresponding to the R value is cyan; if the G value of a pixel is the largest among the R, G, and B values of the pixel, the color corresponding to the G value is green; if the G value of a pixel is the smallest among the R, G, and B values of the pixel, the color corresponding to the G value is magenta; if the B value of a pixel is the largest among the R, G, and B values of the pixel, the color corresponding to the B value is blue; if the B value of a pixel is the smallest among the R, G, and B values of the pixel, the color corresponding to the B value is yellow.

[0123] For example, if the R, G, and B values of pixel x1 are 100, 80, and 20 respectively. The R, G, and B values of pixel x1 can be sorted according to their magnitudes as: R(100) > G(80) > B(20). The maximum value is the R value, the minimum value is the B value, and the middle value is the G value. Then, the first difference between the R value and the G value is calculated as 20, and the second difference between the G value and the B value is calculated as 60. It can be obtained that the first difference 20 is less than the second difference 60, indicating that the color of pixel x1 is closer to the color corresponding to the minimum value B. And when the B value is the smallest among the R, G, and B values of the pixel, the color corresponding to the B value is yellow, so the color of pixel x1 can be determined as yellow.

[0124] For another example, if the R, G, and B values of pixel x2 are 255, 20, and 10 respectively. The R, G, and B values of this pixel x2 can be sorted by size as: R(255) > G(20) > B(10). The maximum value is the R value, the minimum value is the B value, and the middle value is the G value. Then calculate the first difference between the R value and the G value as 235, and calculate the second difference between the G value and the B value as 10. It can be obtained that the first difference 235 is greater than the second difference 10, indicating that the color of this pixel x2 is closer to the color corresponding to the maximum value R. When the R value is the largest in the R, G, and B values of the pixel, the color corresponding to the R value is red, so it can be determined that the color of this pixel x2 is red.

[0125] For another example, if the R, G, and B values of pixel x3 are 10, 105, and 200 respectively. The R, G, and B values of this pixel x3 can be sorted by size as: B(200) > G(105) > R(10). The maximum value is the B value, the minimum value is the R value, and the middle value is the G value. Then calculate the first difference between the B value and the G value as 95, and calculate the second difference between the G value and the R value as 95 as well. It can be obtained that the first difference 95 is equal to the second difference 95, indicating that the color of this pixel x3 can be classified as the color corresponding to the B value with the largest sort in the R, G, and B values - blue, or it can be classified as the color corresponding to the R value with the smallest sort in the R, G, and B values - cyan (when the R value of the pixel is sorted smallest, the corresponding color is cyan).

[0126] S902. Based on the preset color corresponding to each pixel of the fused image and the R, G, and B values of each pixel, determine the color cast correction parameter corresponding to each pixel.

[0127] Specifically, in the embodiments of the present invention, the following formula can be used to determine the color cast correction parameter scale(x) corresponding to each pixel:

[0128]

[0129] where, I r (x), I g (x) and I b (x) are the R, G, and B values corresponding to pixel x in the fused image respectively, C represents the preset color corresponding to pixel x, scale(x) is the color cast correction parameter corresponding to pixel x, max represents the operation of taking the maximum value, med represents the operation of taking the middle value, and min represents the operation of taking the minimum value.

[0130] For example, if the R, G, and B values corresponding to the pixel point x1 in the fused image are 100, 80, and 20 respectively, and the preset color corresponding to the pixel point x1 has been determined to be yellow as described above, then the color cast correction parameter scale(x1) corresponding to the pixel point x1 can be obtained as scale(x1) = med(I r (x),I g (x),I b (x)) - min(I r (x),I g (x),I b (x)) = 80 - 20 = 60.

[0131] S903. Determine the brightness parameter corresponding to each pixel point of the fused image according to the brightness of the fused image and the brightness of the target visible light image.

[0132] Specifically, in this embodiment, the following formula can be used to determine the brightness parameter corresponding to each pixel point of the fused image:

[0133] k(x) = max(Y fus (x) - Y vis (x), 0)

[0134] where k(x) is the brightness parameter corresponding to the pixel point x, Y fus (x) is the Y value of the pixel point x in the fused image, and Y vis (x) is the Y value of the pixel point at the same position as the pixel point x in the target visible light image.

[0135] S904. Based on the color cast correction parameter corresponding to each pixel point of the fused image, the R, G, and B values of each pixel point, and the brightness parameter corresponding to each pixel point, correct the R, G, and B values of the pixel point to obtain a corrected fused image.

[0136] In a possible implementation manner, the step of correcting the R, G, and B values of the pixel point based on the color cast correction parameter corresponding to each pixel point of the fused image, the R, G, and B values of each pixel point, and the brightness parameter corresponding to each pixel point to obtain a corrected fused image may specifically include the following steps D1 - D2:

[0137] Step D1. For each pixel point of the fused image, calculate the correction value corresponding to each color channel of the pixel point according to the gray value of each color channel of the pixel point, the brightness parameter, the minimum value of the difference between the maximum gray value and the gray value of the channel, and the color cast correction parameter corresponding to the pixel point.

[0138] The color channels of the pixel point may include the R channel, the G channel, and the B channel.

[0139] Specifically, the product of the grayscale value, brightness parameter, and the minimum value of the difference between the maximum grayscale value and the grayscale value of each color channel of this pixel point and the color cast correction parameter corresponding to this pixel point can be used as the correction value corresponding to each color channel of this pixel point. Alternatively, it is also possible to first calculate the product of the grayscale value, brightness parameter, and the minimum value of the difference between the maximum grayscale value and the grayscale value of each color channel of this pixel point and the color cast correction parameter corresponding to this pixel point, and then use the value obtained by dividing the product by the maximum grayscale value (255) of each channel as the correction value corresponding to each color channel of this pixel point.

[0140] Step D2: Subtract the corresponding correction value from the grayscale value of each color channel of this pixel point to obtain the corrected fused image.

[0141] Specifically, in the embodiments of the present invention, the following formula can be used to correct the R, G, and B values of this pixel point based on the color cast correction parameter corresponding to each pixel point of the fused image, the R, G, and B values of each pixel point, and the brightness parameter corresponding to each pixel point, so as to obtain the corrected fused image:

[0142]

[0143] Among them, I’ r (x), I’ g (x), and I’ b (x) are the R, G, and B values corresponding to the pixel point x after correction respectively, I r (x), I g (x), and I b (x) are the R, G, and B values corresponding to the pixel point x before correction respectively, k(x) is the brightness parameter corresponding to the pixel point x, and scale(x) is the color cast correction parameter corresponding to the pixel point x.

[0144] Among them, the minimum value "min(255 - I r (x), I r (x), k(x))" of the grayscale value "I r (x)", brightness parameter "k(x)", and the difference between the maximum grayscale value and the grayscale value of this channel "255 - I r (x)" of the R channel of the pixel point x, and the product of the color cast correction parameter scale(x) corresponding to this pixel point x divided by the maximum grayscale value 255, that is, "min(255 - I r (x), I r (x), k(x)) * scale(x) / 255", is the correction value corresponding to the R channel of the pixel point x; in the same way, it can be obtained that: "min(255 - Ig (x), I g (x), k(x)) * scale(x) / 255” is the correction value corresponding to the G channel of pixel x, and “min(255 - I b (x), I b (x), k(x)) * scale(x) / 255”, is the correction value corresponding to the B channel of pixel x.

[0145] Subtracting the corresponding correction value from the gray value of each color channel of pixel x can obtain the corrected gray value I’ corresponding to each channel r (x), I’ g (x) and I’ b (x). After the corrected gray value corresponding to each channel of each pixel of the fused image is obtained, the corrected fused image can be obtained.

[0146] Figure 10 This is a schematic diagram of the corrected fused image provided by an embodiment of the present invention. As Figure 10 shown, by using the image fusion method provided by the embodiment of the present invention, the low-illumination target visible light image 1001 and the target infrared image 1002 are fused, and the fused image is subjected to color cast correction to obtain the corrected fused image 1003. It can be seen that the corrected fused image 1003 and the target infrared image 1002 are very similar in brightness and style, that is, the corrected fused image 1003 also makes full use of the brightness information of the target infrared image 1002, and through the color cast correction operation, the corrected fused image 1003 has the characteristics of being bright, clear and not color cast.

[0147] By using the method provided by the embodiment of the present invention to perform color cast correction on the fused image and further perform color restoration on the fused image, the finally obtained corrected fused image can largely avoid the problem of color distortion. Moreover, the brightness of the corrected fused image is very similar to the brightness of the target infrared image, making full use of the brightness information of the target infrared image, and having the characteristics of being bright, clear and not color cast.

[0148] Corresponding to the above image fusion method, an embodiment of the present invention also provides an image fusion device, which will be introduced below. Figure 11 This is a schematic structural diagram of the image fusion device provided by an embodiment of the present invention. As Figure 11 shown, the device includes:

[0149] An image acquisition module 1101, configured to acquire a target visible light image and a target infrared image;

[0150] The differential feature determination module 1102 is configured to determine a luminance differential feature map between the target visible light image and the target infrared image according to the luminance of the target visible light image and the luminance of the target infrared image; wherein, the luminance differential feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image;

[0151] The first weight determination module 1103 is configured to determine a first fusion weight corresponding to each pixel point of the target infrared image based on the luminance differential feature map and a preset correspondence between the differential feature and the fusion weight; wherein, the correspondence is an inverse correlation relationship between the differential feature and the fusion weight;

[0152] The second weight determination module 1104 is configured to calculate a second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight;

[0153] The image fusion module 1105 is configured to perform image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image.

[0154] It can be seen that by using the device provided in the embodiment of the present invention, the first fusion weight corresponding to each pixel point of the target infrared image can be determined through the luminance differential feature map and the preset correspondence between the differential feature and the fusion weight, and the second fusion weight corresponding to each pixel point of the target visible light image can be calculated according to the first fusion weight. Each pixel point at the corresponding position of the target visible light image and the target infrared image can be fused according to the first fusion weight and the second fusion weight, so that the luminance from the target infrared image and the luminance from the target visible light image in each pixel point of the obtained fused image reach the best ratio. For regions with large infrared reflection characteristic differences, the value of the first fusion weight corresponding to the pixel points of the target infrared image in this region can be reduced, and the luminance of the image in this region from the infrared image in the fused image can be reduced, thereby avoiding color distortion of the image in this region in the fused image. For regions with small infrared reflection characteristic differences, the value of the first fusion weight corresponding to the pixel points of the target infrared image in this region can be increased, and the luminance of the image in this region from the infrared image in the fused image can be increased, enhancing the luminance of the image in this region in the fused image. That is, the device provided in the embodiment of the present invention can obtain a fused image with accurate colors on the premise of ensuring the quality of the fused image. The fused image not only has the color information of visible light but also has the advantage of high signal-to-noise ratio of the infrared light image. Therefore, by using the device provided in the embodiment of the present invention, a clear color image can be obtained in a low-light scene.

[0155] Optionally, as Figure 12As shown, the device further includes:

[0156] A color deviation correction module 1201, configured to determine a preset color corresponding to each pixel point according to the R, G, and B values of each pixel point of the fused image; determine a color deviation correction parameter corresponding to each pixel point based on the preset color corresponding to each pixel point of the fused image and the R, G, and B values of each pixel point; determine a brightness parameter corresponding to each pixel point of the fused image according to the brightness of the fused image and the brightness of the target visible light image; and correct the R, G, and B values of each pixel point based on the color deviation correction parameter corresponding to each pixel point of the fused image, the R, G, and B values of each pixel point, and the brightness parameter corresponding to each pixel point, to obtain a corrected fused image.

[0157] Optionally, the color deviation correction module 1201 is specifically configured to, for each pixel point of the fused image, calculate a correction value corresponding to each color channel of the pixel point according to the gray value of each color channel of the pixel point, the brightness parameter, and the minimum value among the difference between the maximum gray value and the gray value of the channel and the color deviation correction parameter corresponding to the pixel point; and subtract the correction value corresponding to each color channel of the pixel point from the gray value of each color channel of the pixel point, to obtain a corrected fused image.

[0158] Optionally, the differential feature determination module 1102 includes:

[0159] A low-pass filtering sub-module (not shown in the figure), configured to perform low-pass filtering on the brightness channels of the target visible light image and the target infrared image respectively, to obtain a visible light brightness feature image and an infrared brightness feature image;

[0160] A differential feature calculation sub-module (not shown in the figure), configured to calculate the absolute value of the brightness difference between the pixel points at corresponding positions in the visible light brightness feature image and the infrared brightness feature image, to obtain an image with the absolute value as the pixel value as a brightness differential feature map.

[0161] Optionally, the image fusion module 1105 includes:

[0162] A texture layer calculation sub-module (not shown in the figure), configured to calculate the difference between the pixel values of the brightness channels of the corresponding pixel points in the target infrared image and the infrared brightness feature image, to obtain an infrared image detail texture layer;

[0163] A fusion sub-module (not shown in the figure), configured to perform image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight, to obtain a fused image.

[0164] Optionally, the fusion sub-module is specifically configured to sum the product of the Y value of each pixel point of the target infrared image and the first fusion weight, the product of the Y value of the pixel point at the same position as this pixel point in the target visible light image and the second fusion weight, and the pixel value of the pixel point at the same position as this pixel point in the infrared image detail texture layer, and use the obtained sum value as the Y value of the pixel point at this position after fusion. And, based on the ratio of the Y value of the pixel point at this position after fusion to the Y value of the pixel point at this position in the target visible light image as the brightness gain, and determine the products of the U and V values of the pixel point at this position in the target visible light image and the brightness gain respectively as the U and V values of the pixel point at this position after fusion, so as to obtain the fused image.

[0165] Optionally, the first weight determination module 1103 is specifically configured to, for each pixel point of the target infrared image, if the pixel value corresponding to this pixel point in the brightness difference feature map is not greater than a preset difference threshold, determine the first fusion weight corresponding to this pixel point as the target weight value; if the pixel value corresponding to this pixel point in the brightness difference feature map is greater than the preset difference threshold, determine the first fusion weight corresponding to this pixel point based on the pixel value and a preset weight function; wherein, the preset weight function indicates that the first fusion weight decreases linearly with the increase of the pixel value on the basis of the target weight value.

[0166] It can be seen that by using the device provided in the embodiment of the present invention, color cast correction is performed on the fused image, and color restoration is further performed on the fused image. The finally obtained corrected fused image can largely avoid the problem of color distortion. Moreover, the brightness of the corrected fused image is very similar to the brightness of the target infrared image, making full use of the brightness information of the target infrared image, and having the characteristics of being bright, clear and not color cast.

[0167] The embodiment of the present invention also provides an electronic device, as Figure 13 shown, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304. Among them, the processor 1301, the communication interface 1302, and the memory 1303 complete mutual communication through the communication bus 1304.

[0168] The memory 1303 is used to store a computer program;

[0169] The processor 1301 is configured to implement the steps of any of the above-mentioned image fusion methods when executing the program stored on the memory 1303.

[0170] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0171] The communication interface is used for communication between the above electronic device and other devices.

[0172] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0173] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0174] In another embodiment provided by the present invention, a computer-readable storage medium is also provided. A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above image fusion methods are implemented.

[0175] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When it runs on a computer, it causes the computer to execute any of the image fusion methods in the above embodiments.

[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0177] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0178] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0179] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. An image fusion method, characterized in that, Including: Obtain a target visible light image and a target infrared image; Determine a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image; wherein, the brightness difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image; Based on the brightness difference feature map and a preset correspondence between the difference feature and the fusion weight, determine a first fusion weight corresponding to each pixel point of the target infrared image; wherein, the correspondence is that the difference feature and the fusion weight are inversely correlated; Calculate a second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight; Based on the first fusion weight and the second fusion weight, perform image fusion on the target visible light image and the target infrared image to obtain a fused image; The determining a brightness difference feature map between the target visible light image and the target infrared image according to the brightness of the target visible light image and the brightness of the target infrared image includes: Perform low-pass filtering on the brightness channels of the target visible light image and the target infrared image respectively to obtain a visible light brightness feature image and an infrared brightness feature image; Calculate the absolute value of the brightness difference between the pixel points at the corresponding positions in the visible light brightness feature image and the infrared brightness feature image, and obtain an image with the absolute value as the pixel value as the brightness difference feature map.

2. The method according to claim 1, wherein After performing image fusion on the target visible light image and the target infrared image to obtain a fused image, it further includes: Determine a preset color corresponding to each pixel point according to the R, G, B values of each pixel point of the fused image; Based on the preset color corresponding to each pixel point of the fused image and the R, G, B values of each pixel point, determine a color cast correction parameter corresponding to each pixel point; Determine a brightness parameter corresponding to each pixel point of the fused image according to the brightness of the fused image and the brightness of the target visible light image; Based on the color cast correction parameter corresponding to each pixel point of the fused image, the R, G, B values of each pixel point, and the brightness parameter corresponding to each pixel point, correct the R, G, B values of the pixel point to obtain a corrected fused image.

3. The method according to claim 2, wherein The correcting the R, G, B values of the pixel point based on the color cast correction parameter corresponding to each pixel point of the fused image, the R, G, B values of each pixel point, and the brightness parameter corresponding to each pixel point to obtain a corrected fused image includes: For each pixel point of the fused image, calculate a correction value corresponding to each color channel of the pixel point according to the gray value of each color channel of the pixel point, the brightness parameter, the minimum value of the difference between the maximum gray value and the gray value of the channel, and the color cast correction parameter corresponding to the pixel point; Subtract the correction value corresponding to each color channel of the pixel point from the gray value of each color channel of the pixel point to obtain a corrected fused image.

4. The method according to claim 1, wherein Performing image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image, includes: Calculating the difference in pixel values of the luminance channels between corresponding pixels in the target infrared image and the infrared luminance feature image to obtain an infrared image detail texture layer; Performing image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain a fused image.

5. The method according to claim 4, wherein The performing image fusion on the target visible light image and the target infrared image according to the infrared image detail texture layer, the first fusion weight, and the second fusion weight to obtain a fused image, includes: Summing the product of the Y value of each pixel in the target infrared image and the first fusion weight, the product of the Y value of the pixel at the same position as this pixel in the target visible light image and the second fusion weight, and the pixel value of the pixel at the same position as this pixel in the infrared image detail texture layer, and using the obtained sum value as the Y value of the pixel at this position in the fused image. Also, based on the ratio of the Y value of the pixel at this position in the fused image to the Y value of the pixel at this position in the target visible light image as the luminance gain, and respectively determining the product of the U and V values of the pixel at this position in the target visible light image and the luminance gain as the U and V values of the pixel at this position in the fused image to obtain a fused image.

6. The method according to claim 1, wherein The determining the first fusion weight corresponding to each pixel of the target infrared image based on the luminance difference feature map and the preset correspondence between the difference feature and the fusion weight, includes: For each pixel of the target infrared image, if the pixel value corresponding to this pixel in the luminance difference feature map is not greater than the preset difference threshold, determining the first fusion weight corresponding to this pixel as the target weight value; If the pixel value corresponding to this pixel in the luminance difference feature map is greater than the preset difference threshold, determining the first fusion weight corresponding to this pixel based on the pixel value and the preset weight function; wherein, the preset weight function indicates that the first fusion weight linearly decreases with the increase of the pixel value on the basis of the target weight value.

7. An image fusion device, characterized in that, Includes: An image acquisition module, configured to acquire a target visible light image and a target infrared image; A difference feature determination module, configured to determine a luminance difference feature map between the target visible light image and the target infrared image according to the luminance of the target visible light image and the luminance of the target infrared image; wherein, the luminance difference feature map is used to characterize the reflection characteristic difference between the target visible light image and the target infrared image; A first weight determination module, configured to determine the first fusion weight corresponding to each pixel of the target infrared image based on the luminance difference feature map and the preset correspondence between the difference feature and the fusion weight; wherein, the correspondence is an inverse correlation relationship between the difference feature and the fusion weight; The second weight determination module is configured to calculate the second fusion weight corresponding to each pixel point of the target visible light image according to the first fusion weight; The image fusion module is configured to perform image fusion on the target visible light image and the target infrared image based on the first fusion weight and the second fusion weight to obtain a fused image; The differential feature determination module includes: The low-pass filtering sub-module is configured to perform low-pass filtering on the luminance channels of the target visible light image and the target infrared image respectively to obtain a visible light luminance feature image and an infrared luminance feature image; The differential feature calculation sub-module is configured to calculate the absolute value of the luminance difference between the pixel points at corresponding positions in the visible light luminance feature image and the infrared luminance feature image, and obtain an image with the absolute value as the pixel value as the luminance differential feature map.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store computer programs; When the processor is configured to execute the program stored on the memory, it implements the method steps described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-6.

Images