A method and device for fusing visible light image and infrared image

Abstract

The present application discloses a method for fusing a visible light image and an infrared image. The method comprises obtaining a visible light brightness image based on a visible light image and obtaining an infrared brightness image based on an infrared image; using the visible light brightness image and the infrared brightness image, determining the coordinate information of the first pixel in the target area of ​​the visible light brightness image and the coordinate information of the second pixel in the non-target area; fusing the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image; adjusting the brightness component of the pixel in the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast to obtain a second image; for each pixel in the second image, fusing the brightness component of the pixel with the chromaticity component of the pixel in the visible light image having the same coordinates as the pixel to obtain a fused image. The present application solves the color cast problem in visual effects, so that the fused image has a strong color preservation ability.

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular to a method for fusing a visible light image with an infrared image. Background Art

[0002] At present, there are many image fusion solutions in the field of image processing. Most of them use a single camera spectroscopic structure or a dual camera structure to obtain information from different spectral bands, mainly visible light images and non-visible light images, and then combine the advantages of both to perform image fusion to obtain better image effects. Although the fusion algorithms of each solution are different, the ultimate goal is to improve the image effect under low illumination, which can be reflected in multiple aspects such as signal-to-noise ratio, color, and detail contours.

[0003] Given that objects have different reflection and absorption characteristics for different spectra, the brightness and texture of the same object in visible light images and infrared images may vary greatly, especially in low-light environments. In the field of surveillance, under reasonable fill-light conditions, the signal-to-noise ratio and texture of infrared images are often better, but infrared images do not have real color information, and too much infrared information can easily lead to color distortion, unnatural texture, and other phenomena. Summary of the invention

[0004] The present invention provides a method for fusing a visible light image with an infrared image to improve the authenticity of colors in a fused image.

[0005] The present invention provides a method for fusing a visible light image with an infrared image, which is implemented as follows:

[0006] Acquire a visible light brightness image based on the visible light image, and acquire an infrared brightness image based on the infrared image;

[0007] Determine coordinate information of a first pixel point in a target area of ​​the visible light brightness image and coordinate information of a second pixel point in a non-target area by using the visible light brightness image and the infrared brightness image;

[0008] Fusing the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image;

[0009] Determine a target area in the first image according to the coordinate information of the first pixel point, and determine a non-target area in the first image according to the coordinate information of the second pixel point;

[0010] The brightness components of the pixels in the first image are adjusted so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image; wherein the first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image; and the second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image;

[0011] For each pixel in the second image, the brightness component of the pixel is fused with the chrominance component of a pixel in the visible light image having the same coordinates as the pixel to obtain a fused image.

[0012] Preferably, the method of using the visible light brightness image and the infrared brightness image to determine the coordinate information of a first pixel in the target area of ​​the visible light brightness image and the coordinate information of a second pixel in the non-target area includes:

[0013] Calculate the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image to obtain the brightness residual of each pixel;

[0014] The brightness residual of each pixel is weighted by using a first weight calculated based on the brightness component of each pixel in the visible light image and a second weight calculated based on the brightness component of the pixel in the infrared image, so that the brightness residual of each pixel is corrected.

[0015] Based on the corrected brightness residual of each pixel point, the coordinate information of the pixel point whose brightness residual is greater than or equal to the preset brightness threshold is used as the coordinate information of the first pixel point in the target area of ​​the visible light brightness image, and the coordinate information of the pixel point whose brightness residual is less than the brightness threshold is used as the coordinate information of the second pixel point in the non-target area of ​​the visible light brightness image.

[0016] Preferably, the step of calculating the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image comprises:

[0017] For each pixel in the visible light brightness image and / or infrared brightness image:

[0018] If the first brightness component of the pixel in the infrared brightness image is greater than or equal to the second brightness component of the pixel in the visible light brightness image, then the difference between the first brightness component and the second brightness component is calculated to obtain the brightness residual of the pixel;

[0019] If the first brightness component of the pixel in the infrared brightness image is less than the second brightness component of the pixel in the visible light brightness image, the brightness residual of the pixel is set to 0.

[0020] Preferably, the weighting of the brightness residual of each pixel point by using a first weight value calculated based on the brightness component of each pixel point in the visible light image and a second weight value calculated based on the brightness component of the pixel point in the infrared image comprises:

[0021] For each pixel's brightness residual:

[0022] The brightness residual of the pixel point is filtered by using a filter function, and the filtered brightness residual is multiplied by the first weight value and the second weight value of the pixel point to obtain the corrected brightness residual of the pixel point;

[0023] or,

[0024] The brightness residual of the pixel point is multiplied by the first weight of the pixel point and the second weight value to obtain the corrected brightness residual of the pixel point; wherein,

[0025] The first weight is determined as follows:

[0026] When the brightness component of the pixel in the visible light brightness image is less than or equal to a set first threshold, the first weight value is a preset upper limit of the weight value;

[0027] When the brightness component of the pixel in the visible light brightness image is greater than a set first threshold, the first weight value is between a preset weight lower limit and a preset weight upper limit.

[0028] The second weight is determined as follows:

[0029] When the brightness component of the pixel in the infrared brightness image is less than or equal to the set second threshold, the second weight value is less than the preset upper limit of the weight;

[0030] When the brightness component of the pixel in the infrared brightness image is greater than or equal to the second threshold and less than the third threshold, the second weight is set to a preset upper weight limit;

[0031] When the brightness of the pixel in the infrared brightness image is greater than or equal to the third threshold, the second weight is between a preset weight lower limit and a preset weight upper limit;

[0032] The first threshold is determined according to the upper limit of the brightness component of the visible light brightness image;

[0033] The second threshold and the third threshold are determined according to the upper limit of the brightness component of the infrared brightness image.

[0034] Preferably, when the brightness component of the pixel in the visible light brightness image is greater than a set first threshold, the value of the first weight value is: the ratio of the difference between the upper limit of the brightness component of the visible light brightness image and the brightness component of the pixel, to the difference between the upper limit of the brightness component of the visible light brightness image and the first threshold;

[0035] When the brightness component of the pixel in the infrared brightness image is less than or equal to the set second threshold, the value of the second weight is: the ratio of the brightness component of the pixel to the second threshold,

[0036] When the brightness of the pixel in the infrared brightness image is greater than or equal to the third threshold, the value of the second weight is: the ratio of the difference between the upper limit of the brightness component of the infrared brightness image and the brightness component of the pixel, and the difference between the upper limit of the brightness component of the infrared brightness image and the third threshold.

[0037] Preferably, the adjusting the brightness component of the pixels of the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, and obtaining the second image includes:

[0038] Calculate the first brightness contrast,

[0039] Calculating a third brightness mean of a third pixel in the target area of ​​the first image and a fourth brightness mean of a fourth pixel in the non-target area of ​​the first image;

[0040] The product of the first brightness contrast and the fourth brightness mean is calculated to obtain an expected brightness mean value of the third pixel point in the target area of ​​the first image so that the first brightness contrast is equal to the second brightness contrast.

[0041] For each pixel in the first image, weight the brightness component of the pixel in the first image by the third weight of the pixel to obtain a second image;

[0042] in,

[0043] The third weight is related to a brightness adjustment ratio, and the brightness adjustment ratio is determined according to the expected brightness mean value and the third brightness mean value.

[0044] Preferably, the calculating the first brightness contrast comprises:

[0045] Obtaining a first brightness contrast ratio according to a first brightness mean value of a first pixel point in a target area of ​​the visible light brightness image and a second brightness mean value of a second pixel point in a non-target area;

[0046] The step of weighting, for each pixel in the first image, the brightness component of the pixel in the first image by the third weight of the pixel to obtain the second image comprises:

[0047] Calculate the ratio of the third brightness average to the expected brightness average to obtain the brightness adjustment ratio.

[0048] According to the brightness adjustment ratio and the corrected brightness residual of each pixel, the third weight of each pixel is calculated.

[0049] For each pixel in the first image, the third weight of the pixel is multiplied by the brightness component of each pixel in the first image.

[0050] All pixels with weighted brightness components form a second image.

[0051] Preferably, the third weight of each pixel is calculated according to the brightness adjustment ratio and the corrected brightness residual of each pixel, including:

[0052] Calculate the product of the brightness adjustment ratio and the brightness residual after correction of the pixel point to obtain the product result,

[0053] The ratio of the multiplication result to the corrected upper limit of the brightness residual value is calculated to obtain a third weight of the pixel.

[0054] The present invention also provides a device for fusing a visible light image and an infrared image, the device comprising:

[0055] A brightness image acquisition module, used to acquire a visible light brightness image based on the visible light image, and to acquire an infrared brightness image based on the infrared image;

[0056] An area determination module, used to determine the coordinate information of a first pixel point in a target area of ​​the visible light brightness image and the coordinate information of a second pixel point in a non-target area by using the visible light brightness image and the infrared brightness image;

[0057] A first fusion module is used to fuse the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image;

[0058] An adjustment module is used to adjust the brightness components of the pixels in the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image; wherein the first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image; and the second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image;

[0059] The second fusion module is used to fuse, for each pixel in the second image, the brightness component of the pixel with the chrominance component of the pixel in the visible light image having the same coordinates as the pixel, to obtain a fused image.

[0060] The present invention provides a computer-readable storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the method for fusing a visible light image with an infrared image are implemented.

[0061] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for fusing a visible light image with an infrared image are implemented.

[0062] The present application determines, based on a visible light brightness image of a visible light image to be fused and an infrared brightness image of an infrared image to be fused, a target area where the brightness of the visible light image is different from that of the infrared image, and a non-target area where the brightness of the visible light image is the same as that of the infrared image, and adjusts the brightness information of the target area in the fused brightness image in a manner that makes the expected contrast (second brightness contrast) of the target area in the fused brightness image equal to the first brightness contrast of the target area in the visible light brightness image, thereby keeping the local brightness contrast unchanged before and after the fusion, which not only ensures that the fused image has a good signal-to-noise ratio, details and contours, but also improves the authenticity of the color, solves the color cast problem in visual effects, reduces the simultaneous contrast effect, and makes the fused image have a strong color preservation ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic representation of the simultaneous contrast effect.

[0064] Figure 2 The figure is a flow chart of the method for fusing visible light image and infrared image in the present application.

[0065] Figure 3 A flow chart of a method for fusing a visible light image and an infrared image according to an embodiment of the present application.

[0066] Figure 4 A schematic diagram of a device for fusing visible light images and infrared images according to an embodiment of the present application.

[0067] Figure 5 Another schematic diagram of the device for fusing visible light images and infrared images according to an embodiment of the present application.

[0068] Figure 6 It is a schematic diagram of the logical relationship between various physical quantities and images in the embodiments of the present application. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0070] The applicant noted that since the human eye has a strong adaptability to brightness, it is difficult to accurately identify the absolute brightness of each pixel on the image. Even if the brightness of some areas in the fused image is consistent with the brightness of the visible light image, the subjective brightness perceived by the human eye is different due to the different background brightness, which leads to differences in visual color performance, especially for the three colors of black, white and gray. Figure 1 As shown in the figure, the brightness of the small squares in the middle is the same, but under different brightness backgrounds, the visual effects of the human eye are obviously different. This effect is called the simultaneous contrast effect. Therefore, while improving the signal-to-noise ratio and details, ensuring the authenticity of the color and the naturalness of the overall picture is one of the main difficulties of the image fusion algorithm.

[0071] The applicant has found that when fusion is performed only based on weights, more infrared image brightness tends to be selected in certain areas to a certain extent, which changes the brightness contrast between the area and the background, thereby causing a color cast problem in visual effects.

[0072] This application combines the characteristics of human visual characteristics and adjusts the brightness contrast of the selected local area to maintain the color of the fused image. On the basis of ensuring that the fused image has a good signal-to-noise ratio, details and contours, the authenticity of the color is further restored.

[0073] See also Figure 2 As shown, Figure 2 The following is a flow chart of a method for fusing a visible light image and an infrared image of the present invention. The method includes, for the visible light image and the infrared image to be fused:

[0074] Step 201, respectively extracting brightness information of a visible light image and an infrared image to obtain a visible light brightness image and an infrared brightness image;

[0075] Step 202: based on the brightness information of the extracted visible light image and the brightness information of the infrared image, determine the target area where the brightness of the visible light image is different from that of the infrared image, and determine the non-target area where the brightness of the visible light image is not different from that of the infrared image.

[0076] Step 203, calculating a first brightness contrast ratio between the target area and the non-target area in the visible light image,

[0077] The first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image.

[0078] Step 204, performing brightness fusion on the visible light brightness image and the infrared brightness image to obtain a first image;

[0079] Step 205, adjusting the brightness information in the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image with adjusted brightness information;

[0080] The second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image;

[0081] Step 206: Fuse the brightness information in the second image with the chromaticity information in the visible light image to obtain a fused image.

[0082] There is no strict execution order between step 204 and steps 202 and 203, and they can be executed in parallel.

[0083] The present application determines the target area and non-target area based on the brightness information of the visible light image and the infrared image, and can accurately select the area with obvious brightness difference. By maintaining the brightness contrast of the local area, the brightness of the target area in the fused image (first image) is adjusted, and the color visual effect of the target area in the fused image is restored to a certain extent, thereby improving the visual color effect deviation caused by the change of brightness contrast.

[0084] To facilitate understanding of the present application, the following description is made by taking an image to be fused with an image bit width of 8 bits as an example. It should be understood that the present application is not limited to the above image bit width, and for other image bit widths, corresponding modifications can be made with reference to the present application.

[0085] See also Figure 3 As shown, Figure 3 The following is a flow chart of a method for fusing a visible light image and an infrared image according to an embodiment of the present application. For the visible light image and the infrared image to be fused, the method includes:

[0086] Step 301, respectively extracting brightness information of a visible light image and an infrared image to obtain a visible light brightness image and an infrared brightness image;

[0087] The visible light brightness image has the same bit width as the visible light image to be fused, and the infrared brightness image has the same bit width as the visible light image to be fused.

[0088] In this step, since the visible light image and infrared image to be fused usually come from a visible light sensor and an infrared sensor respectively, in practical applications, the pixel size and image distortion of the visible light image and the infrared image may be different. Therefore, preferably, the pixels of the visible light image and the infrared image are matched so that the spatial points corresponding to the pixels of the visible light image and the spatial points corresponding to the pixels of the infrared image are the same, so that a registered visible light image and a registered infrared image can be obtained. Preferably, the bit width of the registered visible light image is the same as the bit width of the registered infrared image.

[0089] For visible light images, if the image data format is RGB, first convert it to YUV format, and then extract the visible light component Y vis (brightness information); if it is YUV data, the visible light component Y and UV component (chromaticity information) can be directly separated to obtain a visible light brightness image.

[0090] For infrared images, only the visible light component (brightness information) Y nir Then, an infrared brightness image can be obtained.

[0091] Preferably, brightness information is extracted based on the registered infrared image and the registered visible light image, so that the bit width of the visible light brightness image is the same as the bit width of the infrared brightness image, and the pixels in the visible light brightness image match the pixels in the infrared brightness image, that is, a registered visible light brightness image and a registered infrared brightness image can be obtained.

[0092] Step 302: Perform a preliminary fusion of the visible light brightness image and the infrared brightness image to obtain a brightness fusion image Y m , for the convenience of description, it is referred to as the first image hereinafter.

[0093] In this step, the method for fusing the visible light brightness image and the infrared brightness image may adopt a multi-scale hierarchical fusion method, for example, a pyramid multi-scale decomposition fusion method, or may be fused based on a preset fusion weight, without limitation to a specific fusion method.

[0094] In the following steps 303 to 306, based on the brightness information of the extracted visible light image and the brightness information of the infrared image, a target area where there is a difference in brightness between the visible light image and the infrared image, and a non-target area where there is no difference in brightness between the visible light image and the infrared image are determined. That is, the coordinate information of the first pixel point in the target area of ​​the visible light brightness image and the coordinate information of the second pixel point in the non-target area are determined using the visible light brightness image and the infrared brightness image.

[0095] Step 303, based on the brightness information of the extracted visible light image and the brightness information of the infrared image, calculate the brightness residual.

[0096] For example, for each pixel in the visible light brightness image and / or infrared brightness image:

[0097] If the first brightness component Y of the pixel in the infrared brightness image nir (i, j) is greater than or equal to the second brightness component Y of the pixel in the visible light brightness image vis (i, j), then calculate the difference between the first brightness component and the second brightness component to obtain the brightness residual of the pixel;

[0098] Otherwise, the brightness residual of the pixel is set to a first value different from any brightness residual, for example, to 0;

[0099] Mathematically expressed as:

[0100]

[0101] Among them, (i, j) is the coordinate of the pixel point, RD is the brightness residual data, Y nir (i, j) is the first brightness component of pixel (i, j) in the infrared brightness image, Y vis (i, j) is the second brightness component of pixel point (i, j) in the visible light brightness image.

[0102] From the calculation of the brightness residual, it can be seen that the greater the difference between the infrared brightness and the visible light brightness, the greater the brightness intensity of the mask image.

[0103] The brightness residual data of all pixels constitute a brightness residual map.

[0104] Step 304, performing brightness correction on the obtained brightness residual data to obtain a mask image.

[0105] In this step, the brightness residual data can be filtered, and the brightness residual map can be brightness corrected based on the extracted visible light image brightness information and the extracted infrared image brightness information to obtain a mask map to avoid processing some overexposed or dark brightness residual data.

[0106] The filtering may be a Gaussian filter or a guided filter, etc. If a guided filter is used, the guided map is an infrared brightness map.

[0107] Preferably, the brightness correction may be, for each pixel in the brightness residual data:

[0108] The brightness residual of the pixel point is filtered by a filter function, and the filtered brightness residual is weighted by the first weight value and the second weight value of the pixel point to obtain the corrected brightness residual of the pixel point, that is, the brightness component of the pixel point in the mask image;

[0109] or,

[0110] Without filtering, the brightness residual is directly weighted by using the first weight and the second weight of the pixel to obtain the corrected brightness residual of the pixel.

[0111] in,

[0112] The first weight is determined as follows:

[0113] If the brightness component of the pixel in the visible light brightness image (i.e., the pixel value of the brightness image) is less than or equal to the first threshold, the first weight of the pixel is taken as the upper limit of the weight, for example, 1.

[0114] Otherwise, the first weight of the pixel is: the ratio of the difference between the upper limit of the brightness component value of the visible light image and the brightness component of the pixel, and the difference between the upper limit of the brightness component value of the visible light image and the first threshold; wherein the upper limit of the brightness component value is determined by the image bit width.

[0115] The second weight is determined as follows:

[0116] If the brightness component of the pixel in the infrared brightness image is less than the second threshold, the second weight value of the pixel is the ratio of the brightness component of the pixel to the second threshold.

[0117] If the brightness component of the pixel in the infrared brightness image is greater than or equal to the second threshold and less than the third threshold, the second weight of the pixel is taken as the upper limit of the weight, for example, 1;

[0118] If the brightness component of the pixel in the infrared brightness image is greater than or equal to the third threshold, the second weight value of the pixel is: the ratio of the difference between the upper limit of the brightness component of the infrared brightness image and the brightness component of the pixel, and the difference between the upper limit of the brightness component of the infrared brightness image and the third threshold.

[0119] The value ranges of the first threshold, the second threshold, and the third threshold are determined according to the upper limit of the brightness component value determined by the bit width of the image. Specifically, the value range of the first threshold is determined according to the upper limit of the brightness component value of the visible light brightness image, and the value ranges of the second threshold and the third threshold are determined according to the upper limit of the brightness component value of the infrared brightness image. For the registered visible light image and the registered infrared image, the bit widths of the two brightness images are the same, and the brightness component value range is [0, 2 b-1], where b is the bit width of the brightness image, and the upper limit of the brightness component is 2 b For example, if the bit width of the image to be fused is 8 bits, and the value range of the brightness component determined by the bit width is 0 to 255, then the threshold value range is within 0 to 255, that is, the threshold value range is greater than or equal to 0 and less than or equal to 255.

[0120] Brightness correction can be expressed mathematically as:

[0121] M(i, j) = W vis (Y vis (i, j))×W nir (Y nir (i, j))×f filter (RD(i, j))

[0122]

[0123]

[0124] Among them, W vis (Y vis (i, j)) is the first weight of the pixel (i, j) in the visible light brightness image, W nir (Y nir (i, j)) is the second weight of the pixel (i, j) in the infrared brightness image, and f filter is the filter function, Y vis-max is the upper limit of the brightness component of the visible light brightness image, according to 2 b1 -1 is determined, b1 is the bit width of the visible light brightness image, Y nir-max is the upper limit of the brightness component of the infrared brightness image, according to 2 b2 -1 is determined, b2 is the bit width of the infrared brightness image.

[0125] When the bit width of the image to be fused is 8 bits, the first weight and the second weight can be expressed as:

[0126]

[0127]

[0128] From the mathematical formula of the first weight and the second weight, we can see that:

[0129] When the brightness of the visible light image is less than or equal to the set first threshold, the first weight is taken as the upper limit of the weight, for example, 1. When the brightness of the visible light image is greater than or equal to the set first threshold, the first weight is taken between the upper limit of the weight and the lower limit of the weight, for example, between 0 and 1.

[0130] When the brightness of the infrared image is less than or equal to the set second threshold, the second weight is less than the upper limit of the weight, for example, it is 1. When the brightness of the infrared image is greater than or equal to the second threshold and less than the third threshold, the second weight is the upper limit of the weight, for example, it is 1. When the brightness of the infrared image is greater than or equal to the third threshold, the first weight is between the upper limit and the lower limit of the weight, for example, between 0 and 1.

[0131] Step 305, binarize the mask image.

[0132] In this step, for each pixel in the mask image:

[0133] If the brightness component of the pixel is greater than or equal to the set brightness threshold, the brightness component of the pixel takes the second value.

[0134] Otherwise, the brightness component of the pixel is taken as the third value;

[0135] wherein the second value is greater than the third value;

[0136] Preferably, the second value is 1 and the third value is 0.

[0137] The mathematical expression is:

[0138]

[0139] Among them, M represents the mask image, M b represents the mask image after binarization, T4 is the brightness threshold, and the value range is determined according to the bit width b of the mask image. Since the bit width of the mask image is the same as that of the brightness image, the value range of the fourth threshold is [0, 2 b -1].

[0140] Step 306, determining the area with the second value in the binary mask image as the target area,

[0141] Preferably, the area having the third value is determined as a non-target area;

[0142] Alternatively, an area with a third value within a certain neighborhood of the determined target area is determined as a non-target area, or an area within a certain neighborhood of the determined target area is determined as a non-target area, or an area in the image excluding the target area is determined as a non-target area.

[0143] Based on the determined target area and non-target area, the pixel position information in the target area and the pixel position information in the non-target area can be determined, which can be expressed as follows:

[0144] S obj ={(i, j)|Mb (i, j) = 1}

[0145] S bk ={(i, j)|M b (i, j) = 0}

[0146] Among them, S obj is the location information set of the target area, S bk It is the location information set of non-target areas.

[0147] According to the coordinate information of the pixel points in the position information set of the target area, the first pixel point with the same coordinate information in the visible light brightness image can be determined, and the area where the first pixel point is located is the target area. Similarly, according to the coordinate information of the pixel points in the position information set of the non-target area, the second pixel point with the same coordinate information in the visible light brightness image can be determined, and the area where the second pixel point is located is the non-target area.

[0148] Through steps 305 and 306, the coordinate information of the pixel points whose brightness residual is greater than or equal to the preset brightness threshold is used as the coordinate information of the first pixel point in the target area of ​​the visible light brightness image, and the coordinate information of the pixel points whose brightness residual is less than the brightness threshold is used as the coordinate information of the second pixel point in the non-target area of ​​the visible light brightness image.

[0149] Step 307, calculating a first brightness contrast ratio between the target area and the non-target area in the visible light brightness image according to the determined position information of the target area and the non-target area,

[0150] In this step, according to the target area and non-target area in the visible light brightness image, a target brightness set including a first pixel and a non-target brightness set including a second pixel are obtained, and the mean is calculated respectively to obtain the first brightness mean of the target area and the second brightness mean of the non-target area in the visible light brightness image. According to the first brightness mean and the second brightness mean, the local brightness contrast, i.e., the first brightness contrast, is calculated. For example, the ratio of the first brightness mean to the second brightness mean is used as the first brightness contrast, or the ratio can be made after taking the logarithm of the mean. The mean can be a summed average or a weighted average based on the brightness intensity of the mask image.

[0151] In this embodiment, the summation and averaging are adopted. It can be expressed as follows:

[0152]

[0153]

[0154] C1=L a1 / L b1

[0155] Among them, L a1 is the first brightness mean of the target area in the visible light image, L b1 is the second brightness mean of the non-target area in the visible light image, C1 is the first brightness contrast, N1 is the first number of pixels in the target area, and N2 is the second number of pixels in the non-target area.

[0156] In the following steps, the brightness information of the target area in the first image is adjusted so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast. Specifically, the second brightness contrast between the target area and the non-target area in the first image is determined, and the brightness information of the target area in the first image is adjusted according to the first brightness contrast and the second brightness contrast. The corresponding steps are as follows:

[0157] Step 308: Calculate a third average brightness value of the target area and a fourth average brightness value of the non-target area in the first image according to the determined position information of the target area and the non-target area.

[0158] In this step, the position information of the determined target area and non-target area is corresponded to the first image, and the fourth brightness mean of the non-target area including the fourth pixel point in the first image and the third brightness mean of the target area including the third pixel point are obtained, wherein the third pixel point has the same coordinate information as the first pixel point, and the fourth pixel point has the same coordinate information as the second pixel point.

[0159] Mathematically expressed as:

[0160]

[0161]

[0162] Among them, L a2 is the third brightness mean of the target area in the first image, L b2 is the fourth brightness mean of the non-target area of ​​the first image, N3 is the third number of pixels in the target area of ​​the first image, which is the same as the first number of pixels, and N4 is the fourth number of pixels in the non-target area of ​​the first image, which is the same as the second number of pixels.

[0163] Step 309 , calculating an expected average brightness value of the target area in the first image so that the first brightness contrast is equal to the second brightness contrast, where the second brightness contrast is the ratio of the expected average brightness value to the fourth average brightness value.

[0164] In this step, in order to maintain the visual color effect, it is necessary to make the local brightness contrast of the target area of ​​the first image consistent with the local brightness contrast of the corresponding target area on the visible light brightness map, that is, to make the first brightness contrast equal to the second brightness contrast.

[0165] Assuming that the second brightness contrast between the target area and the non-target area in the first image is C2, then C1=C2, so the expected brightness mean of the target area can be calculated, which is the product of the first brightness contrast and the fourth brightness mean.

[0166] The mathematical expression is:

[0167]

[0168] but:

[0169] Among them, L a3 is the expected average brightness of the target area in the first image.

[0170] In the following steps 310-311, the brightness information of the target area in the first image is adjusted according to the third brightness mean value and the expected brightness mean value.

[0171] Step 310: Calculate the brightness adjustment ratio according to the third brightness mean and the expected brightness mean, and calculate the third weight for brightness adjustment in combination with the mask image.

[0172] For example, for each pixel in the mask image:

[0173] Calculate: the product of the brightness adjustment ratio and the brightness component of the pixel, and the ratio of the upper limit of the brightness component of the mask image to obtain the third weight of the pixel.

[0174] Mathematically expressed as:

[0175] R=L a3 / L a2

[0176] W r (i, j) = R × M (i, j) / Y r-max

[0177] Among them, R is the brightness adjustment ratio, Y r-max is the upper limit of the brightness component of the mask image, that is, the upper limit of the corrected brightness residual value, which is given by (2 b3 -1) Determine, b3 is the bit width of the mask image, and all pixels can form a weight map for brightness adjustment.

[0178] Step 311, adjusting the brightness information in the first image according to the third weight, so that the brightness information of the target area in the first image is adjusted, and obtaining a second image;

[0179] In this step, the weight map is applied to the first image to adjust the brightness and obtain an adjusted second image.

[0180] For each pixel in the first image:

[0181] Calculate the weighted value of the brightness component of the pixel in the first image by adding the third weight of the pixel to obtain the weighted brightness component of the pixel,

[0182] Among them, all weighted pixel points form a second image.

[0183] Mathematically expressed as:

[0184] Y′m(i, j)=W r (i, j)×Y m (i, j)

[0185] Among them, Y′m(i, j) is the second image, Y m (i, j) is the first image.

[0186] Step 312: Fusing the second image with the chromaticity information of the visible light image to obtain a final fused image.

[0187] In this step, the brightness information of the second image and the U and V components in the visible light image are combined into a YUV format, and the YUV format is converted into an RGB format to obtain a final fused image.

[0188] In the above steps, steps 303 to 307, steps 302 and 308 have no strict order relationship and can be executed in parallel.

[0189] The final fused image of this embodiment has a good improvement in signal-to-noise ratio, detail contour, etc., and also has a more realistic color effect.

[0190] See also Figure 4 As shown, Figure 4 This is a schematic diagram of a fusion device for visible light images and infrared images in this application. The fusion device includes:

[0191] A brightness image acquisition module, used to acquire a visible light brightness image based on the visible light image, and to acquire an infrared brightness image based on the infrared image;

[0192] An area determination module, used to determine the coordinate information of a first pixel point in a target area of ​​the visible light brightness image and the coordinate information of a second pixel point in a non-target area by using the visible light brightness image and the infrared brightness image;

[0193] A first fusion module is used to fuse the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image;

[0194] An adjustment module is used to adjust the brightness components of the pixels in the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image; wherein the first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image; and the second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image;

[0195] The second fusion module is used to fuse, for each pixel in the second image, the brightness component of the pixel with the chrominance component of the pixel in the visible light image having the same coordinates as the pixel, to obtain a fused image.

[0196] Preferably, the contrast calculation module includes:

[0197] A first contrast calculation submodule is used to obtain a first brightness mean value of the target area in the visible light brightness image and a second brightness mean value of the non-target area in the visible light brightness image according to the position information of the target area and the non-target area, and obtain a first brightness contrast according to a ratio of the first brightness mean value to the second brightness mean value;

[0198] The expected contrast calculation submodule is used to obtain a third brightness mean of the target area in the first image and a fourth brightness mean of the non-target area in the first image according to the position information of the target area and the non-target area; and calculate the expected brightness mean of the target area in the first image so that the first brightness contrast is equal to the second brightness contrast, wherein the second brightness contrast is the ratio of the expected brightness mean to the fourth brightness mean.

[0199] See also Figure 5 As shown, Figure 5 This is a schematic diagram of another fusion device for visible light images and infrared images of the present application. The fusion device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the fusion method of the visible light brightness image and the infrared brightness image.

[0200] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0201] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0202] To understand the relationship between the various physical quantities, see Figure 6 As shown, Figure 6 It is a schematic diagram of the relationship between various physical quantities and images in the embodiments of the present application.

[0203] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for fusing a visible light brightness image and an infrared brightness image are implemented.

[0204] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for fusing a visible light image and an infrared image, characterized in that: The method includes, Acquire a visible light brightness image based on the visible light image, and acquire an infrared brightness image based on the infrared image; Determine coordinate information of a first pixel point in a target area of ​​the visible light brightness image and coordinate information of a second pixel point in a non-target area by using the visible light brightness image and the infrared brightness image; Fusing the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image; Determine a target area in the first image according to the coordinate information of the first pixel point, and determine a non-target area in the first image according to the coordinate information of the second pixel point; The brightness components of the pixels in the first image are adjusted so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image; wherein the first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image; and the second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image; For each pixel in the second image, a luminance component of the pixel is fused with a chrominance component of a pixel in the visible light image having the same coordinates as the pixel to obtain a fused image; in, The coordinate information of the first pixel point in the target area of ​​the visible light brightness image and the coordinate information of the second pixel point in the non-target area are determined as follows: Calculate the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image to obtain the brightness residual of each pixel. The coordinate information of the pixel whose brightness residual is greater than or equal to the preset brightness threshold is used as the coordinate information of the first pixel in the target area of ​​the visible light brightness image. Using the coordinate information of the pixel point whose brightness residual is less than the brightness threshold as the coordinate information of the second pixel point in the non-target area of ​​the visible light brightness image; The target area and the non-target area in the first image are determined as follows: The coordinate information of the first pixel point is mapped to the first image, so that the position information of the target area in the visible light brightness image is mapped to the first image, and the target area in the first image is obtained. Corresponding the coordinate information of the second pixel point to the first image, so that the position information of the non-target area of ​​the visible light brightness image corresponds to the first image, and obtaining the non-target area in the first image; The first brightness contrast is determined as follows: Calculate the mean value of the brightness component of the first pixel in the target area of ​​the visible light brightness image to obtain a first brightness mean value. Calculate the mean value of the brightness component of the second pixel in the non-target area of ​​the visible light brightness image to obtain a second brightness mean value. The ratio of the first brightness mean to the second brightness mean is taken as the first brightness contrast. The second brightness contrast is determined as follows: Calculate the mean value of the brightness component of the third pixel in the target area of ​​the first image to obtain a third brightness mean value, Calculate the mean value of the brightness component of the fourth pixel in the non-target area of ​​the first image to obtain a fourth brightness mean value, The ratio of the third brightness average value to the fourth brightness average value is used as the second brightness contrast.

2. The fusion method according to claim 1, characterized in that: Said Calculating the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image further includes: The brightness residual of each pixel is weighted by using a first weight calculated based on the brightness component of each pixel in the visible light image and a second weight calculated based on the brightness component of the pixel in the infrared image, so that the brightness residual of each pixel is corrected.

3. The fusion method according to claim 2, characterized in that: The calculation of the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image, include, For each pixel in the visible light brightness image and / or infrared brightness image: If the first brightness component of the pixel in the infrared brightness image is greater than or equal to the second brightness component of the pixel in the visible light brightness image, then the difference between the first brightness component and the second brightness component is calculated to obtain the brightness residual of the pixel; If the first brightness component of the pixel in the infrared brightness image is less than the second brightness component of the pixel in the visible light brightness image, the brightness residual of the pixel is set to 0.

4. The fusion method according to claim 2 or 3, characterized in that: The method of weighting the brightness residual of each pixel point by using a first weight value calculated based on the brightness component of each pixel point in the visible light image and a second weight value calculated based on the brightness component of the pixel point in the infrared image comprises: For each pixel's brightness residual: The brightness residual of the pixel point is filtered by using a filter function, and the filtered brightness residual is multiplied by the first weight value and the second weight value of the pixel point to obtain a corrected brightness residual of the pixel point; or, The brightness residual of the pixel point is multiplied by the first weight of the pixel point and the second weight value to obtain the corrected brightness residual of the pixel point; wherein, The first weight is determined as follows: When the brightness component of the pixel in the visible light brightness image is less than or equal to a set first threshold, the first weight value is a preset upper limit of the weight value; When the brightness component of the pixel in the visible light brightness image is greater than a set first threshold, the first weight value is between a preset weight lower limit and a preset weight upper limit. The second weight is determined as follows: When the brightness component of the pixel in the infrared brightness image is less than or equal to the set second threshold, the second weight value is less than the preset upper limit of the weight; When the brightness component of the pixel in the infrared brightness image is greater than or equal to the second threshold and less than the third threshold, the second weight is set to a preset upper weight limit; When the brightness of the pixel in the infrared brightness image is greater than or equal to the third threshold, the second weight is between a preset weight lower limit and a preset weight upper limit; The first threshold is determined according to the upper limit of the brightness component of the visible light brightness image; The second threshold and the third threshold are determined according to the upper limit of the brightness component of the infrared brightness image.

5. The fusion method according to claim 4, characterized in that: When the brightness component of the pixel in the visible light brightness image is greater than a set first threshold, the value of the first weight value is: the ratio of the difference between the upper limit of the brightness component of the visible light brightness image and the brightness component of the pixel, and the difference between the upper limit of the brightness component of the visible light brightness image and the first threshold; When the brightness component of the pixel in the infrared brightness image is less than or equal to the set second threshold, the value of the second weight is: the ratio of the brightness component of the pixel to the second threshold, When the brightness of the pixel in the infrared brightness image is greater than or equal to the third threshold, the value of the second weight is: the ratio of the difference between the upper limit of the brightness component of the infrared brightness image and the brightness component of the pixel, and the difference between the upper limit of the brightness component of the infrared brightness image and the third threshold.

6. The fusion method according to claim 1, characterized in that: The step of adjusting the brightness component of the pixels of the first image so that the second brightness contrast ratio between the target area and the non-target area in the first image is the same as the first brightness contrast ratio to obtain the second image includes: Calculate the first brightness contrast, Calculating a third brightness mean of a third pixel in the target area of ​​the first image and a fourth brightness mean of a fourth pixel in the non-target area of ​​the first image; The product of the first brightness contrast and the fourth brightness mean is calculated to obtain an expected brightness mean of the third pixel point in the target area of ​​the first image so that the first brightness contrast is equal to the second brightness contrast. For each pixel in the first image, weight the brightness component of the pixel in the first image by the third weight of the pixel to obtain a second image; in, The third weight is related to a brightness adjustment ratio, and the brightness adjustment ratio is determined according to the expected brightness mean value and the third brightness mean value.

7. The fusion method according to claim 6, characterized in that: The step of weighting, for each pixel in the first image, the brightness component of the pixel in the first image by the third weight of the pixel to obtain the second image comprises: Calculate the ratio of the third brightness average to the expected brightness average to obtain the brightness adjustment ratio. According to the brightness adjustment ratio and the corrected brightness residual of each pixel, the third weight of each pixel is calculated. For each pixel in the first image, the third weight of the pixel is multiplied by the brightness component of each pixel in the first image. All pixels with weighted brightness components form a second image.

8. The fusion method according to claim 7, characterized in that: The third weight of each pixel is calculated according to the brightness adjustment ratio and the corrected brightness residual of each pixel, including: Calculate the product of the brightness adjustment ratio and the brightness residual after correction of the pixel point to obtain the product result, The ratio of the multiplication result to the corrected upper limit of the brightness residual value is calculated to obtain a third weight of the pixel.

9. A device for fusing visible light images and infrared images, characterized in that: The device includes: A brightness image acquisition module, used to acquire a visible light brightness image based on the visible light image, and to acquire an infrared brightness image based on the infrared image; An area determination module, used to determine the coordinate information of a first pixel point in a target area of ​​the visible light brightness image and the coordinate information of a second pixel point in a non-target area by using the visible light brightness image and the infrared brightness image; A first fusion module is used to fuse the visible light brightness image and the infrared brightness image to obtain a brightness-fused first image; An adjustment module is used to adjust the brightness components of the pixels in the first image so that the second brightness contrast between the target area and the non-target area in the first image is the same as the first brightness contrast, thereby obtaining a second image; wherein the first brightness contrast is calculated based on the brightness components of the first pixel in the target area and the second pixel in the non-target area of ​​the visible light brightness image; and the second brightness contrast is calculated based on the brightness components of the third pixel in the target area and the fourth pixel in the non-target area of ​​the first image; A second fusion module is used to fuse, for each pixel in the second image, a brightness component of the pixel with a chrominance component of a pixel in the visible light image having the same coordinates as the pixel, to obtain a fused image; in, The coordinate information of the first pixel point in the target area of ​​the visible light brightness image and the coordinate information of the second pixel point in the non-target area are determined as follows: Calculate the brightness residual between the brightness component of each pixel in the infrared brightness image and the brightness component of the corresponding pixel in the visible light brightness image to obtain the brightness residual of each pixel. The coordinate information of the pixel whose brightness residual is greater than or equal to the preset brightness threshold is used as the coordinate information of the first pixel in the target area of ​​the visible light brightness image. Using the coordinate information of the pixel point whose brightness residual is less than the brightness threshold as the coordinate information of the second pixel point in the non-target area of ​​the visible light brightness image; The target area and the non-target area in the first image are determined as follows: The coordinate information of the first pixel point is mapped to the first image, so that the position information of the target area in the visible light brightness image is mapped to the first image, and the target area in the first image is obtained. Corresponding the coordinate information of the second pixel point to the first image, so that the position information of the non-target area of ​​the visible light brightness image corresponds to the first image, and obtaining the non-target area in the first image; The first brightness contrast is determined as follows: Calculate the mean value of the brightness component of the first pixel in the target area of ​​the visible light brightness image to obtain a first brightness mean value. Calculate the mean value of the brightness component of the second pixel in the non-target area of ​​the visible light brightness image to obtain a second brightness mean value. The ratio of the first brightness mean to the second brightness mean is taken as the first brightness contrast. The second brightness contrast is determined as follows: Calculate the mean value of the brightness component of the third pixel in the target area of ​​the first image to obtain a third brightness mean value, Calculate the mean value of the brightness component of the fourth pixel in the non-target area of ​​the first image to obtain a fourth brightness mean value, The ratio of the third brightness average value to the fourth brightness average value is used as the second brightness contrast.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for fusing a visible light image and an infrared image as claimed in any one of claims 1 to 8 are implemented.

Images