Image fusion method, device, equipment, and storage medium

Abstract

The present application provides an image fusion method, apparatus, image processing device, and computer-readable storage medium. The image fusion method includes acquiring a visible light image and an infrared image simultaneously captured for a target field of view; binarizing the infrared image to obtain a mask image, determining a target fusion region based on the mask image; and fusing the infrared image with the visible light image based on the target fusion region to obtain a fused image. Determining the target fusion region allows extraction of valid information contained in the infrared image, which is then fused with the visible light image. This effectively avoids the degradation of image quality caused by the inclusion of useless information in the fused image, reduces invalid information in the image, reduces computational complexity, and improves system real-time performance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, apparatus, image processing device, and computer-readable storage medium. Background Art

[0002] Images are mainly divided into visible light images and infrared images. Visible light images are rich in high-frequency details and can better reflect the overall detailed characteristics of the shooting scene. However, under poor lighting conditions, the image quality is reduced, and the target to be detected and the environmental background in the image become blurred. The principle of infrared imaging is mainly to display the shape and outline of the object through the intensity of the object's thermal radiation. It has good adaptability to weather and lighting, especially for hidden heat source targets. It has good detectability. However, infrared imaging has problems such as blurred image details, unclear textures, relatively little high-frequency information of the scene, poor contrast, and low clarity.

[0003] In order to take into account the respective advantages of visible light images and infrared images, visible light images and infrared images are fused in application scenarios to obtain a comprehensive and accurate image description of the shooting scene, so as to make full use of the information and improve the accuracy and reliability of system analysis and decision-making.

[0004] Image fusion methods for fusing visible light and infrared images can be categorized into three main types based on the complexity of information processing during the fusion process: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion operates on the pixels of an image to produce a fused image. Its advantage is that it retains much of the information contained in the original images. However, its disadvantage is that it requires traversal analysis and calculation of all the image pixels, resulting in high computational complexity and low real-time performance. Feature-level image fusion first extracts feature information such as edges, shapes, textures, and pixel density from the images to be fused. A multidimensional vector space is then constructed based on these extracted features. The feature vectors in this vector space are then analyzed and processed to form a feature set for the image. This is then trained and the fused image is fused based on the training results. Currently, feature-level image fusion often uses artificial neural network algorithms. Its advantages are fast processing speed and low computational complexity; its disadvantages are significant information loss and high requirements for the operating system. Decision-level image fusion first performs feature extraction, target feature recognition, and decision classification on the images to be fused, establishing a preliminary decision on the same target. The decision information from the visible light and infrared images is then fused based on the reliability of the fusion rules, ultimately yielding a joint decision. At present, decision-level fusion methods mainly include fusion algorithms based on support vector machines, neural networks, evidential reasoning, Bayesian reasoning and fuzzy integrals, which are highly complex and have higher requirements on the operating system. Summary of the Invention

[0005] To solve the existing technical problems, the present application provides an image fusion method, apparatus, image processing device and computer-readable storage medium that can reduce invalid information in images, reduce computational complexity and improve the real-time performance of the system.

[0006] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides an image fusion method, applied to an image processing device, comprising:

[0008] Acquire visible light images and infrared images collected synchronously for the target field of view;

[0009] Binarizing the infrared image to obtain a mask image, and determining a target fusion area according to the mask image;

[0010] The infrared image is fused with the visible light image based on the target fusion area to obtain a fused image.

[0011] In a second aspect, an embodiment of the present application provides an image fusion device, comprising:

[0012] An acquisition module, used to acquire visible light images and infrared images collected synchronously for the target field of view;

[0013] a fusion region determination module, configured to binarize the infrared image to obtain a mask image, and determine a target fusion region based on the mask image;

[0014] A fusion module is used to fuse the infrared image with the visible light image based on the target fusion area to obtain a fused image.

[0015] In a third aspect, an embodiment of the present application provides an image processing device, comprising a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, the image fusion method described in any embodiment of the present application applied to the terminal device side is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the image fusion method as described in any embodiment of the present application is implemented.

[0017] In the above embodiment, the infrared image is binarized to obtain a mask image, the target fusion area is determined according to the mask image, and the infrared image is fused with the visible light image based on the target fusion area to obtain a fused image. In this way, by determining the target fusion area, part of the effective information contained in the infrared image can be extracted, and the extracted effective information is fused with the visible light image, which can effectively avoid the fused image containing useless information and causing the image quality to deteriorate, reduce invalid information in the image, reduce the amount of calculation and complexity, and improve the real-time performance of the system.

[0018] In the above embodiments, the image fusion apparatus, image processing device and computer-readable storage medium belong to the same concept as the corresponding image fusion method embodiments, and thus have the same technical effects as the corresponding image fusion method embodiments, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of an application scenario of an image fusion method in one embodiment;

[0020] Figure 2 is a flow chart of an image fusion method in one embodiment;

[0021] Figure 3 is a flowchart of an image fusion method in another embodiment;

[0022] Figure 4 is a flowchart of an image fusion method in another embodiment;

[0023] Figure 5 is a schematic diagram of a grayscale histogram of an infrared image in an example;

[0024] Figure 6 Schematic diagram showing a unimodal distribution of grayscale histogram data in an example;

[0025] Figure 7 The grayscale histogram of the infrared image is unimodal, and the comparison diagram of the triangle method, Gaussian method and Otsu method is shown;

[0026] Figure 8 The grayscale histogram of the infrared image is roughly evenly distributed, and the comparison diagram of the triangle method, Gaussian method and Otsu method is shown;

[0027] Figure 9 The grayscale histogram of the infrared image shows a bimodal distribution, and the comparison diagram of the triangle method, Gaussian method and Otsu method is shown;

[0028] Figure 10 is a flowchart of an image fusion method in an optional specific example;

[0029] Figure 11 for Figure 10 A schematic diagram of an infrared image used in the illustrated embodiment;

[0030] Figure 12 for Figure 10 A schematic diagram of a visible light image used in the illustrated embodiment;

[0031] Figure 13 Schematic diagram of a fused image obtained by fusing an infrared image and a visible light image using the image fusion method described in this application;

[0032] Figure 14 is a schematic diagram of a fused image obtained by fusing an infrared image and a visible light image using a known low-rank representation principle;

[0033] Figure 15 A schematic diagram of a fused image obtained by fusing an infrared image and a visible light image using a known non-subsampled shearlet transform principle;

[0034] Figure 16 A schematic diagram of a fused image obtained by fusing an infrared image and a visible light image using a known non-subsampled contourlet transform principle;

[0035] Figure 17 Schematic diagram of a fused image obtained by fusing an infrared image and a visible light image using the known Poisson-based image editing principle;

[0036] Figure 18 is a schematic diagram of an image fusion device in one embodiment;

[0037] Figure 19 FIG. 1 is a schematic structural diagram of an image processing device in an embodiment. DETAILED DESCRIPTION

[0038] The technical solution of this application is further elaborated in detail below with reference to the accompanying drawings and specific embodiments.

[0039] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] In the following description, the expression "some embodiments" is involved, which describes a subset of all possible embodiments. It should be noted that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0041] In the following description, the terms "first, second, and third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first, second, and third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] See also Figure 1 , is a schematic diagram of an optional application scenario of the image processing method provided in an embodiment of the present application, wherein an image processing device 11 includes a processor 12, a memory 13 connected to the processor 12, a visible light camera module 15, and an infrared camera module 14. The image processing device 11 synchronously captures visible light images and infrared images in real time through the visible light camera module 15 and the infrared camera module 14 and sends them to the processor 12. The memory 13 stores a computer program that implements the image fusion method provided in an embodiment of the present application. By executing the computer program, the processor 12 binarizes the infrared image to obtain a mask image, determines a target fusion area based on the mask image, and fuses the infrared image with the visible light image based on the target fusion area to obtain a fused image. Among them, the image processing device 11 can be various types of intelligent terminals that are integrated with a visible light shooting module 15 and an infrared shooting module 14 and have storage and processing functions, such as security monitoring equipment, vehicle-mounted equipment, etc.; the image processing device 11 can also be a computing device connected to the visible light shooting module 15 and the infrared shooting module 14; the image processing device 11 can also be a dual-light fusion aiming device of white light and red light.

[0043] See also Figure 2 , an image fusion method provided in an embodiment of the present application, can be applied to Figure 1 The image processing device shown in FIG. The image processing method includes the following steps:

[0044] S101, acquiring a visible light image and an infrared image synchronously collected for a target field of view.

[0045] The visible light image and the infrared image are acquired synchronously with respect to the target field of view, so that the visible light image and the infrared image contain images of objects within the same target field of view. Optionally, the image processing device includes a visible light shooting module and an infrared shooting module, and the acquisition of the visible light image and the infrared image acquired synchronously with respect to the target field of view includes: the image processing device simultaneously acquires the visible light image and the infrared image through the visible light shooting module and the infrared shooting module, and sends the acquired visible light image and infrared image to the processor. In other optional embodiments, the image processing device does not include an image shooting module, and the acquisition of the visible light image and the infrared image acquired synchronously with respect to the target field of view includes: the image processing device acquires the visible light image and the infrared image acquired synchronously with respect to the target field of view sent by other smart devices with visible light image and infrared image shooting functions, where the other smart devices may include infrared detectors, mobile phone terminals, cloud computing, etc.

[0046] S103 , binarizing the infrared image to obtain a mask image, and determining a target fusion area according to the mask image.

[0047] Binarization of an infrared image refers to assigning grayscale values ​​to each pixel on the infrared image to obtain a binary image that can reflect the overall and local features of the image.

[0048] In an optional embodiment, the step S103 of binarizing the infrared image to obtain a mask image, and determining the target fusion area according to the mask image includes:

[0049] Comparing the grayscale value of each pixel in the infrared image with a binarization threshold, setting the grayscale value of the pixel whose grayscale value is less than the binarization threshold to a first set value, and setting the grayscale value of the pixel whose grayscale value is greater than or equal to the binarization threshold to a second set value, to obtain a mask image;

[0050] At least a portion of a pixel distribution area of ​​the second set value in the mask image is selected as a target fusion area.

[0051] The binarization threshold can be pre-set or calculated based on the distribution characteristics of the grayscale values ​​of the pixels in the infrared image. The first set value and the second set value can be selected from the maximum value and the minimum value of the grayscale value interval, respectively, or two grayscale values ​​close to the maximum value and the minimum value in the grayscale value interval can be selected. In a specific example, the first set value is 0 and the second set value is 255, so that the entire image appears to be a black and white image effect. The infrared image is binarized to obtain a mask image, including: binarizing the grayscale image of 256 brightness levels in the infrared image through a binarization threshold, comparing the grayscale value of each pixel in the infrared image with the binarization threshold, setting the grayscale value of the pixel whose grayscale value is less than the binarization threshold to 0, and setting the grayscale value of the pixel whose grayscale value is greater than the binarization threshold to 255, thereby obtaining a binarized image that can reflect the overall and local features of the image, that is, a mask image. Correspondingly, the target fusion area is determined according to the mask image, and the target fusion area can be determined according to the pixel distribution area of ​​the second set value in the mask image, that is, the white part. For example, the entire white part in the mask image can be selected as the target fusion area, or a part of the white part in the mask image can be selected as the target fusion area.

[0052] S105 , fusing the infrared image with the visible light image based on the target fusion area to obtain a fused image.

[0053] Fusing the infrared image with the visible light image based on the target fusion area to obtain a fused image may mean fusing the infrared image and the visible light image with image parts corresponding to the target fusion area respectively, while retaining the image part of the visible light image in other parts to obtain a fused image; or extracting the target fusion area of ​​the infrared image to form an image to be fused, and fusing the image to be fused with the visible light image, etc.

[0054] In the above embodiment, the infrared image is binarized to obtain a mask image, the target fusion area is determined according to the mask image, and the infrared image is fused with the visible light image based on the target fusion area to obtain a fused image. In this way, by determining the target fusion area, part of the effective information contained in the infrared image can be extracted, and the extracted effective information is fused with the visible light image, which can effectively avoid the fused image containing useless information and causing the image quality to deteriorate, reduce invalid information in the image, reduce the amount of calculation and complexity, and improve the real-time performance of the system.

[0055] Optional, see Figure 3 S105, fusing the infrared image with the visible light image based on the target fusion area to obtain a fused image, including:

[0056] S1051 , performing channel separation on the infrared image and the visible light image respectively, and fusing the two separated brightness channel components representing the brightness of the images according to the target fusion area to obtain a brightness channel fused image.

[0057] A digital image appears to the human eye as a single image, but from a computer's perspective, it's a collection of points of varying brightness. For example, an M×N digital image can be represented by an M×N matrix, where the values ​​of each element in the matrix represent the brightness of the corresponding pixel at that location, with larger values ​​representing brighter pixels. Typically, grayscale images are represented by two-dimensional matrices, while color images are represented by three-dimensional matrices (M×N×3), also known as multi-channel images.

[0058] Channels can be used to alter the hue and color of an image. For example, if only the red channel is saved, the image retains only red elements and information. Each individual channel can be displayed as a grayscale image (not black and white). The brightness and darkness of a single channel's grayscale image correspond to the brightness and darkness of the channel's color, and thus represent the distribution of that channel's color / light across the image.

[0059] Separating the infrared image and the visible light image through channels, fusing the separated luminance channel components representing the image brightness according to a target fusion region, and obtaining a luminance channel fused image can refer to separating the infrared image and the visible light image through channels, fusing the portion of the luminance channel components representing the image brightness separated from the infrared image corresponding to the target fusion region, and fusing the portion of the luminance channel components representing the image brightness separated from the visible light image corresponding to the target fusion region, and obtaining a luminance channel fused image. Fusion of the separated luminance channel components from the infrared image and the visible light image according to the target fusion region can reduce the amount of computation required for fusion and preserve valid information within the target fusion region.

[0060] S1052: Fuse the brightness channel fused image and the visible light image to obtain a fused image.

[0061] Image fusion involves applying image processing technology to image data collected from multiple channels on the same target, maximizing the extraction of useful information from each channel and ultimately synthesizing it into a high-quality image. This improves the utilization of image information, the accuracy and reliability of computer interpretation, and the spatial and spectral resolution of the original image, facilitating monitoring. A luminance channel fused image contains valid information from both the luminance channel components of the visible light image and the luminance channel components of the infrared image. The luminance channel fused image is fused with the visible light image, merging the luminance channel components of the luminance channel fused image with the other channel components of the visible light image to produce a fused image.

[0062] In the above embodiment, the infrared image is binarized to obtain a mask image, a target fusion region is determined based on the mask image, the infrared image and the visible light image are channel-separated, the separated luminance channel components representing the image brightness are fused based on the target fusion region to obtain a luminance channel fused image, and the luminance channel fused image is fused with the visible light image to obtain a fused image. In this way, by determining the target fusion region, it is possible to ensure that the effective information contained in the infrared image is extracted. The luminance channel components separated from the infrared image and the visible light image are fused and then fused with the visible light image. This can effectively prevent the fused image from containing useless information and thus causing image quality degradation, reduce invalid information in the image, reduce the amount of computation and complexity, and improve the real-time performance of the system. The fused image can simultaneously retain the advantages of both the visible light image and the infrared image. Whether it is a fused image obtained after imaging under sufficient lighting conditions or a fused image obtained after imaging under poor lighting conditions, it can better highlight the target, ensure that the target of interest in the image is presented more clearly, and is more convenient for human eye observation and recognition.

[0063] In some embodiments, see Figure 4 In the step of binarizing the infrared image to obtain a mask image and determining the target fusion area according to the mask image in S103, before comparing the grayscale value of each pixel in the infrared image with the binarization threshold, the following steps are included:

[0064] S1031, determining a matching binarization strategy based on the grayscale histogram and the distribution characteristics of the average gradient of the infrared image;

[0065] S1032: Determine the binarization threshold according to the binarization strategy.

[0066] The infrared image is binarized according to a binarization threshold determined by the binarization strategy to obtain a mask image, and a target fusion area is determined according to the mask image.

[0067] Different binarization strategies employ different binarization methods based on different principles and are applicable to different target images. The distribution characteristics of the infrared image's grayscale histogram and average gradient can be used to determine whether the infrared image's grayscale histogram exhibits a unimodal, bimodal, or roughly uniform distribution, thereby determining the appropriate binarization strategy. For example, among the binarization strategies available, the triangle method is suitable if the infrared image's grayscale histogram exhibits a unimodal distribution; the Gaussian method is suitable if the infrared image's grayscale histogram exhibits a relatively uniform distribution; and the Otsu method is suitable if the infrared image's grayscale histogram exhibits a bimodal distribution. By determining the appropriate binarization strategy for the infrared image, the infrared image is binarized using the corresponding binarization strategy to generate a mask image. The white area within the mask image is then identified as the target fusion region.

[0068] In the above embodiment, the distribution characteristics of the grayscale histogram and the average gradient of the infrared image are analyzed to determine a binarization strategy adapted thereto, so as to ensure that after the infrared image is binarized, the image area where the target is located in the image can be more accurately binarized into a white part, so as to facilitate determination of the target fusion area based on the mask image, and after the brightness channel components separated from the visible light image and the infrared image are fused according to the target fusion area, the effective information in the image can be more completely and comprehensively retained.

[0069] The step S1031, determining a matching binarization strategy based on the grayscale histogram and the distribution characteristics of the average gradient of the infrared image, includes:

[0070] determining, based on a distribution characteristic of a grayscale histogram of the infrared image, whether the grayscale histogram presents a unimodal distribution;

[0071] If so, determine that the matching binarization strategy is the triangle method;

[0072] If not, the matching binarization strategy is determined to be the Gaussian method or the Otsu method according to a comparison result of the average gradient of the infrared image and the average gradient of the visible light image.

[0073] When determining a suitable binarization strategy for an infrared image, the distribution characteristics of the infrared image's histogram are first used to determine whether the image is suitable for the triangle method. If not, the Gaussian method or the Otsu method is selected based on the average gradient. To determine whether the current infrared image binarization strategy is suitable for the triangle method, the infrared image's grayscale histogram data is used to determine whether it exhibits a unimodal distribution. Assuming that the maximum peak of the infrared image's grayscale histogram is near the brightest side, the optimal threshold for binarization is determined. If the infrared image's grayscale histogram data does not satisfy a unimodal distribution, the average gradient of the infrared image is compared with the average gradient of the visible light image to determine whether it exhibits a relatively uniform or bimodal distribution, thus determining whether the Gaussian method or the Otsu method is suitable.

[0074] In the above embodiment, the binarization strategies are set to include the triangle method, the Gaussian method, and the Otsu method, and the distribution characteristics of the grayscale histogram and the average gradient of the infrared image are analyzed to determine that the binarization strategy adapted thereto is the triangle method, the Gaussian method, or the Otsu method, so as to ensure that after the infrared image is binarized, the image area where the target in the image is located can be more accurately binarized as a white part, so that after the target fusion area is determined based on the mask image, the brightness component fusion image obtained by fusing the brightness channel components separated from the visible light image and the infrared image according to the target fusion area can be more completely and comprehensively retained.

[0075] In some embodiments, the step S1031 of determining a matching binarization strategy based on the grayscale histogram and the distribution characteristics of the average gradient of the infrared image includes:

[0076] Determining a difference between a grayscale mode and a grayscale average of the infrared image, and if the difference is less than or equal to a preset value, determining that a grayscale histogram of the infrared image presents a unimodal distribution;

[0077] Determine a triangle with the maximum peak in the grayscale histogram as a vertex;

[0078] The maximum straight-line distance is determined by the triangle, and a binarization threshold is determined according to the histogram grayscale level corresponding to the maximum straight-line distance.

[0079] According to the grayscale histogram data of the infrared image, if the grayscale values ​​in the grayscale histogram are concentrated at a certain value, then the difference between the grayscale value mode and the grayscale average value of the corresponding infrared image is much smaller than that of the infrared image whose grayscale histogram is in other forms. For example, the absolute value of the difference between the grayscale value mode and the grayscale average value can be recorded as AM (average-mode). By judging whether the difference AM between the grayscale value mode and the grayscale average value is less than a preset value, the degree of unimodal distribution of the grayscale histogram can be judged. If the difference AM is less than the preset value, it means that the grayscale histogram of the corresponding infrared image is unimodal distribution. Otherwise, it means that the grayscale histogram of the corresponding infrared image does not meet the characteristics of unimodal distribution. Figure 5 As shown in the figure, the grayscale histogram data of the infrared image is used to find the optimal binarization threshold based on a pure geometric method. Assuming that the maximum peak in the grayscale histogram is close to the brightest side, the maximum straight-line distance is calculated through the triangle, and the histogram grayscale level corresponding to the maximum straight-line distance is determined as the segmentation threshold.

[0080] like Figure 6As shown in the figure, the grayscale mode and grayscale average value in the grayscale histogram of the corresponding infrared image are 121 and 127.734 respectively. The grayscale mode is the grayscale value with the most repetitions among the grayscale values. The grayscale average value AG is calculated as follows:

[0081] (Formula 1)

[0082] H(i, j) represents the grayscale value of the pixel at coordinates (i, j), M represents the maximum value of the horizontal coordinate, and N represents the maximum value of the vertical coordinate. Assuming the preset value is 10, if the difference AM between the grayscale mode and the grayscale mean is less than 10, the infrared image is binarized using the triangulation method. A triangle is formed with the maximum peak in the grayscale histogram as its vertex. The maximum straight-line distance is determined through this triangle, and the binarization threshold is determined based on the histogram grayscale level corresponding to this maximum straight-line distance.

[0083] In the above embodiment, by calculating the difference between the grayscale value mode and the grayscale average in the grayscale histogram, based on the relative size of the difference and the preset value, it is measured whether the grayscale value is concentrated on a certain indicator to determine whether the grayscale histogram of the corresponding infrared image has a unimodal distribution, so as to achieve fast and accurate judgment of the distribution characteristics of the grayscale histogram of the infrared image.

[0084] See also Figure 7 , which is a comparative diagram of the infrared image grayscale histogram showing a unimodal distribution. After the infrared image is binarized using the triangle method, Gaussian method and Otsu method as the corresponding binarization strategies to obtain the mask image, it is fused with the visible light image. Among them, after using the triangle method as the binarization strategy to determine the binarization threshold, the mask image obtained by binarizing the infrared image can highlight the image target more comprehensively and completely. Finally, the triangle method fused image obtained after fusion with the visible light image has the smallest loss of effective image information.

[0085] In some embodiments, the step S1031 of determining a matching binarization strategy based on the grayscale histogram and the distribution characteristics of the average gradient of the infrared image further includes:

[0086] If the difference is greater than the preset value, determining a first average gradient of the infrared image and a second average gradient of the visible light image;

[0087] If the second average gradient is greater than or equal to the first average gradient, a Gaussian mean of the grayscale values ​​of the infrared image within the target window function is calculated, and a binarization threshold is determined according to the Gaussian mean.

[0088] If the difference AM between the grayscale mode and the grayscale mean is greater than a preset value, it indicates that the grayscale histogram of the corresponding infrared image does not meet the characteristic of a unimodal distribution. By determining the relative size of the average gradient of the infrared image and the average gradient of the visible light image, it is determined whether the grayscale histogram of the corresponding infrared image is relatively evenly distributed or bimodal. For ease of distinction, the average gradient of the infrared image is referred to as the first average gradient, and the average gradient of the visible light image is referred to as the second average gradient. The average gradient of the image can be calculated as follows:

[0089] (Formula 2)

[0090] H(i, j) represents the grayscale value of the pixel at coordinates (i, j), M represents the maximum value of the abscissa, and N represents the maximum value of the ordinate. If the second average gradient is greater than the first average gradient, it indicates that the grayscale histogram is relatively evenly distributed, and the Gaussian method is the appropriate binarization strategy for the current infrared image. The Gaussian method determines the binarization threshold by calculating the Gaussian mean of the image grayscale within the window function and using this Gaussian mean as the binarization threshold for binarization of that portion of the image. The Gaussian method realizes binarization by obtaining a local threshold value. By optimizing the scale of the window function, the part to be fused and the part not to be fused in the corresponding window function are determined. For infrared images whose grayscale values ​​in the grayscale histogram are not concentrated on a certain indicator, that is, infrared images whose grayscale values ​​are concentrated on multiple indicators, the image areas where the targets corresponding to the multiple indicators in the infrared image are located can be binarized into white parts in the mask image. After determining the target fusion area based on the white part in the mask image, the brightness channel components separated from the visible light image and the infrared image are fused according to the target fusion area to obtain a brightness component fusion image. This can more completely and comprehensively retain the effective information in the image, and the detail information in the fused image is enhanced and the edges are prominent.

[0091] In the above embodiment, the relative sizes of the average gradient of the visible light image and the average gradient of the infrared image are used to determine whether the grayscale values ​​are roughly evenly distributed, so as to quickly and accurately determine whether the binarization strategy of the infrared image complies with the Gaussian method. In the case where the average gradient of the visible light image is greater than the average gradient of the infrared image, the visible light image is clear and contains most of the effective information, so the Gaussian method is used to obtain the binarized image to ensure that after the infrared image is binarized, the image area where each target in the image is located can be more accurately binarized into a white part. After the target fusion area is determined based on the mask image, the brightness channel components separated from the visible light image and the infrared image are fused according to the target fusion area to obtain a brightness component fusion image, which can more completely and comprehensively retain the effective information in the image.

[0092] See also Figure 8, which is a comparative diagram of the infrared image grayscale histogram showing a roughly uniform distribution. The triangle method, Gaussian method, and Otsu method are used as the corresponding binarization strategies to binarize the infrared image to obtain the mask image, and then fuse it with the visible light image. Among them, after using the Gaussian method as the binarization strategy to determine the binarization threshold, the mask image obtained by binarizing the infrared image has a clearer and more prominent target outline. Finally, the Gaussian fused image obtained after fusion with the visible light image has the least loss of image effective information.

[0093] In some embodiments, determining a matching binarization strategy based on the distribution characteristics of the grayscale histogram and the average gradient of the infrared image further includes:

[0094] If the second average gradient is less than the first average gradient, segmenting the infrared image into a foreground image and a background image;

[0095] A binarization threshold is determined according to the inter-class variance values ​​of the foreground image and the background image.

[0096] If the difference AM between the grayscale mode and the grayscale mean is greater than a preset value, the grayscale histogram of the corresponding infrared image does not meet the unimodal distribution characteristic. By determining the relative magnitude of the average gradient of the infrared image and the average gradient of the visible light image, we can determine whether the grayscale histogram of the corresponding infrared image is relatively uniformly distributed or bimodal. For ease of distinction, the average gradient of the infrared image is referred to as the first average gradient, and the average gradient of the visible light image is referred to as the second average gradient. If the second average gradient is less than the first average gradient, it indicates that the grayscale histogram has a bimodal distribution, and the binarization strategy applicable to the current infrared image is the Otsu method. The principle of determining the binarization threshold using the Otsu method is to divide the image into background and target components based on the grayscale characteristics of the image. The greater the inter-class variance between the background and target, the greater the difference between the two components of the image. If part of the target is misclassified as background, or part of the background is misclassified as target, the difference between the two components will decrease. Therefore, binarization threshold segmentation based on the maximum inter-class variance can minimize the probability of misclassification.

[0097] In the above embodiment, the relative size of the average gradient of the visible light image and the average gradient of the infrared image is used to determine whether the grayscale histogram presents a bimodal distribution, thereby enabling a quick and accurate determination of whether the binarization strategy of the infrared image complies with the Otsu method. The infrared image is segmented into a foreground image and a background image using the Otsu method, wherein the foreground image contains primary information, such as energy radiated outward by a person or other heat-generating target object. The grayscale values ​​of pixels in the region of the foreground image containing the primary information differ significantly from those of adjacent pixels. Therefore, in the case where the average gradient of the infrared image is greater than the average gradient of the visible light image, it is more appropriate to determine the region containing valid information based on the foreground image of the infrared image. A binarized image of the infrared image is obtained using the Otsu method, ensuring that after the infrared image is binarized, the image region where the target is located in the image can be more accurately binarized into a white portion. After determining the target fusion region based on the mask image, the brightness channel components separated from the visible light image and the infrared image are fused according to the target fusion region to obtain a brightness component fusion image, which can more completely and comprehensively retain the valid information in the image.

[0098] See also Figure 9 , which is a comparative diagram of the grayscale histogram of the infrared image showing a bimodal distribution. The triangle method, Gaussian method and Otsu method are used as the corresponding binarization strategies to binarize the infrared image to obtain the mask image, and then fuse it with the visible light image. Among them, after using the Otsu method as the binarization strategy to determine the binarization threshold, the mask image obtained by binarizing the infrared image has clearer and more prominent contours of multiple targets. Finally, the Otsu method fused image obtained after fusion with the visible light image has the least loss of image effective information.

[0099] In some embodiments, the performing channel separation on the infrared image and the visible light image, and fusing the separated brightness channel components representing image brightness according to the target fusion area to obtain a brightness channel fused image, includes:

[0100] Separating the infrared image and the visible light image by HSI channels respectively, and fusing the two separated I channel components according to the Poisson image editing principle according to the target fusion area to obtain an I channel fused image;

[0101] The step of fusing the brightness channel fused image and the visible light image to obtain a fused image includes:

[0102] Merging the I channel component of the I channel fused image with the H channel component and the S channel component separated from the visible light image to obtain a fused reference image;

[0103] The fused reference image is converted into RGB color space to obtain a fused image.

[0104] Among them, HSI (Hue-Saturation-Intensity (Lightness)) refers to the color model of a digital image. The HSI color model uses three parameters: H, S, and I to describe the color characteristics of an image. H defines the frequency of the color, called hue; S represents the depth of the color, called saturation; and I represents intensity or brightness.

[0105] Optionally, the infrared image and the visible light image are respectively subjected to HSI channel separation, and the two I channel components separated from the target fusion areas corresponding to the infrared image and the visible light image are respectively fused according to the Poisson image editing principle to obtain an I channel fusion image, which includes: performing HSI channel separation on the visible light image to separate the H channel component, S channel component, and I channel component of the visible light image; determining a matching binarization strategy based on the distribution characteristics of the grayscale histogram and the average gradient of the infrared image, binarizing the infrared image according to the binarization strategy to obtain a mask image, and determining the target fusion area according to the mask image; performing HSI channel separation on the infrared image to separate the H channel component, S channel component, and I channel component of the infrared image; fusing the I channel component of the infrared image and the I channel component of the visible light image according to the target fusion area determined by the mask image according to the Poisson image editing principle to obtain an I channel fusion image. If the infrared image and the visible light image are non-HSI format images, the method further includes converting the infrared image and the visible light image into HSI format infrared images and visible light images before performing HSI channel separation on the infrared image and the visible light image.

[0106] In the above embodiment, the I-channel component of the visible light image to be fused and the I-channel component of the infrared image to be fused are extracted and fused according to the target fusion area specified by the mask image. The obtained I-channel fused image is merged with the H-channel component and the S-channel component separated from the visible light image, and then converted to the RGB color space to obtain a fused image. In this way, by respectively extracting the I-channel components of the visible light image and the infrared image, and determining the I-channel components within the target fusion area for fusion, it can be ensured that the obtained fused image can retain the effective information in the visible light image and the infrared image, and retain the detail information contained in the infrared image and the visible light image, while improving the objective evaluation index of the fused image, reducing the amount of calculation, and improving the processing efficiency of the fused image.

[0107] In order to have a more comprehensive understanding of the image fusion method provided in the embodiment of the present application, please refer to Figures 10 to 12 , taking an optional example as an example, the image fusion method is explained.

[0108] S11, read the infrared image and the visible light image; Figure 11 and Figure 12As shown, the infrared image IR_1 and the visible light image VIS_2;

[0109] Select the adaptive threshold binarization strategy for infrared image matching to binarize the infrared image. The adaptive threshold binarization strategies include the triangle method, Gaussian method, and Otsu method. The methods for selecting the adaptive binarization strategy include:

[0110] S121, according to the gray histogram of the infrared image Figure 2 The value distribution characteristics are calculated to calculate the difference AM value between the gray value mode and the gray value average of the image gray value;

[0111] S122, determining whether the AM value is greater than a preset value;

[0112] S123, if the AM value is less than or equal to the preset value, it indicates that the grayscale histogram has a unimodal distribution, and the infrared image is binarized using the triangle method;

[0113] S124, if the AM value is greater than a preset value, calculating the average gradient AG1 of the infrared image and the average gradient AG2 of the visible light image;

[0114] S125, determining whether the difference between AG1 and AG2 is greater than 0;

[0115] S126, if the difference between AG1 and AG2 is less than or equal to 0, it indicates that the grayscale histogram is relatively evenly distributed, and the infrared image is binarized using the Gaussian method;

[0116] S127, if the difference between AG1 and AG2 is greater than 0, it indicates that the grayscale histogram has a bimodal distribution, and the infrared image is binarized using the Otsu method;

[0117] S13, generating a mask image based on the binarized image after the binarization process, and specifying the area to be fused based on the mask image;

[0118] S14, performing HSI channel separation on the infrared image to obtain an H channel component, an S channel component, and an I channel component of the infrared image;

[0119] S15, performing HSI channel separation on the visible light image to obtain an H channel component, an S channel component, and an I channel component of the visible light image;

[0120] S16, fusing the I channel components of the infrared image and the visible light image according to the Poisson principle in the area to be fused specified by the mask image;

[0121] S17, merging the I channel of the fused image with the H and S channels of the visible light image;

[0122] S18, converting the merged image into RGB color space to obtain a fused image.

[0123] The image fusion method provided in the above embodiment has at least the following characteristics:

[0124] First, the infrared image is binarized using a matching binarization strategy, and the area to be fused is determined based on the binarized image. The area to be fused is locked to reduce the amount of fusion calculations and improve the fusion processing efficiency, while effectively retaining the valid information in the image.

[0125] Second, a method is provided for selecting a matching binarization strategy based on the distribution characteristics of the infrared image's grayscale histogram and average gradient, determining whether the grayscale histogram exhibits a unimodal distribution, a relatively uniform distribution, or a bimodal distribution. This ensures that after the region to be fused is determined based on the binarization result, the effective information and details contained in the infrared and visible light images are preserved.

[0126] Third, after fusing the I channel components of the infrared image and the visible light image, they are combined with the H and S channel components of the visible light image to obtain a fused image. This can effectively avoid the fused image containing useless information and causing image quality degradation, reduce invalid information in the image, reduce the amount of calculation and complexity, and improve the real-time performance of the system. Figure 13 As shown in FIG, a fused image is obtained by fusing the infrared image and the visible light image using the image fusion method described in this application. Figure 14 The fusion contrast effect of infrared image and visible light image fusion based on the known low-rank representation principle, Figure 15 To use the known non-subsampled shearlet transform principle to fuse infrared images and visible light images to achieve fusion contrast effect, Figure 16 To use the known non-subsampled contourlet transform principle to fuse infrared images and visible light images, Figure 17 The fusion contrast effect is achieved by directly fusing the infrared image and the visible light image as a whole using the known Poisson image editing principle.

[0127] Figures 13 to 17 The comparison of the evaluation indicators of the corresponding fusion images is shown in Table 1 below:

[0128]

[0129] Here, IE (Information Entropy) refers to information entropy; SF (Spatial Frequency) refers to spatial frequency; RMSE (Root Mean Sqaured Error) refers to root mean square error; SSIM (Structural Similarity Index) refers to the structural similarity index; and TIME refers to the fusion processing time. NSST (Non-subsampled Shearlet Transform) refers to the non-subsampled shearlet transform; and NSCT (Non-subsampled contourlet transform) refers to the non-subsampled contourlet transform principle. As shown in the figure and Table 1, the fused image obtained by fusing the infrared image and the visible light image using the image fusion method described in this application has the lowest root mean square error, significantly reduced fusion processing time, a structural similarity index close to 1, and relatively high information entropy and spatial frequency. The overall image performance is significantly superior to that of fused images obtained by other fusion methods.

[0130] See also Figure 18 On the other hand, the present application provides an image fusion device, including: an acquisition module 131, used to acquire a visible light image and an infrared image synchronously collected for a target field of view; a fusion area determination module 132, used to binarize the infrared image to obtain a mask image, and determine a target fusion area based on the mask image; a fusion module 134, used to fuse the infrared image with the visible light image based on the target fusion area to obtain a fused image.

[0131] Optionally, the fusion module 134 is specifically used to separate the channels of the infrared image and the visible light image, fuse the separated brightness channel components representing the image brightness according to the target fusion area to obtain a brightness channel fusion image; and fuse the brightness channel fusion image and the visible light image to obtain a fusion image.

[0132] Optionally, the fusion area determination module 132 is specifically used to compare the grayscale value of each pixel in the infrared image with a binarization threshold, set the grayscale value of the pixel whose grayscale value is less than the binarization threshold to a first set value, and set the grayscale value of the pixel whose grayscale value is greater than or equal to the binarization threshold to a second set value, to obtain a mask image; and select at least a part of the pixel distribution area of ​​the second set value in the mask image as the target fusion area.

[0133] Optionally, the fusion region determination module 132 is further configured to determine a matching binarization strategy according to the grayscale histogram and the distribution characteristics of the average gradient of the infrared image; and determine a binarization threshold according to the binarization strategy.

[0134] Optionally, the fusion area determination module 132 is also used to determine whether the grayscale histogram of the infrared image has a unimodal distribution based on the distribution characteristics of the grayscale histogram; if so, determine that the matching binarization strategy is the triangle method; if not, determine that the matching binarization strategy is the Gaussian method or the Otsu method based on the comparison result of the average gradient of the infrared image and the average gradient of the visible light image.

[0135] Optionally, the fusion area determination module 132 is also used to determine the difference between the grayscale value mode and the grayscale average of the infrared image. If the difference is less than or equal to a preset value, it is determined that the grayscale histogram of the infrared image has a unimodal distribution; a triangle is determined with the maximum peak in the grayscale histogram as the vertex; the maximum straight-line distance is determined through the triangle, and a binarization threshold is determined according to the histogram grayscale level corresponding to the maximum straight-line distance.

[0136] Optionally, the fusion area determination module 132 is also used to determine the first average gradient of the infrared image and the second average gradient of the visible light image if the difference is greater than the preset value; if the second average gradient is greater than or equal to the first average gradient, calculate the Gaussian mean of the grayscale value of the infrared image within the target window function, and determine the binarization threshold based on the Gaussian mean.

[0137] Optionally, the fusion area determination module 132 is further used to segment the infrared image into a foreground image and a background image if the second average gradient is smaller than the first average gradient; and determine a binarization threshold according to the inter-class variance value of the foreground image and the background image.

[0138] Optionally, the fusion module 134 is further used to perform HSI channel separation on the infrared image and the visible light image respectively, fuse the two separated I channel components according to the Poisson image editing principle based on the target fusion area to obtain an I channel fused image; merge the I channel component of the I channel fused image with the H and S channel components separated from the visible light image to obtain a fused reference image; and convert the fused reference image into RGB color space to obtain a fused image.

[0139] It should be noted that the image fusion device provided in the above embodiment, in the process of implementing visible light image and infrared image fusion processing, is only illustrated by the division of the above-mentioned program modules. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the method steps described above. In addition, the image fusion device provided in the above embodiment and the image fusion method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and is not repeated here.

[0140] Another aspect of the present application provides an image processing device. Figure 19 , is an optional hardware structure diagram of the image processing device provided in an embodiment of the present application, the image processing device includes a processor 111 and a memory 112 connected to the processor 111, the memory 112 is used to store various categories of data to support the operation of the image processing device, and stores a computer program for implementing the image processing method provided in any embodiment of the present application. When the computer program is executed by the processor, the steps of the image processing method provided in any embodiment of the present application are implemented and the same technical effects can be achieved. To avoid repetition, it will not be described here.

[0141] Optionally, the image processing device also includes an infrared shooting module and a visible light shooting module connected to the processor 111, and the infrared shooting module and the visible light shooting module are used to synchronously shoot infrared images and visible light images of the same target field of view and send them to the processor 111 as images to be fused.

[0142] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effects are achieved. To avoid repetition, the details are not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0143] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0145] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image fusion method, applied to an image processing device, characterized in that: include: Acquire visible light images and infrared images collected synchronously for the target field of view; Determine a difference between a grayscale mode and a grayscale average of the infrared image; if the difference is less than or equal to a preset value, determine that a grayscale histogram of the infrared image presents a unimodal distribution, and determine that a matching binarization strategy is a triangle method; If the difference is greater than the preset value, determining a first average gradient of the infrared image and a second average gradient of the visible light image; If the second average gradient is greater than or equal to the first average gradient, determining that the matching binarization strategy is the Gaussian method; If the second average gradient is less than the first average gradient, determining that the matching binarization strategy is the Otsu method; Determine the binarization threshold according to the binarization strategy; Comparing the grayscale value of each pixel in the infrared image with a binarization threshold to obtain a target fusion area; Separating the infrared image and the visible light image by channel, respectively, fusing the portion of the luminance channel component representing image brightness separated from the infrared image corresponding to the target fusion area with the portion of the luminance channel component representing image brightness separated from the visible light image corresponding to the target fusion area, to obtain a luminance channel fused image; The brightness channel fused image and the visible light image are fused to obtain a fused image.

2. The image fusion method according to claim 1, wherein: The step of comparing the grayscale value of each pixel in the infrared image with a binarization threshold to obtain a target fusion area includes: Comparing the grayscale value of each pixel in the infrared image with a binarization threshold, setting the grayscale value of the pixel whose grayscale value is less than the binarization threshold to a first set value, and setting the grayscale value of the pixel whose grayscale value is greater than or equal to the binarization threshold to a second set value, to obtain a mask image; At least a portion of a pixel distribution area of ​​the second set value in the mask image is selected as a target fusion area.

3. The image fusion method according to claim 1, wherein: When the matching binarization strategy is the triangle method, the binarization threshold is determined according to the binarization strategy as follows: Determine a triangle with the maximum peak in the grayscale histogram as a vertex; The maximum straight-line distance is determined by the triangle, and a binarization threshold is determined according to the histogram grayscale level corresponding to the maximum straight-line distance.

4. The image fusion method according to claim 1, wherein: When the matched binarization strategy is the Gaussian method, determining the binarization threshold according to the binarization strategy specifically includes: calculating the Gaussian mean of the grayscale values ​​of the infrared image within the target window function, and determining the binarization threshold according to the Gaussian mean.

5. The image fusion method according to claim 1, wherein: When the matching binarization strategy is the Otsu method, the binarization threshold is determined according to the binarization strategy as follows: Segmenting the infrared image into a foreground image and a background image; A binarization threshold is determined according to the inter-class variance values ​​of the foreground image and the background image.

6. The image fusion method according to claim 1, wherein: The step of performing channel separation on the infrared image and the visible light image, and fusing the separated brightness channel components representing the brightness of the images according to the target fusion area to obtain a brightness channel fused image includes: Separating the infrared image and the visible light image by HSI channels respectively, and fusing the two separated I channel components according to the Poisson image editing principle according to the target fusion area to obtain an I channel fused image; The step of fusing the brightness channel fused image and the visible light image to obtain a fused image includes: Merging the I channel component of the I channel fused image with the H channel component and the S channel component separated from the visible light image to obtain a fused reference image; The fused reference image is converted into RGB color space to obtain a fused image.

7. An image fusion device, characterized in that: include: An acquisition module, used to acquire visible light images and infrared images collected synchronously for the target field of view; a fusion region determination module, configured to determine a difference between a grayscale mode and a grayscale average of the infrared image; if the difference is less than or equal to a preset value, determining that the grayscale histogram of the infrared image presents a unimodal distribution, and determining that a matching binarization strategy is a triangle method; If the difference is greater than the preset value, determining a first average gradient of the infrared image and a second average gradient of the visible light image; If the second average gradient is greater than or equal to the first average gradient, determining that the matching binarization strategy is the Gaussian method; If the second average gradient is less than the first average gradient, determining that the matching binarization strategy is the Otsu method; Determine the binarization threshold according to the binarization strategy; compare the grayscale value of each pixel in the infrared image with the binarization threshold to obtain the target fusion area; a fusion module, configured to perform channel separation on the infrared image and the visible light image, respectively, and fuse a portion of a luminance channel component representing image brightness separated from the infrared image corresponding to a target fusion area with a portion of a luminance channel component representing image brightness separated from the visible light image corresponding to the target fusion area, to obtain a luminance channel fused image; The brightness channel fused image and the visible light image are fused to obtain a fused image.

8. An image processing device, characterized in that The method comprises a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor, wherein the computer program implements the image fusion method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image fusion method according to any one of claims 1 to 6 is implemented.

Images