Image fusion method and device, image processing apparatus and storage medium

Abstract

The application discloses an image fusion method and device, an image processing device and a storage medium. The image fusion method comprises the following steps: decomposing a source image into a base layer and a detail layer through a bilateral texture filter, wherein the source image comprises an infrared image and a visible light image; fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fused image; fusing the detail layer of the source image based on visual fidelity to obtain a second fused image; and fusing the first fused image and the second fused image to obtain a fused image of the source image. The above technical scheme fuses the base layer of the source image through the saliency detection algorithm, fuses the detail layer through the visual fidelity, retains the contrast information and texture of the source image, avoids the generation of a pseudo-edge phenomenon, and improves the definition of the fusion of the infrared image and the visible light image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and particularly relate to an image fusion method and device, an image processing apparatus, and a storage medium. BACKGROUND

[0002] An infrared image reflects the temperature distribution of an object surface, and can separate a target from a background region, but it has problems of single color, low clarity, and serious loss of important detail information. A visible light image can provide background detail information and has good contour expressiveness. Fusion of the infrared image and the visible light image can improve the clarity and reliability of the image. With extensive research on fusion of the infrared image and the visible light image at home and abroad, the fusion of the infrared image and the visible light image is increasingly widely used in video monitoring, military, and night driving safety auxiliary driving, etc.

[0003] However, the existing technology cannot well preserve contrast information of source images in the fusion of the infrared image and the visible light image, and the texture of the fused image is not clear, and there can be a false edge phenomenon. Therefore, how to improve the clarity of the fusion of the infrared image and the visible light image is a technical problem to be solved at present. SUMMARY

[0004] The present application provides an image fusion method and device, an image processing apparatus, and a storage medium to solve the problem of low clarity in fusion of an infrared image and a visible light image.

[0005] In a first aspect, an embodiment of the present application provides an image fusion method, comprising:

[0006] decomposing a source image into a base layer and a detail layer by a bilateral texture filter, the source image comprising an infrared image and a visible light image;

[0007] fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fused image;

[0008] fusing the detail layer of the source image based on visual fidelity to obtain a second fused image;

[0009] fusing the first fused image and the second fused image to obtain a fused image of the source image.

[0010] In a second aspect, an embodiment of the present application provides an image fusion device, comprising:

[0011] a source image decomposition module configured to decompose a source image into a base layer and a detail layer by a bilateral texture filter, the source image comprising an infrared image and a visible light image;

[0012] a base layer fusion module configured to fuse a base layer of the source image based on a saliency detection algorithm to obtain a first fused image;

[0013] a detail layer fusion module configured to fuse a detail layer of the source image based on visual fidelity to obtain a second fused image;

[0014] a source image fusion module configured to fuse the first fused image and the second fused image to obtain a fused image of the source image.

[0015] In a third aspect, an embodiment of the present application provides an image processing device, comprising:

[0016] at least one processor; and

[0017] a memory in communication with the at least one processor; wherein

[0018] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the image fusion method according to the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image fusion method according to the first aspect.

[0020] The technical scheme of the embodiment of the present application fuses the base layer of the source image through the saliency detection algorithm, fuses the detail layer through the visual fidelity, retains the contrast information and the texture of the source image, avoids the generation of the pseudo edge phenomenon, and improves the definition of the fusion of the infrared image and the visible light image.

[0021] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is a flow chart of an image fusion method according to an embodiment of the present application;

[0024] Figure 2is a flow chart of an image fusion method according to the second embodiment of the present application;

[0025] Figure 3 is a flow chart of an image fusion method according to the third embodiment of the present application;

[0026] Figure 4 is a flow chart of an image fusion method according to the third embodiment of the present application;

[0027] Figure 5 is a structural schematic diagram of an image fusion device according to the fourth embodiment of the present application;

[0028] Figure 6 is a structural schematic diagram of an image processing device implementing the image fusion method according to the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the personnel in the field without creative labor should belong to the protection scope of the present application.

[0030] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] It can be understood that before using the technical solutions disclosed in the embodiments of the present application, the user should be informed of the type, use range and use scenario of the personal information involved in the present disclosure and obtain the authorization of the user in accordance with relevant laws and regulations.

[0032] Embodiment one

[0033] Figure 1This is a flowchart of an image fusion method provided in Embodiment 1 of the present invention. This embodiment is applicable to the fusion of infrared and visible light images. The method can be executed by an image fusion device, which can be implemented in software and / or hardware and integrated into an image processing device. Further, the image processing device includes, but is not limited to, computers, laptops, smartphones, servers, etc. Figure 1 As shown, the method includes:

[0034] S110. The source image is decomposed into a base layer and a detail layer using a bilateral texture filter. The source image includes infrared images and visible light images.

[0035] Specifically, the source image can be decomposed into a base layer and a detail layer. The base layer contains low-frequency information of the source image, reflecting the intensity changes of the source image at a large scale; the detail layer contains high-frequency information of the source image, reflecting the details of the source image at a small scale.

[0036] A bilateral filter is a non-linear filtering method that combines spatial proximity and pixel value similarity in an image, taking into account both spatial information and grayscale similarity to achieve edge preservation and noise reduction.

[0037] The bilateral texture filter is an improved version of the bilateral filter. It extends the bilateral filter by replacing the input image in the bilateral filter range kernel function with a texture description image. The bilateral texture filter defines the output image as follows:

[0038]

[0039] The input image is a source image that includes both infrared and visible light images. The source image is composed of pixels, and u and v refer to the pixels in the source image. u As a regularization term, the output image B at pixel u u isI u In the spatial neighborhood Ω u The weighted average in I u and I v G represents the input image at pixels u and v, respectively. u and G v Let u and v represent the texture description images at pixels u and v, respectively. The spatial kernel function f and the range kernel function g are typical Gaussian functions.

[0040] The texture description image can describe the texture of the image, wherein the texture can refer to a visual feature reflecting homogeneity in the image, embodying surface structure organization arrangement attributes of an object surface with slow change or periodic change, and the texture is expressed by the gray scale distribution of a pixel and its surrounding spatial neighborhood, i.e., local texture information. In addition, the repeatability of the local texture information to different degrees is global texture information.

[0041] The output image B at all pixels u is obtained by combining the texture description image and the source image I u , i.e., the base layer B of the source image after the bilateral texture filter. n , wherein n = 1 or 2, B1 and B2 represent the base layers of the infrared image and the visible light image, respectively.

[0042] The detail layer is represented as the difference between the source image and the base layer, and the calculation formula of the detail layer is:

[0043] D n = I n - B n

[0044] , wherein n = 1 or 2, I n is the source image, I1 and I2 represent the infrared image and the visible light image of the source image, respectively, D n is the detail layer, and D1 and D2 represent the detail layers of the infrared image and the visible light image, respectively.

[0045] Therefore, the base layer of the source image is obtained by the bilateral texture filter first, and the detail layer can be obtained by the difference between the source image and the base layer, so that the source image is decomposed into the base layer and the detail layer by the bilateral texture filter.

[0046] S120, fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fused image.

[0047] Specifically, the saliency detection algorithm can refer to an algorithm capable of detecting the visual saliency of the source image. Saliency is an important visual feature in the image, which can reflect the degree of emphasis of the human eye on some regions of the image. The first fused image can refer to an image fused from the base layer of the source image.

[0048] Wherein, the way of detecting the visual saliency of the source image is not limited, as long as it can detect the visual saliency of the source image. For example, the visual saliency of the source image can be detected based on a contrast feature; for another example, the visual saliency of the source image can be detected based on a spatial feature. The contrast feature can refer to the feature of different brightness levels between the brightest white and the darkest black in the light and dark regions of the source image, which can reflect the size of the gray scale contrast of the source image. The spatial feature can refer to the feature of the mutual spatial position or relative direction relationship between multiple targets segmented in the source image.

[0049] In one embodiment, the visual saliency of the source image can be detected by combining the features such as color, direction and intensity in a linear and non-linear manner.

[0050] The manner of fusing the base layer of the source image based on the saliency detection algorithm is not limited, as long as the base layer of the source image can be fused based on the saliency detection algorithm. For example, the visual saliency of the infrared image and the visible light image corresponding to the source image is extracted by the saliency detection algorithm, so as to obtain the saliency map of the infrared image and the visible light image. Then, the saliency map of the infrared image and the visible light image is compared to obtain the decision map of the infrared image and the visible light image. Finally, the saliency map and the decision map of the infrared image and the visible light image are combined, and the fusion image of the base layer of the source image, i.e., the first fusion image, is obtained by weighted average.

[0051] In S130, the detail layer of the source image is fused based on visual fidelity to obtain a second fusion image.

[0052] Specifically, the visual fidelity can refer to a method for evaluating image quality, which is an image quality evaluation parameter based on natural scene statistics and the concept of image signal extracted by the human visual system. The second fusion image can refer to an image fused from the detail layer of the source image.

[0053] The principle of evaluating image quality based on visual fidelity is as follows: first, a natural image is decomposed into several subbands, and each subband is further divided into blocks; then, in different models of each block and each subband, visual information is measured by calculating mutual information; finally, the image quality value is measured by integrating visual information of all modules and all subbands. A high visual fidelity indicates a high image quality, and a low visual fidelity indicates a low image quality. The meaning of image quality includes, but is not limited to, the fidelity of the image and the intelligibility of the image.

[0054] The manner of fusing the detail layer of the source image based on visual fidelity is not limited, as long as the detail layer of the source image can be fused based on visual fidelity. For example, the detail layer is decomposed into a plurality of image block groups, the visual fidelity of each image block group is calculated, the weight map is obtained by comparing the visual fidelity of each image block group and the visual fidelity threshold, and then the second fusion image is obtained by combining the visual fidelity texture.

[0055] The decomposition of the detail layer into a plurality of image block groups can be achieved by block decomposition. The purpose of block decomposition of the detail layer image can be to load large-size images into memory for processing in batches, enhance the applicability of the algorithm, and avoid the problem of being unable to process large images. After the large-size image is decomposed into blocks, the data amount and computing pressure of each operation are reduced.

[0056] The manner of decomposing the detail layer into a plurality of image block groups by block decomposition is not limited, and can be decomposed according to actual needs. For example, the detail layer is decomposed into 4*4 blocks, i.e., the detail layer is divided into 4 rows and 4 columns, and a total of 16 image block groups, each of which contains an image block of the detail layer of the infrared image and an image block of the detail layer of the corresponding visible light image. For another example, the detail layer is decomposed into 2*3 blocks, i.e., the detail layer is divided into 2 rows and 3 columns, and a total of 6 image block groups.

[0057] The manner of calculating the visual fidelity of each image block group is not limited, for example, a low-resolution image can be obtained by performing Gaussian pyramid transformation on each image block group, and the visual fidelity values of the low-resolution images from the detail layers of the infrared image and the visible light image are calculated. The visual fidelity threshold is not limited, and can be a value set according to actual needs. The weight map can be obtained by comparing the visual fidelity of each image block group with the visual fidelity threshold. When the visual fidelity value of the image exceeds the visual fidelity threshold, it can be understood that the visual fidelity of the image is high, and the weight map with high visual fidelity can be obtained. Otherwise, the visual fidelity of the image is low, and the weight map with low visual fidelity can be obtained.

[0058] The visual fidelity texture can refer to a texture capable of describing the visual fidelity of an image. The visual fidelity of each image block group and the visual fidelity threshold are compared, and the texture corresponding to the part exceeding the visual fidelity threshold is the texture with high visual fidelity, and vice versa, which is referred to as the texture with low visual fidelity.

[0059] The manner of obtaining the second fusion image according to the weight map and the visual fidelity texture is not limited, as long as the second fusion image can be obtained. For example, the second fusion image can be obtained by weighted average of the weight map and the visual fidelity texture.

[0060] S140, fusing the first fusion image and the second fusion image to obtain a fusion image of the source image.

[0061] Specifically, the fusion image of the source image can refer to an image obtained by fusing the infrared image and the visible light image of the source image.

[0062] The manner of fusing the first fusion image and the second fusion image is not limited, as long as the first fusion image and the second fusion image can be fused. For example, the fusion image of the source image can be obtained by adding the first fusion image and the second fusion image.

[0063] The technical scheme of the embodiment of the present application fuses the base layer of the source image by the saliency detection algorithm, fuses the detail layer by the visual fidelity, retains the contrast information and texture of the source image, avoids the generation of pseudo-edge phenomenon, and improves the clarity of the fusion of the infrared image and the visible light image.

[0064] Embodiment two

[0065] Figure 2 A flow chart of an image fusion method provided for embodiment two of the present application, the embodiment is based on the above-mentioned embodiment one, further refining the fusion of the base layer of the source image based on the saliency detection algorithm to obtain a first fusion image, as shown in the figure, the method comprises: Figure 2

[0066] S110, decompose the source image into a base layer and a detail layer through a bilateral texture filter, the source image comprising an infrared image and a visible light image.

[0067] S121, extract a first saliency map of the base layer of the infrared image and a second saliency map of the base layer of the visible light image.

[0068] Specifically, the first saliency map can refer to the saliency map of the base layer of the infrared image. The second saliency map can refer to the saliency map of the base layer of the visible light image.

[0069] The way of extracting the first saliency map of the base layer of the infrared image and the second saliency map of the base layer of the visible light image is not limited, as long as the first saliency map of the base layer of the infrared image and the second saliency map of the base layer of the visible light image can be extracted. For example, the first saliency map and the second saliency map can be obtained through a saliency map calculation formula.

[0070] When fusing the base layer of the source image, in order to avoid the problem of incoherence at the edge, clustering can be used to obtain a high cohesion global constraint, so as to measure the spatial contrast difference. The contrast based on clustering represents the image as C={C 1 ,C 2 ,...,C k}, different k values can be understood as clustering the base layer B n into different clusters C k .

[0071] The saliency map calculation formula is:

[0072]

[0073] Wherein, p(C k ) is the saliency value calculated by contrast, p(u|C k ) is the saliency possibility of pixel u belonging to cluster C k satisfying Gaussian distribution.

[0074] Respectively cluster the base layers B1 and B2 of the infrared image and the visible light image into different clusters C k ​Then, a first saliency map S1 of the base layer of the infrared image and a second saliency map S2 of the base layer of the visible light image are obtained through a saliency map calculation formula.

[0075] S122, the first saliency map and the second saliency map are compared to obtain a first decision map and a second decision map.

[0076] Specifically, the first decision map can refer to a decision map P1 of the base layer of the infrared image. The second decision map can refer to a decision map P2 of the base layer of the visible light image.

[0077] The manner of comparing the first saliency map and the second saliency map is not limited, as long as the first decision map and the second decision map can be obtained by comparing the first saliency map and the second saliency map. For example, a fusion rule can be designed according to the saliency map, and the two saliency maps are compared. If the saliency difference is large, the saliency of the larger one is taken; if the saliency is equivalent, the average of the two is taken.

[0078] The calculation formula of the first decision map is:

[0079]

[0080] Wherein, P1 is the first decision map, S1 is the first saliency map of the base layer of the infrared image, and S2 is the second saliency map of the base layer of the visible light image. When S1>S2, P1=1 is taken, which can be understood as when the saliency of S1 is greater than S2, P1=1 is taken; when S1≈S2, P1=0.5 is taken, S1≈S2 can be determined by the formula S1-S2<T, wherein T can refer to a saliency similarity threshold, the value of T can be set according to actual needs, as long as it can be used as a basis for determining S1≈S2, when S1-S2<T, it indicates that the saliency of S1 and S2 is equivalent, P1=0.5 is taken; when S1<S2, P1=0 is taken, which can be understood as when the saliency of S1 is less than S2, P1=0 is taken.

[0081] The calculation formula of the second decision map is:

[0082] P2=1-P1

[0083] Wherein, P1 is the first decision map, and P2 is the second decision map, which can be calculated according to the first decision map P1.

[0084] S123, a first fused image is generated according to the base layer of the infrared image, the base layer of the visible light image, the first decision map and the second decision map.

[0085] The calculation formula for generating the first fused image according to the base layer of the infrared image, the base layer of the visible light image, the first decision map and the second decision map is:

[0086] BF=B1×P1+B2×P2

[0087] BF=B1+P1*P2, wherein, BF is the first fusion image, B1 is a base layer of the infrared image, B2 is a base layer of the visible light image, P1 is the first decision map, and P2 is the second decision map.

[0088] S130, fusing the detail layer of the source image based on visual fidelity to obtain a second fusion image.

[0089] S140, fusing the first fusion image and the second fusion image to obtain a fusion image of the source image.

[0090] The technical scheme of the embodiment of the application extracts the saliency maps of the infrared image and the visible light image corresponding to the source image through the saliency detection algorithm, compares the saliency maps to obtain the decision maps of the infrared image and the visible light image, and obtains the first fusion image in combination with the saliency maps and the decision maps, thereby improving the definition of the fusion of the infrared image and the visible light image.

[0091] Embodiment three

[0092] Figure 3 A flowchart of an image fusion method provided by the embodiment three of the application, the embodiment is further refined based on the fusion of the detail layer of the source image based on visual fidelity to obtain a second fusion image, as shown in the following formula: Figure 3 The method comprises the following steps.

[0093] S110, decomposing the source image into a base layer and a detail layer through a bilateral texture filter, the source image comprising an infrared image and a visible light image.

[0094] S120, fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fusion image.

[0095] S131, decomposing the detail layer of the infrared image and the detail layer of the visible light image into a plurality of image blocks, each image block of the detail layer of the infrared image and the corresponding image block of the detail layer of the visible light image serving as an image block group.

[0096] The manner of decomposing the detail layer of the infrared image and the detail layer of the visible light image into a plurality of image blocks is not limited, for example, the decomposition can be achieved through block decomposition.

[0097] In one embodiment, the detail layer is decomposed into 4x4 blocks, i.e., the detail layer is divided into 4 rows and 4 columns, a total of 16 image block groups, each image block group comprising an image block of the detail layer of the infrared image and an image block of the detail layer of the corresponding visible light image.

[0098] S132, calculating the visual fidelity of each image block group.

[0099] The manner of calculating the visual fidelity of each image block group is not limited as long as the visual fidelity of each image block group can be calculated.

[0100] Optionally, before the visual fidelity of each image block group is calculated, the method further comprises: performing Gaussian pyramid transformation on each image block group.

[0101] The Gaussian pyramid can refer to a multi-scale representation of a signal, and different groups of signals or images of different scales can be obtained through Gaussian blur and downsampling. Through Gaussian pyramid transformation on each image block group, a low-resolution image corresponding to the detail layer of the infrared image and the detail layer of the visible light image can be obtained. and Wherein, i and j represent the horizontal and vertical coordinates of the image block group in the detail layer respectively, the values of i and j can be set according to actual needs, and i and j in combination can represent the position of the image block group in the detail layer.

[0102] In one embodiment, the detail layer is divided into 4x4 blocks, i.e., the detail layer is divided into 4 rows and 4 columns, a total of 16 image block groups, so 1≤i,j≤4, and i=1,j=3 can represent that the image block group is in the first row and the third column in the detail layer.

[0103] The low-resolution image corresponding to the detail layer of the infrared image and the detail layer of the visible light image can be used as the input of the fusion algorithm. and The visual fidelity of each image block group is calculated, and the calculation formula of the visual fidelity is:

[0104]

[0105] Wherein, VIFF can refer to Visual Information Fidelity for Fusion (VIFF), which is an image fusion measurement method based on visual fidelity, and this method has good effect on the prediction of image quality. For human eye perception, the higher the VIFF value obtained by calculation is, the less the distortion information is, and the higher the image quality is.

[0106] The VIFF of each subband can be defined as the ratio of visual information with distortion information (VID) and visual information without distortion information (VIND), and the formula is:

[0107]

[0108] Wherein, b represents a subband, and respectively represent low-resolution images corresponding to the detail layer of the infrared image and the detail layer of the visible light image, and viff ij and The calculation formula of the visual fidelity can obtain the visual fidelity of each image block group.

[0109] S133, according to the visual fidelity of each image block group and the visual fidelity threshold, determine the first weight map corresponding to the first type of visual fidelity and the second weight map corresponding to the second type of visual fidelity, wherein the first type of visual fidelity is higher than the second type of visual fidelity.

[0110] The first type of visual fidelity can refer to the visual fidelity of the image block group whose visual fidelity is higher than the visual fidelity threshold. The second type of visual fidelity can refer to the visual fidelity of the image block group whose visual fidelity is lower than the visual fidelity threshold. The first type of visual fidelity is higher than the second type of visual fidelity. The first type of visual fidelity corresponds to the first weight map, and the second type of visual fidelity corresponds to the second weight map. The visual fidelity of the first weight map is higher than the second weight map.

[0111] Specifically, the calculation formula of the first weight map is:

[0112]

[0113] Wherein, PD1 is the first weight map corresponding to the first type of visual fidelity, DT is the visual fidelity threshold, and the visual fidelity threshold can be a value set according to actual needs. When viff ij >DT, it indicates that the visual fidelity is high, and PD1=1.

[0114] The calculation formula of the second weight map is:

[0115] PD2=1-PD1

[0116] Wherein, PD2 is the second weight map corresponding to the second type of visual fidelity.

[0117] S134, according to the first type of visual fidelity texture corresponding to the first type of visual fidelity, the second type of visual fidelity texture corresponding to the second type of visual fidelity, the first weight map and the second weight map, generate a second fusion image.

[0118] Wherein, the first type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image, and the second type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image.

[0119] Specifically, the first type of visual fidelity corresponds to a first type of visual fidelity texture, i.e., a texture corresponding to a part higher than the visual fidelity threshold; and the second type of visual fidelity corresponds to a second type of visual fidelity texture, i.e., a texture corresponding to a part lower than the visual fidelity threshold.

[0120] Further, the image fusion method further comprises:

[0121] determining the first type of visual fidelity texture corresponding to the first type of visual fidelity based on the saliency estimation;

[0122] determining the second type of visual fidelity texture corresponding to the second type of visual fidelity based on the guided filtering.

[0123] The saliency estimation can refer to obtaining a saliency metric value by calculating the consistency and spatial distribution of image decomposition elements. The image decomposition elements can refer to decomposing an image into basic elements while retaining relevant structures. The consistency of the elements can be understood as evaluating the degree of difference of each element from all other elements in the image, which essentially measures the "uniqueness" of each element. The spatial distribution of the elements can be used as a judgment condition for saliency estimation. Elements distributed in a specific area rather than uniformly distributed in the entire image are more salient.

[0124] The guided filtering can refer to using the local linear relationship between the guide image and the filter output image to perform least squares between the calculated output image and the image to be filtered, so that the output image approximates the image to be filtered as much as possible. If the guide image is selected as the image to be filtered, the filtering and smoothing of the image to be filtered can be realized on the basis of retaining the local features of the image to be filtered.

[0125] Determining the first type of visual fidelity texture corresponding to the first type of visual fidelity based on the saliency estimation and determining the second type of visual fidelity texture corresponding to the second type of visual fidelity based on the guided filtering can retain the contrast information and texture of the source image while avoiding the generation of false edges.

[0126] Further, determining the first type of visual fidelity texture corresponding to the first type of visual fidelity based on the saliency estimation comprises:

[0127] determining the saliency estimation value of each element in the image block group of the first type of visual fidelity according to the regularization result of the element consistency and element distribution;

[0128] comparing the saliency estimation values of the elements to obtain the first type of visual fidelity texture corresponding to the first type of visual fidelity.

[0129] Specifically, the detail layer of the infrared image corresponding to the source image is decomposed into basic elements i, and a calculation formula of the significance estimation value of each element in the image block group of the first type of visual fidelity is determined according to the regularization result of element consistency and element allocation, as follows:

[0130] SD i = U i × exp (-k × D i )

[0131] wherein U i is the calculation result of element consistency, D i is the calculation result of element allocation, and SD i may be the significance estimation value of each element in the image block group of the first type of visual fidelity, which is regularized to the interval [0...1] according to the element consistency U i and the element allocation D i .

[0132] The significance estimation values of the elements are compared to obtain the decision map PSD of the detail layer of the infrared image, and the calculation formula is as follows:

[0133]

[0134] wherein SD1 may be the significance estimation value corresponding to the detail layer of the infrared image, and SD i is combined to obtain SD1. IT is a significance estimation threshold, and the value of IT can be set according to actual needs.

[0135] The first type of visual fidelity texture corresponding to the first type of visual fidelity can be obtained through the decision map PSD of the detail layer of the infrared image, the detail layer D1 of the infrared image, and the detail layer D2 of the visible light image. The calculation formula of the first type of visual fidelity texture T1 is as follows:

[0136] T1 = PSD × D1 + (1-PSD)D2

[0137] Further, the second type of visual fidelity texture corresponding to the second type of visual fidelity is determined based on guided filtering, including:

[0138] An initial decision map of the second type of visual fidelity is determined according to the comparison result of the detail layer of the infrared image and the detail layer of the visible light image corresponding to the image block group of the second type of visual fidelity.

[0139] The initial decision map and the visible light image are subjected to guided filtering according to the filtering scale and the blurring degree, to obtain the second type of visual fidelity texture corresponding to the second type of visual fidelity.

[0140] Specifically, the initial decision map M of the second type of visual fidelity can be obtained by the maximum absolute value method, that is, by comparing the absolute values of the detail layers of the infrared image corresponding to the image block group of the second type of visual fidelity and the absolute values of the detail layers of the visible light image, the initial decision map M of the second type of visual fidelity is determined, and the calculation formula is as follows:

[0141]

[0142] Since the initial decision map M is obtained only by the maximum absolute value method, spatial inconsistency is caused, which leads to poor visual effect, therefore, the guided filtering method can be introduced to solve this problem.

[0143] According to the filtering scale and the blur degree, the initial decision map and the visible light image are guided filtered to obtain the second type of visual fidelity corresponding to the second type of visual fidelity texture T2, and the calculation formula is as follows:

[0144] T2=GuidedFiltering(M,I2,r,e)

[0145] Wherein, T2 is the second type of visual fidelity corresponding to the second type of visual fidelity texture, M is the initial decision map of the second type of visual fidelity, r is the filtering scale of guided filtering, and e is the blur degree of guided filtering.

[0146] According to the first type of visual fidelity corresponding to the first type of visual fidelity texture, the second type of visual fidelity corresponding to the second type of visual fidelity texture, the first weight map and the second weight map, a second fusion image is generated, and the calculation formula of the second fusion image is:

[0147] TF=T1×PD1+T2×PD2

[0148] Wherein, TF is the second fusion image, T1 is the first type of visual fidelity corresponding to the first type of visual fidelity texture, T2 is the second type of visual fidelity corresponding to the second type of visual fidelity texture, PD1 is the first type of visual fidelity corresponding to the first weight map, and PD2 is the second type of visual fidelity corresponding to the second weight map.

[0149] S140, the first fusion image and the second fusion image are fused to obtain the fusion image of the source image.

[0150] Specifically, the calculation formula of the fusion image of the source image is:

[0151] F=BF+TF

[0152] Wherein, F is the fusion image of the source image, BF is the first fusion image, and TF is the second fusion image.

[0153] In one embodiment, Figure 4Fig. 1 is a flow chart of an image fusion method according to an embodiment of the present application. Figure 4 It can be known that:

[0154] Firstly, the infrared image I1 and the visible light image I2 of the source image are decomposed into a base layer and a detail layer by a bilateral texture filter, B1 and B2 represent the base layers of the infrared image and the visible light image respectively, and D1 and D2 represent the detail layers of the infrared image and the visible light image respectively.

[0155] Secondly, the base layers of the source image are fused based on a saliency detection algorithm, a first saliency map S1 of the base layer of the infrared image and a second saliency map S2 of the base layer of the visible light image are extracted, the first saliency map S1 is compared with the second saliency map S2 to obtain a first decision map P1 and a second decision map P2, and a first fusion image BF is generated according to the base layer B1 of the infrared image, the base layer B2 of the visible light image, the first decision map P1 and the second decision map P2, i.e. BF = B1 x P1 + B2 x P2.

[0156] Then, the detail layers of the source image are fused based on visual fidelity, the visual fidelity of the detail layers D1 and D2 of the infrared image and the visible light image is calculated, and compared with a visual fidelity threshold to determine a first weight map PD1 corresponding to a first type of visual fidelity and a second weight map PD2 corresponding to a second type of visual fidelity; the process of fusing the detail layer corresponding to the first type of visual fidelity can be summarized as follows: the detail layer of the infrared image is decomposed into basic elements, and a saliency estimation value SD1 of the detail layer of the infrared image is obtained based on saliency estimation, and the saliency estimation values of the elements are compared to obtain a decision map PSD of the detail layer of the infrared image, and a first type of visual fidelity texture T1 corresponding to the first type of visual fidelity can be obtained from the decision map PSD of the detail layer of the infrared image, the detail layer D1 of the infrared image and the detail layer D2 of the visible light image, i.e. T1 = PSD x D1 + (1- PSD) D2; the process of fusing the detail layer corresponding to the second type of visual fidelity can be summarized as follows: the absolute value |D1| of the detail layer of the infrared image is compared with the absolute value |D2| of the detail layer of the visible light image in a manner of maximum absolute value to determine an initial decision map M of the second type of visual fidelity, and the initial decision map and the visible light image are guided filtered according to a filtering scale and a blur degree to obtain a second type of visual fidelity texture T2 corresponding to the second type of visual fidelity, i.e. T2 = GuidedFiltering (M, I2, r, e), and a second fusion image TF is generated according to the first type of visual fidelity texture T1, the second type of visual fidelity texture T2, the first weight map PD1 and the second weight map PD2, i.e. TF = T1 x PD1 + T2 x PD2.

[0157] Finally, the first fusion image BF and the second fusion image TF are fused to obtain the fusion image F of the source image, that is, F = BF + TF.

[0158] The technical scheme of the embodiment of the present application fuses the detail layer of the source image based on the visual fidelity method, divides the detail layer into the first visual fidelity texture corresponding to the first visual fidelity and the second visual fidelity texture corresponding to the second visual fidelity through the visual fidelity threshold, adopts different fusion methods for different visual fidelity textures, better retains the effective part of the texture, and improves the definition of the infrared image and the visible light image fusion.

[0159] Embodiment four

[0160] Figure 5 A structural schematic diagram of an image fusion device provided by the fourth embodiment of the present application is shown in the figure. The present embodiment can be applied to the case of fusing the infrared image and the visible light image. As shown in the figure, the specific structure of the device includes: Figure 5

[0161] The source image decomposition module 21 is configured to decompose the source image into a base layer and a detail layer through a bilateral texture filter, and the source image includes an infrared image and a visible light image.

[0162] The base layer fusion module 22 is configured to fuse the base layer of the source image based on a saliency detection algorithm to obtain a first fusion image.

[0163] The detail layer fusion module 23 is configured to fuse the detail layer of the source image based on visual fidelity to obtain a second fusion image.

[0164] The source image fusion module 24 is configured to fuse the first fusion image and the second fusion image to obtain a fusion image of the source image.

[0165] The image fusion device provided by the present embodiment first decomposes the source image into a base layer and a detail layer through a bilateral texture filter by the source image decomposition module 21, and the source image includes an infrared image and a visible light image. Secondly, the base layer fusion module 22 fuses the base layer of the source image based on a saliency detection algorithm to obtain a first fusion image. Then, the detail layer fusion module 23 fuses the detail layer of the source image based on visual fidelity to obtain a second fusion image. Finally, the source image fusion module 24 fuses the first fusion image and the second fusion image to obtain a fusion image of the source image.

[0166] Further, the base layer fusion module 22 is specifically configured to:

[0167] extract a first saliency map of the base layer of the infrared image and a second saliency map of the base layer of the visible light image;

[0168] ​The first saliency map and the second saliency map are compared to obtain a first decision map and a second decision map.

[0169] The first fusion image is generated according to the base layer of the infrared image, the base layer of the visible light image, the first decision map and the second decision map.

[0170] Further, the detail layer fusion module 23 is specifically configured to:

[0171] The detail layer of the infrared image and the detail layer of the visible light image are correspondingly decomposed into a plurality of image blocks, and each image block of the detail layer of the infrared image and a corresponding image block of the detail layer of the visible light image are taken as an image block group;

[0172] The visual fidelity of each image block group is calculated.

[0173] According to the visual fidelity of each image block group and a visual fidelity threshold, a first weight map corresponding to a first type of visual fidelity and a second weight map corresponding to a second type of visual fidelity are determined, wherein the first type of visual fidelity is higher than the second type of visual fidelity.

[0174] The second fusion image is generated according to the first type of visual fidelity texture corresponding to the first type of visual fidelity, the second type of visual fidelity texture corresponding to the second type of visual fidelity, the first weight map and the second weight map. Wherein the first type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image, and the second type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image.

[0175] Further, in the detail layer fusion module 23, before calculating the visual fidelity of each image block group, further comprising:

[0176] The Gaussian pyramid transformation is performed on each image block group.

[0177] Further, the detail layer fusion module 23 is specifically configured to:

[0178] The first type of visual fidelity texture corresponding to the first type of visual fidelity is determined based on the saliency estimation.

[0179] The second type of visual fidelity texture corresponding to the second type of visual fidelity is determined based on the guided filtering.

[0180] Further, the detail layer fusion module 23 is specifically configured to:

[0181] The saliency estimation value of each element in the image block group of the first type of visual fidelity is determined according to the element consistency and the regularization result of element allocation.

[0182] The saliency estimation value of each element is compared to obtain the first type of visual fidelity texture corresponding to the first type of visual fidelity.

[0183] Further, the detail layer fusion module 23 is specifically used for:

[0184] According to the comparison result of the detail layer of the infrared image corresponding to the second type of visual fidelity image block and the detail layer of the visible light image, an initial decision map of the second type of visual fidelity is determined.

[0185] According to the filtering scale and the blur degree, the initial decision map and the visible light image are guided filtering to obtain the second type of visual fidelity corresponding to the second type of visual fidelity texture.

[0186] The image fusion device provided in the embodiments of the present application can execute the image fusion method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0187] Embodiment five

[0188] Figure 6 A structural schematic diagram of an image processing device 10 that can be used to implement embodiments of the present application is shown. The image processing device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document.

[0189] As shown in Figure 6 The image processing device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the image processing device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0190] The plurality of components in the image processing device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the image processing device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0191] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the image fusion method.

[0192] In some embodiments, the image fusion method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the image processing device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the image fusion method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the image fusion method by any other appropriate means, such as by means of firmware.

[0193] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0194] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, or entirely on a remote machine or server.

[0195] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal form, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] To provide for interaction with a user, the systems and techniques described here can be implemented on an image processing device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the image processing device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0197] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0198] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0199] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited in this regard.

[0200] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, and / or alternatives come within the scope of the present disclosure as recited by the claims.

Claims

1. An image fusion method, characterized by, The method comprises the following steps: decomposing a source image into a base layer and a detail layer by a bilateral texture filter, the source image comprising an infrared image and a visible light image; fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fused image; fusing the detail layer of the source image based on visual fidelity to obtain a second fused image; fusing the first fused image and the second fused image to obtain a fused image of the source image; wherein the fusing the detail layer of the source image based on visual fidelity to obtain a second fused image comprises: correspondingly decomposing the detail layer of the infrared image and the detail layer of the visible light image into a plurality of image blocks, each image block of the detail layer of the infrared image and the corresponding image block of the detail layer of the visible light image being an image block group; calculating the visual fidelity of each image block group; determining a first weight map corresponding to a first type of visual fidelity and a second weight map corresponding to a second type of visual fidelity according to the visual fidelity of each image block group and a visual fidelity threshold, wherein the first type of visual fidelity is higher than the second type of visual fidelity; generating the second fused image according to a first type of visual fidelity texture corresponding to the first type of visual fidelity, a second type of visual fidelity texture corresponding to the second type of visual fidelity, the first weight map and the second weight map; wherein the first type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image, and the second type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image.

2. The method of claim 1, wherein, The fusing the base layer of the source image based on a saliency detection algorithm to obtain a first fused image comprises: extracting a first saliency map of the base layer of the infrared image and a second saliency map of the base layer of the visible light image; comparing the first saliency map and the second saliency map to obtain a first decision map and a second decision map; generating the first fused image according to the base layer of the infrared image, the base layer of the visible light image, the first decision map and the second decision map.

3. The method of claim 1, wherein, Before calculating the visual fidelity of each image block group, the method further comprises: performing Gaussian pyramid transformation on each image block group.

4. The method of claim 1, wherein, The method further comprises: determining a first type of visual fidelity texture corresponding to the first type of visual fidelity based on saliency estimation; determining a second type of visual fidelity texture corresponding to the second type of visual fidelity based on guided filtering.

5. The method of claim 4, wherein, The determining a first type of visual fidelity texture corresponding to the first type of visual fidelity based on saliency estimation comprises: determining a saliency estimation value of each element in the image block group of the first type of visual fidelity according to the regularization result of element consistency and element allocation; comparing the saliency estimation values of the elements to obtain the first type of visual fidelity texture corresponding to the first type of visual fidelity.

6. The method of claim 4, wherein, The determining a second type of visual fidelity texture corresponding to the second type of visual fidelity based on guided filtering comprises: determine an initial decision map of the second type of visual fidelity according to a comparison result of the detail layer of the infrared image and the detail layer of the visible light image corresponding to the second type of visual fidelity image block group; perform guided filtering on the initial decision map and the visible light image according to a filtering scale and a blur degree to obtain a second type of visual fidelity texture corresponding to the second type of visual fidelity.

7. An image fusion apparatus characterized by comprising: The method comprises the following steps: a source image decomposition module is configured to decompose a source image into a base layer and a detail layer by using a bilateral texture filter, the source image comprising an infrared image and a visible light image; a base layer fusion module is configured to fuse the base layer of the source image based on a saliency detection algorithm to obtain a first fused image; a detail layer fusion module is configured to fuse the detail layer of the source image based on visual fidelity to obtain a second fused image; a source image fusion module is configured to fuse the first fused image and the second fused image to obtain a fused image of the source image; wherein the detail layer fusion module is specifically configured to: decompose the detail layer of the infrared image and the detail layer of the visible light image into a plurality of image block groups, each image block of the detail layer of the infrared image and the corresponding image block of the detail layer of the visible light image serving as an image block group; calculate the visual fidelity of each image block group; determine a first weight map corresponding to a first type of visual fidelity and a second weight map corresponding to a second type of visual fidelity according to the visual fidelity of each image block group and a visual fidelity threshold, wherein the first type of visual fidelity is higher than the second type of visual fidelity; generate the second fused image according to a first type of visual fidelity texture corresponding to the first type of visual fidelity, a second type of visual fidelity texture corresponding to the second type of visual fidelity, the first weight map and the second weight map; wherein the first type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image, and the second type of visual fidelity texture corresponds to the detail layer of the infrared image and the detail layer of the visible light image.

8. An image processing apparatus characterized by comprising: The method comprises the following steps: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image fusion method according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the image fusion method according to any one of claims 1-6.

Images