Image fusion method, device, equipment and storage medium

Abstract

The present application provides an image fusion method, apparatus, device and storage medium, the method comprising: acquiring an infrared image and a visible light image of a target scene that are collected synchronously; extracting a brightness channel component of the visible light image to obtain a corresponding brightness component image; decomposing the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively; fusing the base layer images of the infrared image and the brightness component image respectively to obtain a base layer fused image; fusing the plurality of detail layer images of the infrared image and the brightness component image respectively to obtain a plurality of corresponding detail layer fused images respectively; superimposing the base layer fused image and the plurality of detail layer fused images to obtain a brightness component fused image, and merging the brightness component fused image with other channel components of the visible light image to obtain a fused image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device, equipment and storage medium. Background Art

[0002] Infrared detectors mainly rely on the thermal radiation of the object itself for imaging. They are not easily affected by the environment such as climate and light, have strong anti-interference ability, and can more accurately describe the thermal radiation information of the target, and can well capture hidden heat source targets. However, the imaging quality of infrared images is affected by factors such as the thermal radiation characteristics of the scene, the system operating wavelength, the transmission distance, and atmospheric attenuation, resulting in low contrast in infrared images and poor reflection of target details. Visible light images are reflective images with many high-frequency components, high image resolution, and more comprehensive description of the details of the scene. However, the imaging process of visible light images is easily affected by environmental and lighting conditions.

[0003] In order to combine the advantages of both infrared images and visible light images, the industry has gradually proposed to fuse infrared images with visible light images under low light conditions through image fusion technology, so that the fused image can make up for the shortcomings of a single image. However, the currently known image fusion methods are prone to unclear description of image structure information, blurred image details, and poor visual effects after fusing infrared images and visible light images. Summary of the invention

[0004] In view of this, the present application provides an image fusion method, apparatus, device and computer storage medium, which can increase the amount of information of the fused image, improve the quality of the fused image, and enhance the visual effect.

[0005] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image fusion method, comprising:

[0007] Acquire infrared images and visible light images of the target scene collected synchronously;

[0008] Extract the brightness channel component of the visible light image to obtain the corresponding brightness component image;

[0009] Decomposing the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively;

[0010] Fusing the infrared image and the base layer images of the brightness component image to obtain a base layer fused image;

[0011] The infrared image and the brightness component image are fused with their respective multiple detail layer images to obtain corresponding multiple detail layer fused images;

[0012] The base layer fusion image and multiple detail layer fusion images are superimposed to obtain a brightness component fusion image, and the brightness component fusion image and other channel components of the visible light image are merged to obtain a fusion image.

[0013] In a second aspect, an embodiment of the present application provides an image fusion device, including:

[0014] An image acquisition module, used to acquire infrared images and visible light images of a target scene collected synchronously;

[0015] An image acquisition module is used to extract the brightness channel component of the visible light image and obtain the corresponding brightness component image;

[0016] An image decomposition module, used to decompose the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively;

[0017] A base layer image fusion module, used for fusing the infrared image and the base layer images of the brightness component image to obtain a base layer fused image;

[0018] A detail layer image fusion module is used to fuse the multiple detail layer images of the infrared image and the brightness component image respectively to obtain multiple corresponding detail layer fusion images;

[0019] The fused image obtaining module is used to superimpose the base layer fused image and multiple detail layer fused images to obtain a brightness component fused image, and to merge the brightness component fused image with other channel components of the visible light image to obtain a fused image.

[0020] In a third aspect, an embodiment of the present application provides an image recognition device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, the image fusion method as described in any embodiment of the present application is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a controller, the image fusion method described in any embodiment of the present application is implemented.

[0022] The image fusion method, device, equipment and computer storage medium provided by the above-mentioned embodiments of the present application, the image fusion method obtains the infrared image and visible light image of the target scene collected synchronously, extracts the brightness channel component of the visible light image, obtains the corresponding brightness component image, decomposes the infrared image and the brightness component image respectively, and obtains their respective base layer images and multiple detail layer images. The base layer image of the infrared image and the brightness component image are fused to obtain a base layer fused image, and the multiple detail layer images of the infrared image and the brightness component image are fused respectively to obtain multiple corresponding detail layer fused images, and finally the base layer fused image and all the detail layer fused images are superimposed to obtain a brightness component fused image, and the brightness component fused image is combined with other channel components to obtain a fused image of the target scene. In this way, by extracting the color-independent brightness component image from the visible light image, and decomposing the brightness component image and the infrared image, the corresponding base layer image and multiple detail layer images are obtained, and the brightness component image is fused with the detail layer image and the base layer image of the infrared image respectively to obtain a brightness component fused image that retains the information of the source image as much as possible, reducing the interference of image color during image fusion, and then merging the brightness component fused image with the original other components to obtain a fused image of the target scene. The fused image can inherit the information of the source image as much as possible, increase the amount of information of the fused image, improve the quality of the fused image, and enhance the contrast of the fused image to achieve a good visual effect. The fused image better combines the advantages of the infrared image and the visible light image, and overcomes the defects of the single images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the architecture of an image fusion system provided in an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram showing a flow chart of an image fusion method provided in an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram showing a flow chart of an image decomposition method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram showing a correspondence between a detail layer image and neurons in an adaptive pulse coupled neural network provided by an embodiment of the present application;

[0027] Figure 5 A simplified model schematic diagram of a single neuron of an adaptive pulse coupled neural network provided in an embodiment of the present application is shown;

[0028] Figure 6 An infrared image provided by an embodiment of the present application is shown;

[0029] Figure 7 A visible light image provided by an embodiment of the present application is shown;

[0030] Figure 8 A schematic diagram showing a processing result of an image fusion method provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram showing a processing result of an ADF method provided by the prior art;

[0032] Fig.10 A schematic diagram showing a processing result of a FPDE method provided by the prior art;

[0033] Fig.11 A schematic diagram showing a GFF method processing result provided by the prior art;

[0034] Fig.12 A schematic diagram showing a processing result of a GVT method provided by the prior art;

[0035] Fig.13 A schematic diagram showing a processing result of a GRAD method provided by the prior art;

[0036] Fig.14 A schematic diagram showing a processing result of a NSCT method provided by the prior art;

[0037] Fig.15 A schematic diagram showing a comparison of edge strength provided by an embodiment of the present application is shown;

[0038] Fig.16 A schematic diagram showing a comparison of variances provided in an embodiment of the present application is shown;

[0039] Fig.17 A schematic diagram showing a comparison of mutual information provided by an embodiment of the present application is shown;

[0040] Fig.18 A schematic diagram showing a comparison of spatial frequencies provided by an embodiment of the present application is shown;

[0041] Fig.19 A comparative schematic diagram of an average gradient provided in an embodiment of the present application is shown;

[0042] Fig. 20 A schematic diagram showing a comparison of information entropy provided in an embodiment of the present application is shown;

[0043] Fig.21 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application is shown;

[0044] Fig. 22 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] The technical solution of the present application is further elaborated in detail below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0047] In the following description, the expression "some embodiments" is involved, which describes a subset of all possible embodiments. It should be noted that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0048] In the following description, the terms "first, second, third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second, third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] Image fusion is an image enhancement technology that aims to merge images obtained by different types of sensors to generate robust or informative images that can facilitate subsequent processing or assist in decision-making, thereby improving the utilization of image information, improving the accuracy and reliability of computer interpretation, and enhancing the spatial and spectral resolution of the original image, which is beneficial to applications such as target recognition and monitoring.

[0051] The fusion result image has both the edge features of the visible light image and the temperature distribution information of the target object or scene in the infrared image, realizing the effective combination of the characteristics of the visible light image and the infrared image, and can more comprehensively and accurately express the information contained in the target object or scene, and can well realize practical applications such as target recognition and monitoring.

[0052] RGB (Red-Green-Blue) color space can refer to the three basic colors of red, green and blue, which are superimposed to varying degrees to produce rich and wide colors.

[0053] The HSI (Hue-Saturation-Intensity) color space can be used to describe the color of an object using hue, saturation, and brightness. Hue can represent the frequency of a color. Saturation can represent the depth of a color. Brightness can represent the degree of brightness. In the HSI color space, the I component has nothing to do with the color information of the image, while the H component and the S component are closely related to the way people perceive color.

[0054] The singular value decomposition of a matrix can be simply described as follows: if the matrix C∈R m×n , then there exists a decomposition such that C = USV * , where U is an m×m unitary matrix, S is an m×n non-negative real diagonal matrix, and V * It is an n×n unitary matrix, and the elements on the diagonal of S are the singular values ​​of C.

[0055] In the process of image fusion, PCNN (Pulse Coupled Neural Network) is a commonly used method. PCNN is proposed to simulate the process of visual cortical cells in the brains of mammals such as cats and monkeys processing visual signals. PCNN can be used in many fields of image processing. Using PCNN to process image information can retain more detailed information and improve target recognition rate. However, a large number of parameters in pulse coupled neural networks need to be set manually, and they are not adaptive.

[0056] The working process of pulse coupled neural network can be described as follows: m*n neurons of a two-dimensional PCNN network correspond to m*n pixels of a two-dimensional input image respectively. In the first iteration, the internal activity item of the neuron is equal to the external stimulus. If it is greater than the threshold, the neuron output is 1, indicating natural activation. At this time, its threshold will increase sharply and then decay exponentially with time. In each iteration thereafter, the activated neuron stimulates the adjacent neuron through the connection with its adjacent neurons. If the internal activity item of the adjacent neuron is greater than the threshold, it is captured and activated. If the adjacent neuron has a similar intensity to the pixel corresponding to the neuron activated in the previous iteration, the adjacent neuron is easily captured and activated, otherwise it cannot be captured and activated. Therefore, the natural activation of a neuron will trigger the collective activation of similar neurons around it, generating a pulsating output sequence, and they form a neuron cluster, which can realize image segmentation, denoising and recognition.

[0057] An infrared image refers to an image formed by collecting infrared light passing through an object within the target field of view and converting the infrared light signal into an image signal. A visible light image refers to an image formed by collecting visible light reflected by the surface of an object within the target field of view and converting the visible light signal into an image signal. The imaging signal of the infrared image and the imaging signal of the visible light image come from different modalities and can provide target scene information from different aspects. Visible light images have higher spatial resolution and more details and light and dark contrast. However, they are easily disturbed by harsh conditions. However, infrared images can resist these conditions. Therefore, infrared images and visible light images have complementary characteristics. Based on this, in an embodiment of the present application, infrared images and visible light images are used for image fusion, so that a robust and information-rich fused image can be obtained.

[0058] Figure 1 FIG. 1 is a schematic diagram showing the architecture of an image fusion system provided by an embodiment of the present application. Figure 1 As shown, the architecture of the image fusion system may include an infrared image acquisition device 11, a visible light image acquisition device 12, and an image fusion device 13. The infrared image acquisition device 11 is used to acquire an infrared image of a target scene. The visible light image acquisition device 12 is used to acquire a visible light image of a target scene. Here, the infrared image acquisition device 11 may be an infrared camera device, and the visible light image acquisition device 12 may be a digital color camera device.

[0059] The image fusion device 13 can be used to obtain the infrared image and the visible light image synchronously captured by the infrared image acquisition device 11 and the visible light image acquisition device 12, and use the image fusion method provided in the embodiment of the present application to fuse the infrared image and the visible light image to obtain a high-quality fused image.

[0060] In some embodiments, Figure 1As shown, the image fusion device 13 can be used to obtain the infrared image and the visible light image synchronously collected by the infrared image acquisition device 11 and the visible light image acquisition device 12, and align the infrared image with the visible light image; perform color model conversion on the registered visible light image to obtain the brightness component image corresponding to the visible light image; decompose the registered infrared image and the brightness component image through sub-window variance filtering to obtain their respective base layer images and N detail layer images, and fuse the two base layer images through a singular value decomposition fusion algorithm to obtain a base layer fused image; fuse the infrared image and the multiple matching detail layer images of the brightness component image through an adaptive PCNN (Pulse Coupled Neural Network) to obtain a fused image of each detail layer; fuse the base layer fused image and the multiple detail layer fused images to obtain a brightness channel fused image; merge the brightness channel fused image with other channel component images of the visible light image and perform color model conversion to obtain a color fused image.

[0061] In the above embodiment, the infrared image acquisition device and the visible light image acquisition device respectively acquire infrared images and visible light images, and transmit the images to the image fusion device. The image fusion device performs multi-scale decomposition on the brightness component image of the visible light image and the infrared image to obtain respective base layer images and detail layer images, and uses different image fusion methods to fuse the base layer image and the detail layer image respectively to obtain a brightness fusion image, and finally merges the brightness fusion image with other components, so that the fused image can retain the detail information and structural information of the source image as much as possible, improve the contrast of the fused image, and achieve good visual effects.

[0062] Figure 2 A schematic diagram showing a flow chart of an image fusion method provided in an embodiment of the present application is shown, which can be applied to Figure 1 The image fusion device 13 is shown. The image fusion method may include but is not limited to S21, S22, S23, S24, S25 and S26, which are specifically described as follows:

[0063] S21, acquiring the infrared image and visible light image of the target scene collected synchronously.

[0064] S22, extracting the brightness channel component of the visible light image to obtain a corresponding brightness component image;

[0065] S23, respectively decomposing the infrared image and the brightness component image to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image.

[0066] S24, fusing the infrared image and the base layer images of the brightness component image to obtain a base layer fused image.

[0067] S25, fusing the multiple detail layer images of the infrared image and the brightness component image respectively to obtain corresponding multiple detail layer fused images.

[0068] S26, superimposing the base layer fusion image and the plurality of detail layer fusion images to obtain a brightness component fusion image, and merging the brightness component fusion image with other channel components of the visible light image to obtain a fusion image.

[0069] In the above embodiment, a brightness component image that is independent of color is extracted from a visible light image, and the brightness component image and the infrared image are decomposed to obtain respective corresponding base layer images and multiple detail layer images. The brightness component image is fused with the detail layer image and the base layer image of the infrared image respectively to obtain a brightness component fused image that retains the information of the source image as much as possible, thereby reducing the interference of image color during image fusion. The brightness component fused image is then merged with the other original components to obtain a fused image of the target scene. The fused image can inherit the information of the source image as much as possible, increase the amount of information in the fused image, improve the quality of the fused image, and enhance the contrast of the fused image, thereby achieving a good visual effect. The fused image better combines the respective advantages of the infrared image and the visible light image, and overcomes the respective defects of the single images.

[0070] Below Figure 2 The implementation of each step in the embodiment is further explained as follows.

[0071] In step S21, the target scene may be a scene containing any scene or person. The infrared image of the target scene may be obtained by Figure 1 The infrared image acquisition device 11 shown in the figure is used to acquire the visible light image of the target scene. Figure 1 The visible light image acquisition device 12 shown is used for acquisition.

[0072] In an embodiment of the present application, the image fusion device can obtain an infrared image of the target scene from an infrared image acquisition device, and obtain a visible light image of the target scene from a visible light image acquisition device. Alternatively, the infrared image acquisition device uploads the infrared image of the target scene to a network, and the visible light image acquisition device uploads the visible light image of the target scene to a network, and the image fusion device can obtain the infrared image and visible light image of the target scene from the network. In an embodiment of the present application, the method of obtaining the infrared image and the visible light image is not limited.

[0073] In S21, the infrared image of the target scene and the visible light image of the target scene are both images describing the target scene at a certain time point, and are acquired at the same time point by different image acquisition devices.

[0074] In step S22, the brightness channel component may be a component representing the brightness of the image. In a visible light image, the brightness channel component is irrelevant to the color information of the image.

[0075] In the embodiment of the present application, when the visible light image is an HSI color space image, the image fusion device directly extracts the brightness channel component of the visible light image, thereby obtaining a brightness component image corresponding to the visible light image. When the visible light image is an image in a color space other than the HSI color space, the image fusion device converts the color space of the visible light image into the HSI color space, and then extracts the brightness channel component of the converted visible light image, thereby obtaining a brightness component image corresponding to the visible light image.

[0076] In step S23, decomposition may refer to image decomposition based on multi-scale transformation. In an embodiment of the present application, the image fusion device decomposes the image into a base layer image and a detail layer image through an image decomposition algorithm. The base layer image may include a low-frequency component of the image, reflecting the intensity change of the image on a large scale. The detail layer image contains a high-frequency component of the image, reflecting the details of the image on a small scale. Here, the image decomposition method may include but is not limited to a spatial domain decomposition method and a transform domain decomposition method.

[0077] In S23, the infrared image and the brightness component image are decomposed respectively to obtain the base layer image and multiple detail layer images corresponding to the infrared image and the brightness component image, respectively. It can be understood that the image fusion device decomposes the infrared image based on the image decomposition algorithm to obtain the base layer image and multiple detail layer images corresponding to the infrared image, and the image fusion device decomposes the brightness component image based on the image decomposition algorithm to obtain the base layer image and multiple detail layer images corresponding to the brightness component image. Optionally, the number of detail layer images can be 3 or more.

[0078] In step S24, fusion may refer to image fusion. In the embodiment of the present application, the image fusion device fuses the base layer image of the infrared image with the base layer image of the brightness component image through an image fusion algorithm to obtain a base layer fused image of the target scene. Here, the fusion algorithm of the base layer image may include but is not limited to a weighted fusion algorithm and an average fusion algorithm.

[0079] In step S25, fusion may refer to image fusion. In an embodiment of the present application, the image fusion device fuses each layer of detail layer image of the infrared image with the corresponding layer of detail layer image of the brightness component image based on an image fusion algorithm, thereby obtaining each layer of detail layer fusion image. For example, the image fusion device fuses the first layer of detail layer image of the infrared image with the first layer of detail layer image of the brightness component image based on an image fusion algorithm to obtain the first layer of detail layer fusion image, and so on to obtain detail layer fusion images of multiple layers of detail layers. Here, the fusion method of the detail layer images can adopt image fusion methods such as neural networks and deep learning.

[0080] In step S26, the brightness component fused image may be a fused image of a single channel component. Superimposing the base layer fused image and the detail layer fused image may be understood as the image fusion device superimposing the base layer fused image and the plurality of detail layer fused images by addition to obtain the brightness component fused image.

[0081] In S26, the other channel components of the visible light image may be other multiple channel components retained after the brightness channel component is extracted from the visible light image in S22. Here, the other channel components of the visible light image may include but are not limited to the hue channel component and the saturation channel component of the visible light image.

[0082] In S26, the fusion image is obtained by merging the brightness component fusion image with other channel components of the visible light image. This can be understood as merging the brightness component corresponding to the brightness component fusion image with other channel components to obtain the fusion image.

[0083] In order to further improve the fusion effect and obtain the brightness component of the image, in some embodiments, before extracting the brightness channel component of the visible light image in S22 to obtain the corresponding brightness component image, the image fusion method further includes:

[0084] Register infrared images and visible light images;

[0085] The registered colored visible light image is transformed into a color space to obtain a visible light image in the HSI color space.

[0086] Here, registration refers to image registration. Image registration can be a preprocessing step for image fusion. Image registration can refer to mapping one image to another image by finding a spatial transformation so that images of the same scene from different perspectives can be geometrically aligned. Image registration methods can include but are not limited to image region-based registration methods and image feature-based registration methods. In this way, after the infrared image and the visible light image of the target scene are registered, a better fusion effect can be achieved.

[0087] In an embodiment of the present application, the visible light image in the HSI color space may be an image that describes the target scene through an H component, an S component, and an I component. The color space conversion may be a conversion between an HSI color space image and a non-HSI color space image. The image fusion device may perform color space transformation through a color space conversion model to obtain a visible light image in the HSI color space. The colored visible light image is an image in a non-HSI color space. For example, the visible light image is an image in an RGB color space. The image fusion device may perform color space transformation through an RGB-HSI color space conversion model to obtain a visible light image in the HSI color space. In an embodiment of the present application, the RGB-HSI color space conversion model includes multiple conversion models such as a geometric model, a standard model, a segmented model, and a coordinate conversion model. For example, the RGB-HSI color space conversion model may be:

[0088]

[0089] Where I represents the brightness component, H = tan -1 (V1 / V2) represents the hue component, represents the saturation component, R represents the red component, G represents the green component, and B represents the blue component.

[0090] In order to obtain a fused image in a non-HSI color space, in some embodiments, S26, the base layer fused image and the detail layer fused image are superimposed to obtain a luminance component fused image, and the luminance component fused image is combined with other channel components of the visible light image to obtain a fused image, including:

[0091] The base layer fusion image and the detail layer fusion image are superimposed to obtain a brightness component fusion image, and the brightness component fusion image and other channel components of the visible light image are combined to obtain an initial fusion image;

[0092] The initial fused image is converted into a color space to obtain the final fused image.

[0093] Here, the other channel components of the visible light image may be a hue component and a saturation component of the visible light image. The initial fused image may be an HSI color space image. The color space conversion may be a conversion between an HSI color space image and a non-HSI color space image. The final fused image may be a non-HSI color space image. For example, the final fused image may be an RGB color space image.

[0094] It should be noted that the final fused image and the visible light image obtained in S21 are the same type of color space images. For example, if the obtained visible light image is an RGB color space image, the final fused image is also an RGB color space image. This is beneficial to the color display of the fused image.

[0095] In an embodiment of the present application, the initial fused image is converted into a final fused image of the same color space type as the original visible light image through color space conversion. In this way, the source image information can be retained as much as possible, the authenticity of the fused image is improved, and it is beneficial to subsequent detection and target recognition applications.

[0096] Existing image decomposition methods may include multi-scale based image decomposition methods. Multi-scale based image decomposition methods usually use edge-preserving filters to decompose the source image. Edge-preserving filters mainly include bilateral filtering, weighted least squares filtering, guided filtering, and anisotropic diffusion guided filtering. However, when the above edge-preserving filters perform edge-preserving filtering on an image, the gradient information of the edge of the image texture changes, the contrast of the image is reduced, and the details of the subsequent fused image are distorted and the contrast is reduced.

[0097] In order to solve the above problem, in some embodiments, S23, respectively decomposing the infrared image and the brightness component image to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image, respectively, may include:

[0098] The infrared image and the brightness component image are filtered respectively by sub-window variance filtering to obtain the detail layer image and the intermediate layer image corresponding to each other;

[0099] The infrared image and the intermediate layer image of the brightness component image are filtered respectively by sub-window variance filtering to obtain another corresponding detail layer image and another intermediate layer image, and the infrared image and the intermediate layer image of the brightness component image are filtered respectively by sub-window variance filtering until the corresponding last detail layer image and base layer image are obtained.

[0100] Here, the result of the subwindow variance filtering can be a linear mixture of the original image and the image filtered by the box filter, wherein the linear weight is related to the pixel variance of the original image block and the pixel variance of the subwindow of the block. Compared with other edge-preserving filtering methods, this method can preserve the contrast information and edge gradient information of the image to the greatest extent during the image filtering process.

[0101] Specifically, the subwindow variance filter can be described as follows: for a given input image I, the image block I is divided with pixel k as the center k , and then the image block I k Divide into multiple sub-windows of the same size, calculate the variance of the pixels in each sub-window, based on the image block I k The variance of the pixels of the original image block I is determined by the variance of the pixels of its subwindow k The image block I' obtained by sub-window variance filtering kThe retention factor in and the original image block I k After the box filter is smoothed and filtered, the image block I' is obtained by sub-window variance filtering. k The retention factor in .

[0102] For example, for a given input image I, the image is divided into blocks I with pixel k as the center k (The size is r×r

[0103] ), and then the image block I k Divide into 4 sub-windows of the same size A, B, C and D (size is

[0104] ), and then calculate the variance of the pixels in each sub-window And image block I k The variance of the pixels

[0105] Image Block I k The retention factor A k The calculation formula is as follows:

[0106]

[0107] in, ε is a positive parameter.

[0108] Image Block I k Filtered block I' k The calculation formula is as follows:

[0109]

[0110] I' k =A k I k +B k (4)

[0111] Among them, ω k can be the local area (size is r×r) covered by the filter with pixel k as the center, |ω| is the number of pixels in the local area, and p i is the pixel value of the i-th pixel, The item represents block I k The result after smoothing with a box filter.

[0112] In the filtering process, since each pixel has been subjected to the same number of filtering blocks, the value of each pixel i after filtering can be expressed as:

[0113]

[0114]

[0115]

[0116] Among them, ω i It can be the local area of ​​all covering pixel i, and ω is the number of all local areas covering pixel i.

[0117] In the above embodiment of the present application, the image fusion device continuously performs multiple sub-window variance filtering on the original image, thereby obtaining a base layer image and multiple detail layer images. Here, the final filtering result is used as the base layer image, and the differential image of each filtering is used as the detail layer image. In this way, the multi-scale decomposition of the original image is achieved. Among them, the original image can be an infrared image or a brightness component image. It should be noted that in the embodiment of the present application, the number of detail layer images is the same as the number of filtering times. For example, the image fusion device continuously performs L sub-window variance filtering on the original image, thereby obtaining L detail layer images and a base layer image in sequence.

[0118] Taking L times as three times as an example, the original image is filtered three times continuously to illustrate the image decomposition method provided by the embodiment of the present application. Figure 3 FIG. 1 is a flow chart of an image decomposition method provided by an embodiment of the present application. Figure 3 As shown, the image fusion device performs three consecutive filterings through the subwindow variance filter. First, the source image is filtered by the first subwindow variance filter to obtain the intermediate layer image U1 and the detail layer image 1, wherein the detail layer image 1 is the difference image between the original image and the intermediate layer image U1. Then, the intermediate layer image U1 is subjected to the second subwindow variance filtering to obtain the detail layer image 2 and the intermediate layer image U2, wherein the detail layer image 2 is the difference image between the intermediate layer image U1 and the intermediate layer image U2. Finally, the intermediate layer image U3 is subjected to the third subwindow variance filtering to obtain the detail layer image 3 and the base layer image. That is to say, after the filtering operation is performed, the infrared image and the brightness component image of the visible light will each generate three detail layer images and one base layer image.

[0119] In the above embodiment, the subwindow variance filter is a type of edge preserving filter. Compared with other edge preserving filters for multi-scale decomposition of images, the subwindow variance filter can be used to decompose images to maintain the gradient information of image texture, suppress the distortion caused by image decomposition, and maintain the image detail information. In addition, during the decomposition process, the spatial scale and amplitude scale of the image signal can be kept unified, that is, during the decomposition, the signal with a larger amplitude is also regarded as a large-scale signal, and is retained in the large-scale space together with the signal with a larger spatial scale, which is conducive to improving the visual contrast of the subsequent fused image.

[0120] In order to further enhance the ability of the fused image to retain the structural information of the source image, in some embodiments, when the base layer image may include a first base layer image of an infrared image and a second base layer image of a brightness component image, S24, fusing the base layer images of the infrared image and the brightness component image to obtain a base layer fused image, may include:

[0121] Singular value decomposition is performed on the first base layer image and the second base layer image respectively, and linear weighted fusion is performed on the first base layer image and the second base layer image using their respective singular value vectors to obtain a base layer fused image.

[0122] Here, the singular value vector of each base layer image may include the singular value of each pixel in each base layer image. For example, the singular value vector of the first base layer image may include the singular value of each pixel in the first base layer image. The singular value vector of the second base layer image may include the singular value of each pixel in the second base layer image. It should be noted that the scale of the singular value vector of the base layer image is consistent with the scale of the base layer image. For example, if the size of the base layer image is S*H, the size of the singular value vector of the base layer image is also S*H.

[0123] The "linear weighted fusion" involved in the embodiment of the present application can be understood as the addition of each base layer image after multiplying each weight. Since the singular value vector of an image can reflect most of the structural information of the image. After stripping the singular value vector from the image, most of the structural information of the image can be shielded. Therefore, in the embodiment of the present application, the two base layer images are linearly weighted fused using the singular value vector of the first base layer image and the singular value vector of the second base layer, and a fused image that inherits more structural information of the two base layer images can be obtained.

[0124] In the above embodiment, since the singular values ​​of an image can reflect the properties of the image structural information, when the base layer images are fused, the base layer images are decomposed into singular values ​​and all the base layer images are linearly weighted fused based on the sizes of their respective singular values. This helps the base layer fused image inherit the structural information in the source image and retain the structural information of the source image as much as possible.

[0125] Since at least one side of the image boundary pixels is empty, the boundary pixels cannot be processed during the image processing. Therefore, before performing singular value decomposition on the first base layer image and the second base layer image respectively, the image fusion method may further include: performing image filling on the boundary of the base layer image.

[0126] Here, performing image filling on the edge of the base layer image can be understood as filling the boundary of the base layer image on the basis of the base layer image. In the embodiment of the present application, the image filling method can be an existing image filling method, including but not limited to a symmetric filling method. In this way, by filling the image boundary, the image boundary pixels can be processed.

[0127] In some embodiments, performing singular value decomposition on the first base layer image and the second base layer image respectively, and performing linear weighted fusion on the first base layer image and the second base layer image using singular value vectors to obtain a base layer fused image may include:

[0128] Performing singular value decomposition on the first base layer image and the second base layer image respectively to obtain respective singular value vectors in the first base layer image and the second base layer image;

[0129] Based on the respective singular value vectors, a first pixel weight map of the first base layer image and a second pixel weight map of the second base layer image are obtained;

[0130] Based on the first pixel weight map and the second pixel weight map, the first base layer image and the second base layer image are linearly fused to obtain a base layer fused image.

[0131] Here, the first pixel weight map may include a first pixel weight for each pixel in the first base layer image. The second pixel weight map may include a second pixel weight for each pixel in the second base layer image. The sum of the first pixel weight and the second pixel weight corresponding to the pixels at the same position in the first base layer image and the second base layer image is 1.

[0132] In an embodiment of the present application, the first pixel weight may be determined according to the ratio of the singular value of the pixel of the first base layer image to the sum of the singular values ​​of the pixels of the first base layer image and the second base layer image. The second pixel weight may be determined according to the ratio of the singular value of the pixel of the second base layer image to the sum of the singular values ​​of the pixels of the first base layer image and the second base layer image. Alternatively, the second pixel weight may be the difference between 1 and the first pixel weight.

[0133] Specifically, the singular value of the (x, y)th pixel in the first base layer image is IR SVD (x, y), the singular value of the (x, y)th pixel in the second base layer image is Vis SVD (x,y). The first pixel weight IR of the (x,y)th pixel in the first base layer image Base_Weightmap (x,y) can be expressed as:

[0134]

[0135] The second pixel weight Vis of the (x,y)th pixel in the second base layer image Base_Weightmap (x,y) can be expressed as:

[0136]

[0137] Alternatively, Vis BaseWeightmap (x,y)=1-IR BaseWeightmap (x,y) (10)

[0138] Here, the first pixel weight value of each pixel in the first base layer image can be obtained based on Formula 8, thereby obtaining a first pixel weight map corresponding to the first base layer image. The second pixel weight value of each pixel in the second base layer image can be obtained based on Formula 9 or Formula 10, thereby obtaining a second pixel weight map corresponding to the second base layer image.

[0139] In an embodiment of the present application, based on the first pixel weight map and the second pixel weight map, the first base layer image and the second base layer image are linearly fused to obtain a base layer fused image. It can be understood that the pixel value of the (x, y)th pixel in the base layer fused image can be the sum of the pixel value of the (x, y)th pixel in the first base layer image and the product of the first pixel weight and the pixel value of the (x, y)th pixel in the second base layer image and the product of the second pixel weight.

[0140] Here, the pixel value of the (x, y)th pixel in the base layer fusion image is Fusion Base (x,y) can be expressed as:

[0141] Fusion Base (x,y)=Vis BaseWeightmap (x,y)*Vis Base (x,y)+IR BaseWeightmap (x,y)*IR Base (x,y) (11)

[0142] Among them, Vis Base (x, y) is the pixel value of the (x, y)th pixel in the second base layer image, IR Base (x,y) The pixel value of the (x,y)th pixel in the second base layer image.

[0143] In the above embodiment, the image fusion device can obtain the singular value vector of the first base layer image by performing singular value decomposition on the first base layer image, and can obtain the singular value vector of the second base layer image by performing singular value decomposition on the second base layer image, and based on the singular value vector of the first base layer image and the singular value vector of the second base layer image, determine the first pixel weight map corresponding to the first base layer image and the second pixel weight map corresponding to the second base layer image, and linearly fuse the first base layer image and the second base layer image based on these pixel weights, so as to obtain a base layer fused image. In this way, the fused base layer image can better inherit the structural information of the two base layer images.

[0144] In some embodiments, performing singular value decomposition on the first base layer image and the second base layer image respectively to obtain the singular value vectors of the first base layer image and the second base layer image respectively may include:

[0145] For each pixel in the first base layer image and the second base layer image, perform the following steps:

[0146] Get an M-order neighborhood matrix centered on the pixel, where M is a positive integer greater than 2;

[0147] Perform singular value decomposition on the domain matrix to obtain M singular values;

[0148] The average of the M singular values ​​is taken as the singular value of the corresponding pixel.

[0149] Here, the M-order neighborhood matrix can be an M*M matrix, which can contain M 2 elements. For each pixel in the first base layer image and the second base layer image, the image fusion device forms a neighborhood matrix with each pixel as the center, and performs singular value decomposition on the neighborhood matrix to obtain a non-negative diagonal matrix corresponding to the neighborhood matrix, and the elements on the diagonal of the non-negative diagonal matrix are the singular values ​​of the neighborhood matrix. After averaging the singular values ​​of the neighborhood matrix, the average value of the singular values ​​is used as the singular value of the corresponding pixel. By analogy, the singular value of each pixel in the first base layer image can be calculated to obtain the singular value vector of the first base layer image, and the singular value of each pixel in the second base layer image can be calculated to obtain the singular value vector of the second base layer image.

[0150] Taking the neighborhood matrix C as a 3rd-order matrix centered on the pixel with a pixel value of 76 as an example, the process of obtaining the singular value of the pixel is explained:

[0151]

[0152] Perform singular value decomposition on C:

[0153]

[0154] It can be seen that the singular values ​​of C are 164.2, 63.75 and 7.36. Taking the average of the singular values ​​of C, the singular value corresponding to the center pixel is 78.43.

[0155] In the above embodiment, on the one hand, compared with directly performing singular value decomposition on the entire base layer image, obtaining singular values ​​based on the pixel neighborhood matrix can reduce the calculation difficulty. On the other hand, the average value of the singular values ​​is used to calculate the weight of each pixel when fusing the base layer image, and the structural information within the neighborhood of each pixel of the base layer image is reflected in the weight during fusion, so that the fused base layer image can better inherit the structural information of the source image.

[0156] Based on the characteristics of the detail layer images, a neural network may be used for fusion. In some embodiments, the multiple detail layer images include N layers of infrared detail layer images of the infrared image and N layers of brightness detail layer images of the brightness component image. S25, fusing the multiple detail layer images of the infrared image and the brightness component image respectively to obtain the corresponding multiple detail layer fused images, may include:

[0157] Through an adaptive pulse coupled neural network, N layers of infrared detail layer images and N layers of brightness detail layer images are fused one by one according to the matching relationship of corresponding layers to obtain an N layer of detail layer fused image.

[0158] Here, the adaptive pulse coupled neural network can be an improved pulse coupled neural network. For example, setting some fixed parameters of the pulse coupled neural network to be adjusted according to the input content is conducive to improving the adaptability of the network and improving the fusion accuracy.

[0159] In the embodiment of the present application, N is a positive integer. In order to achieve a better fusion effect and reduce the amount of calculation in the fusion process, N can be 3, optionally. The image fusion device fuses the infrared detail layer image and the brightness detail layer image one by one according to the matching relationship of the corresponding layers. It can be understood that the image fusion device fuses the R-th infrared detail layer image and the R-th brightness detail layer image through an adaptive pulse coupled neural network to obtain the R-th detail layer fused image, where R is a positive integer less than or equal to N, and so on, N layers of detail layer fused images can be obtained.

[0160] In the above embodiment, on the one hand, in the image fusion process, according to the characteristics of the detail layer image, the adaptive PCNN is selected to fuse the detail layer image, which can better retain the image detail information. On the other hand, compared with PCNN, the improved PCNN is more adaptive and can improve the image fusion effect. In addition, for the fusion of the detail layer image, a fusion method different from that of the base layer image can be selected, so that in the image fusion process, according to the respective characteristics of the detail layer image and the base layer image, a suitable fusion method can be selected for fusion, which is further conducive to improving the image fusion quality.

[0161] In some embodiments, the N layers of infrared detail layer images and the N layers of brightness detail layer images are fused one by one according to the matching relationship of the corresponding layers by an adaptive pulse coupled neural network, and before obtaining the N layers of detail layer fused images, the image fusion method may further include:

[0162] Calculate the average gradient value of each pixel in the N layers of infrared detail layer images and the N layers of brightness detail layer images to obtain a first gradient map corresponding to the infrared detail layer image and a second gradient map corresponding to the brightness detail layer image;

[0163] The synaptic link strength coefficient in the adaptive pulse coupled neural network is determined based on the first gradient map and the second gradient map.

[0164] Here, the gradient value of the image can indicate the change trend of the image in different directions. The first gradient map can be a gradient map corresponding to each layer of infrared detail layer image. The first gradient map may include the gradient value of each pixel in the infrared detail layer image. The second gradient map may be a gradient map corresponding to each layer of brightness detail layer image. The second gradient map may include the gradient value of each pixel in the brightness detail layer image. When the detail layer images are fused one by one according to the matching relationship, the network parameters are set by the image gradient value, so that the PCNN can adapt to the image characteristics.

[0165] In some embodiments, the average gradient value of a pixel in the detail layer image can be obtained by the square difference between a plurality of adjacent pixels and the pixel. Specifically, the average gradient value AG(x, y) of a pixel (x, y) in the detail layer image can be:

[0166]

[0167] Where I(x,y) represents the pixel value of pixel (x,y).

[0168] In the above embodiment, in order to calculate the gradient value of the image boundary, the image boundary may be filled with pixels to calculate the gradient value of the image boundary pixels. It should be noted that the image gradient may also be calculated using an existing method for calculating the image gradient.

[0169] Here, the synaptic connection strength coefficient in the adaptive pulse coupled neural network can be the synaptic connection strength coefficient between a neuron at a position and the surrounding neurons. The coupling between different neurons in the visual cortex is different, and the difference is reflected in the different synaptic connection strength coefficients of neurons in PCNN. The synaptic connection strength coefficient should change with the change of image features, so as to characterize the difference in coupling between pixels in different areas of the image.

[0170] In an embodiment of the present application, determining the synaptic connection strength coefficient in an adaptive pulse coupled neural network based on a first gradient map and a second gradient map can be understood as determining the synaptic connection strength coefficient when an infrared detail layer image is input into an adaptive pulse coupled neural network based on the first gradient map, and determining the synaptic connection strength coefficient when a brightness detail layer image is input into an adaptive pulse coupled neural network based on the second gradient map.

[0171] Specifically, the image fusion device can use the gradient value in the first gradient map as the synaptic connection strength coefficient when the infrared detail layer image is input into the adaptive pulse coupled neural network, and use the gradient value in the second gradient map as the synaptic connection strength coefficient when the brightness detail layer image is input into the adaptive pulse coupled neural network.

[0172] In the above embodiment, by setting the synaptic connection strength coefficient of the adaptive PCNN through the image gradient, the synaptic connection strength coefficient can represent the feature information of different areas of the image, thereby improving the adaptability of the PCNN, which is beneficial to improving the detail layer image fusion effect and improving the visual effect of the fused image.

[0173] In order to improve the accuracy of the adaptive pulse coupled neural network, in some embodiments, the image fusion device can normalize the first gradient map and the second gradient map, and use the gradient value of the processed first gradient map and the gradient value of the processed second gradient map as the synaptic connection strength coefficient in the adaptive pulse coupled neural network.

[0174] In order to make the fused detail layer image retain more detail information, the detail layer image fusion can be performed by using a pulse coupled neural network improved based on the image average gradient. In some embodiments, the N layers of infrared detail layer images and the N layers of brightness detail layer images are fused one by one according to the matching relationship of the corresponding layers by using an adaptive pulse coupled neural network to obtain the N layers of detail layer fused image, which may include:

[0175] According to the matching relationship of the corresponding layers, the infrared detail layer images and the brightness detail layer images of the N layers are subjected to data preprocessing respectively to obtain the first absolute value image and the second absolute value image of the corresponding layer;

[0176] The first absolute value image and the second absolute value image of each corresponding layer are input into the adaptive pulse coupled neural network as initial excitations, and iterate according to the set number of times to obtain the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer;

[0177] Based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer, the infrared detail layer image and the brightness detail layer image are fused to obtain the detail layer fusion images of each corresponding layer.

[0178] Here, data preprocessing may be replacing the pixel value of each pixel in the detail image with the absolute value of the original pixel value. The first absolute valued image may be the absolute valued image corresponding to each layer of infrared detail layer image. The first absolute valued image may include the absolute value of each pixel value in each layer of infrared detail layer image. That is, the first absolute valued image is equal in size to the infrared detail layer image, but at the same position, the pixel value of the first absolute valued image is the absolute value of the pixel value of the infrared detail layer image.

[0179] And, the second absolute value image may be an absolute value image corresponding to each layer of brightness detail layer image. The second absolute value image may include the absolute value of each pixel value in each layer of brightness detail layer image. That is to say, the second absolute value image is equal to the brightness detail layer image in size, but at the same position, the pixel value of the second absolute value image is the absolute value of the pixel value of the brightness detail layer image.

[0180] In the embodiment of the present application, the image fusion device can match N layers of infrared detail layer images with N layers of brightness detail layer images according to the matching relationship of the corresponding layers to obtain N pairs of detail layer images to be fused. That is to say, the infrared detail layer image and the brightness detail layer image of a corresponding layer can be the first layer of infrared detail layer image and the first layer of brightness detail layer image, or the second layer of infrared detail layer image and the second layer of brightness detail layer image, and so on.

[0181] Here, the image fusion device performs data preprocessing on the infrared detail layer images and the brightness detail layer images of each corresponding layer respectively, which can be understood as the image fusion device performs data preprocessing on all infrared detail layer images and all brightness detail layer images.

[0182] In the embodiment of the present application, the adaptive pulse coupled neural network may be a pulse coupled neural network improved based on the image average gradient. Figure 4 A schematic diagram showing the correspondence between a detail layer image and neurons in an adaptive pulse coupled neural network provided by an embodiment of the present application is shown. Figure 4As shown, in a single-layer network of the adaptive pulse coupled neural network, the neurons in the network correspond one to one with the pixels of the detail layer image. The adaptive pulse coupled neural network iterates the input image multiple times, and the structure of the adaptive pulse coupled neural network is explained by taking the detail layer image as the input image and the nth iteration as an example. Figure 5 FIG. 1 is a simplified model diagram of a single neuron of an adaptive pulse coupled neural network provided by an embodiment of the present application. Figure 5 As shown, the adaptive pulse coupled neural network may include a receiving domain, a modulation domain, and a pulse generating domain. The adaptive pulse coupled neural network is specifically as follows:

[0183] In the receiving domain,

[0184]

[0185] Among them, L ij [n] represents the nth linearly connected input part of the neuron at position (i, j) in the input image. L Indicates the attenuation constant of the linear connection input section. V L represents the intrinsic voltage of the neuron, and W is a third-order neuron internal connection matrix. Each element in W represents the weight of the impulse response value of the neuron in the neighborhood participating in the connection input of the current neuron, which can be determined by the inverse of the Euclidean distance of the neuron in the neighborhood from the current neuron. (k, l) represents the neuron position index of the neighborhood range covered by W. e represents the natural index.

[0186] In the modulation domain, and

[0187] F ij [n] = S ij (16)

[0188] Among them, F ij [n] is the feedback input part of the nth iteration of the neuron at position (i, j) in the input image. ij is the external excitation input part. As shown in formula (16), in this method, the feedback input part is directly simplified to be equal to the input part of the external excitation. In the embodiment of the present application, S ij It can be the absolute value of the pixel value at position (i, j) in the input image.

[0189] In the pulse generation domain,

[0190] U ij [n] = F ij [n](1+β ij L ij [n]) (17)

[0191]

[0192]

[0193] Among them, U ij Represents the modulation part of the nonlinear connection, that is, the internal activity term of the neuron, which is obtained by multiplying the biased linear connection input part with the feedback input part. ij It can be the synaptic connection strength coefficient between the neuron at the current (i, j) position and the surrounding neurons. In the embodiment of the present application, β ij It can be the gradient value at position (i, j) in the input image. ij [n] represents the dynamic threshold required for the neuron's internal activity term U to stimulate the generation of pulses. θ Represents the dynamic threshold attenuation constant, V θ is the intrinsic voltage of the neuron. kl [n-1] represents the output state of the neuron at the position (k, l) in the neighborhood at the n-1th iteration. ij [n] represents the output state of the neuron at position (i, j) at the nth iteration.

[0194] It should be noted that, in the embodiment of the present application, when the output state of the neuron is 1, it indicates that the excitation is successful or the ignition is successful, and when the output state is 0, it indicates that the neuron is not excited or the ignition is not successful.

[0195] Each element in the "ignition matrix" involved in the embodiment of the present application is the total number of times each pixel of the detail layer image is successfully excited or ignited after multiple iterations of the adaptive pulse coupled neural network. The infrared detail layer ignition matrix can be the ignition matrix output after the infrared detail layer image is subjected to multiple iterations of the adaptive pulse coupled neural network. The brightness detail layer ignition matrix can be the ignition matrix output after the brightness detail layer image is subjected to multiple iterations of the adaptive pulse coupled neural network.

[0196] It should be noted that after multiple iterations of the adaptive pulse coupled neural network, each infrared detail layer image can obtain the corresponding infrared detail layer ignition matrix. After multiple iterations of the adaptive pulse coupled neural network, each brightness detail layer image can obtain the corresponding brightness detail layer ignition matrix.

[0197] In the embodiment of the present application, the set number of times can be set according to the requirements. In order to improve the fusion efficiency, the set number of times can be set between 30 and 60 times.

[0198] In the embodiment of the present application, the infrared detail layer ignition matrix of each infrared detail layer image and the brightness detail layer ignition matrix of each brightness detail layer image can be obtained through an adaptive pulse coupled neural network.

[0199] In order to obtain the H-th layer of detail layer fusion image, where H is a positive integer less than or equal to N, the image fusion device fuses the H-th layer of infrared detail layer image and the H-th layer of brightness detail layer image according to the H-th layer of infrared detail layer ignition matrix and the H-th layer of brightness detail layer ignition matrix, and can obtain the H-th layer of detail layer fusion image. By analogy, the image fusion device can obtain N detail layer fusion images.

[0200] In the embodiment of the present application, during the detail layer image fusion process, a fusion weight can be set according to the infrared detail layer ignition matrix and the brightness detail layer ignition matrix. The infrared detail layer image and the brightness detail layer image are linearly fused according to their respective fusion weights to obtain a detail layer fused image.

[0201] In the above embodiment, the pixel intensity value is used as the input of the adaptive PCNN, and each detail layer image is processed by the adaptive PCNN to obtain the corresponding ignition matrix, and each pixel of the detail layer fused image is determined according to the ignition matrix. In this way, more detail information can be retained by the adaptive PCNN, thereby improving the fusion effect of the detail layer fused image.

[0202] In order to retain as much detail information as possible, in some embodiments, based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer, the infrared detail layer image and the brightness detail layer image are fused to obtain the detail layer fused images of each corresponding layer, which may include:

[0203] Based on the infrared detail layer ignition matrix of each corresponding layer and the brightness detail layer ignition matrix, compare the size of the (i, j)th element in the infrared detail layer ignition matrix and the brightness detail layer ignition matrix; wherein i and j are both integers;

[0204] When the (i, j)th element in the infrared detail layer ignition matrix is ​​greater than the (i, j)th element in the brightness detail layer ignition matrix, determining the (i, j)th pixel value in the infrared detail layer image to be the (i, j)th pixel value in the detail layer fused image of the corresponding layer;

[0205] When the (i, j)th element in the infrared detail layer ignition matrix is ​​less than the (i, j)th element in the brightness detail layer ignition matrix, determining the (i, j)th pixel value in the brightness detail layer image to be the (i, j)th pixel value in the detail layer fused image of the corresponding layer;

[0206] When the (i,j)th element in the infrared detail layer ignition matrix is ​​equal to the (i,j)th element in the brightness detail layer ignition matrix, the mean of the (i,j)th pixel in the brightness detail layer image and the infrared detail layer image is determined to be the (i,j)th pixel value in the detail layer fusion image of the corresponding layer.

[0207] Here, the larger the element value in the ignition matrix, the more activation or ignition times, that is, the more detailed information the pixel in the image corresponding to the element contains. Therefore, the pixel value of the corresponding position of the fused image can be determined by comparing the size of the elements at the same position in the infrared ignition matrix and the brightness ignition matrix of each corresponding layer.

[0208] In the embodiment of the present application, at position (i, j), the infrared ignition matrix of the Tth layer Greater than the brightness ignition matrix of the Tth layer When the Tth infrared detail layer image is The fusion weight of the pixel at is set to 1, and the T-th layer brightness detail layer image The fusion weight of the pixel at position (i, j) is set to 0. That is, the pixel value at position (i, j) in the T-th layer detail fusion image is equal Wherein, T is a positive integer not greater than N.

[0209] Or, the infrared ignition matrix of layer T Less than the brightness ignition matrix of the Tth layer When the Tth infrared detail layer image is The fusion weight of the pixel at is set to 0, and the T-th layer brightness detail layer image The fusion weight of the pixel at position (i, j) is set to 1. That is, the pixel value at position (i, j) in the T-th layer detail fusion image is equal

[0210] Or, the infrared ignition matrix of layer T Equal to the brightness ignition matrix of the Tth layer When the Tth infrared detail layer image is The fusion weight of the pixel at is set to 0.5, and the T-th layer brightness detail layer image The fusion weight of the pixel at position (i, j) is set to 0.5. That is, the pixel value at position (i, j) in the T-th layer detail fusion image is equal and sum.

[0211] Specifically, the pixel value at position (i, j) in the T-th layer detail fusion image It can be expressed as:

[0212]

[0213] In this way, according to the above fusion method, each pixel value of the detail layer fusion image can be obtained, and then the detail layer fusion images of each layer can be obtained.

[0214] In the above embodiment, by comparing the element sizes at the same position of each ignition matrix, the pixel of the fused image is determined to select the pixel of the infrared detail layer image or the pixel of the brightness detail layer image, or the average of the pixel values ​​of the two. In this way, the fused image can inherit the detail information of the original image and improve the effect of the detail layer fused image. Figures 6 to 14 As shown, the image fusion method described in the embodiment of the present application is used to fusion the infrared image ( Figure 6 ) and visible light images ( Figure 7 ) after fusion to obtain the fused image ( Figure 8 ) and the fusion image obtained by the prior art fusion method. Among them, Figures 9 to 14 They refer to the existing image fusion technologies ADF (Anisotropic Diffusion Filter), FPDE (Fourth-order Partial Differential Equations), GFF (Guided Filter), CVT (Curvelet Transform), GRAD (Gradient Pyramid) and NSCT (Non-Subsampled Contourlet Transform). Figure 6 The images shown and Figure 7 The fused image is obtained after image fusion of the images shown. Figure 8 The results shown are compared to Figures 9 to 14 As shown in the processing results, the image is clearer and has a high contrast, and the structural information and detail information of the source image are retained, achieving a good visual effect.

[0215] In addition, Figure 8 The fused image shown is Figures 9 to 14 The image evaluation indicators of each fusion image in the paper are compared, such as Figures 15 to 20 As shown in Figure 1. Image evaluation indicators may include but are not limited to edge strength, variance, mutual information, spatial frequency, average gradient, and information entropy. Figures 15 to 20 As shown, all indicators of this method are higher than those of other fusion methods, that is to say, the image fusion method provided by this application performs better than other fusion methods.

[0216] On the other hand, an embodiment of the present application further provides an image fusion device. Fig.21 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application is shown as follows: Fig.21As shown, the image fusion device includes an image acquisition module 211, which is used to acquire an infrared image and a visible light image of a target scene that are collected synchronously;

[0217] An image acquisition module 212 is used to extract the brightness channel component of the visible light image to obtain a corresponding brightness component image;

[0218] An image decomposition module 213 is used to decompose the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively;

[0219] A base layer image fusion module 214 is used to fuse the infrared image and the base layer images of the brightness component image to obtain a base layer fused image;

[0220] The detail layer image fusion module 215 is used to fuse the multiple detail layer images of the infrared image and the brightness component image respectively to obtain multiple corresponding detail layer fusion images;

[0221] The fused image obtaining module 216 is used to superimpose the base layer fused image and the detail layer fused image to obtain a brightness component fused image, and to combine the brightness component fused image with other channel components of the visible light image to obtain a fused image.

[0222] In some embodiments, the base layer image includes a first base layer image of an infrared image and a second base layer image of a luminance component image, and the base layer image fusion module 214 may include:

[0223] The basic fusion submodule is used to perform singular value decomposition on the first base layer image and the second base layer image respectively, and use their respective singular value vectors to perform linear weighted fusion on the first base layer image and the second base layer image to obtain a base layer fused image.

[0224] In some embodiments, the plurality of detail layer images include N layers of infrared detail layer images of infrared images and N layers of brightness detail layer images of brightness component images, and the detail layer image fusion module 215 may include:

[0225] The detail layer fusion submodule is used to fuse N layers of infrared detail layer images and N layers of brightness detail layer images one by one according to the matching relationship of the corresponding layers through an adaptive pulse coupled neural network to obtain an N layer of detail layer fused image.

[0226] In some embodiments, the basic fusion submodule may include:

[0227] A gradient map obtaining unit is used to calculate the average gradient value of each pixel in the N layers of infrared detail layer images and the N layers of brightness detail layer images to obtain a first gradient map corresponding to the infrared detail layer image and a second gradient map corresponding to the brightness detail layer image;

[0228] The coefficient determination unit is used to determine the synaptic link strength coefficient in the adaptive pulse coupled neural network based on the first gradient map and the second gradient map.

[0229] In some embodiments, the detail layer fusion submodule may include:

[0230] The absolute value image obtaining unit is used to perform data preprocessing on the infrared detail layer images and the brightness detail layer images of each corresponding layer according to the matching relationship of the corresponding layers, so as to obtain the first absolute value image and the second absolute value image of each corresponding layer;

[0231] An ignition matrix obtaining unit is used to input the first absolute value image and the second absolute value image of each corresponding layer as initial excitations into the adaptive pulse coupled neural network, iterate according to a set number of times, and obtain the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer;

[0232] The detail layer fusion image obtaining unit is used to fuse the infrared detail layer image and the brightness detail layer image based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer, and obtain the detail layer fusion images of each corresponding layer respectively.

[0233] In some embodiments, the detail layer fusion image obtaining unit is specifically used to compare the size of the (i, j)th element in the infrared detail layer ignition matrix and the brightness detail layer ignition matrix based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer; wherein i and j are both integers;

[0234] When the (i,j)th element in the infrared detail layer ignition matrix is ​​greater than the (i,j)th element in the brightness detail layer ignition matrix, the (i,j)th pixel value in the infrared detail layer image is determined to be the (i,j)th pixel value in the detail layer fusion image of the corresponding layer;

[0235] When the (i,j)th element in the infrared detail layer ignition matrix is ​​less than the (i,j)th element in the brightness detail layer ignition matrix, the (i,j)th pixel value in the brightness detail layer image is determined to be the (i,j)th pixel value in the detail layer fusion image of the corresponding layer;

[0236] When the (i,j)th element in the infrared detail layer ignition matrix is ​​equal to the (i,j)th element in the brightness detail layer ignition matrix, the mean of the (i,j)th pixel in the brightness detail layer image and the infrared detail layer image is determined to be the (i,j)th pixel value in the detail layer fusion image of the corresponding layer.

[0237] In some embodiments, the image fusion device further includes:

[0238] A registration module, used for registering infrared images and visible light images;

[0239] The color conversion module is used to transform the registered colored visible light image into a color space to obtain a visible light image in the HSI color space.

[0240] The fused image obtaining module 216 is specifically used to superimpose the base layer fused image and the detail layer fused image to obtain a brightness component fused image, and to combine the brightness component fused image with other channel components of the visible light image to obtain an initial fused image;

[0241] The initial fused image is converted into a color space to obtain the final fused image.

[0242] It should be noted that: the image fusion device provided in the above embodiment only uses the division of the above program modules as an example to illustrate the process of implementing the image fusion method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the method steps described above. In addition, the image fusion device provided in the above embodiment and the image fusion method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0243] In another aspect, the present application also provides an image fusion device. Fig. 22 FIG. 1 is a schematic diagram showing the structure of an image fusion device provided in an embodiment of the present application. Fig. 22 As shown, the image fusion device includes a processor 221 and a memory 222, and the memory 222 stores a computer program that can be executed by the processor. When the computer program is executed by the processor, the image fusion method described in any embodiment of the present application is implemented. The image fusion device and the image fusion method provided in the aforementioned embodiment can achieve the same technical effect. To avoid repetition, they are not described here.

[0244] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned image fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium, such as a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a disk or an optical disk, etc.

[0245] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image fusion method, characterized in that: include: Acquire infrared images and visible light images of the target scene collected synchronously; Extracting the brightness channel component of the visible light image to obtain a corresponding brightness component image; Decomposing the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively; Fusing the base layer images of the infrared image and the brightness component image to obtain a base layer fused image; Fusing the plurality of detail layer images of the infrared image and the brightness component image respectively to obtain a plurality of corresponding detail layer fused images; The base layer fusion image and the plurality of detail layer fusion images are superimposed to obtain a brightness component fusion image, and the brightness component fusion image is combined with other channel components of the visible light image to obtain a fusion image; The base layer image includes a first base layer image of the infrared image and a second base layer image of the brightness component image, and fusing the base layer images of the infrared image and the brightness component image to obtain a base layer fused image, including: Performing singular value decomposition on the first base layer image and the second base layer image respectively to obtain respective singular value mean vectors in the first base layer image and the second base layer image; Obtaining a first pixel weight map of the first base layer image and a second pixel weight map of the second base layer image based on the respective singular value mean vectors; Based on the first pixel weight map and the second pixel weight map, linearly fuse the first base layer image and the second base layer image to obtain a base layer fused image; The multiple detail layer images include N layers of infrared detail layer images of the infrared image and N layers of brightness detail layer images of the brightness component image, and the multiple detail layer images of the infrared image and the brightness component image are respectively fused to obtain the corresponding multiple detail layer fused images, including: Calculating the average gradient value of each pixel in the N layers of infrared detail layer images and the N layers of brightness detail layer images to obtain a first gradient map corresponding to the infrared detail layer image and a second gradient map corresponding to the brightness detail layer image; performing normalization processing on the first gradient map and the second gradient map, and determining the synaptic link strength coefficient in the adaptive pulse coupled neural network based on the gradient values ​​of the processed first gradient map and the second gradient map; By using an adaptive pulse coupled neural network, the N layers of infrared detail layer images and the N layers of brightness detail layer images are fused one by one according to the matching relationship of the corresponding layers to obtain an N layer of detail layer fused image; The performing singular value decomposition on the first base layer image and the second base layer image respectively to obtain the respective singular value mean vectors of the first base layer image and the second base layer image includes: For each pixel in the first base layer image and the second base layer image, perform the following steps: Obtain an M-order neighborhood matrix centered on the pixel, where M is a positive integer greater than 2; Performing singular value decomposition on the neighborhood matrix to obtain M singular values; An average value of the M singular values ​​is taken as the singular value corresponding to the pixel.

2. The image fusion method according to claim 1, characterized in that: The method of fusing the N layers of infrared detail layer images and the N layers of brightness detail layer images one by one according to the matching relationship of the corresponding layers through the adaptive pulse coupled neural network to obtain the N layers of detail layer fused images includes: According to the matching relationship of the N layers of infrared detail layer images and the N layers of brightness detail layer images, data preprocessing is performed on the infrared detail layer images and the brightness detail layer images of each corresponding layer to obtain a first absolute value image and a second absolute value image of each corresponding layer; Input the first absolute value image and the second absolute value image of each corresponding layer as initial excitation into an adaptive pulse coupled neural network, iterate according to a set number of times, and obtain an infrared detail layer ignition matrix and a brightness detail layer ignition matrix of each corresponding layer; Based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer, the infrared detail layer image and the brightness detail layer image are fused to obtain the detail layer fused images of each corresponding layer.

3. The image fusion method according to claim 2, characterized in that: The infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer are used to fuse the infrared detail layer image and the brightness detail layer image to obtain the detail layer fused images of each corresponding layer, including: Based on the infrared detail layer ignition matrix and the brightness detail layer ignition matrix of each corresponding layer, the infrared detail layer ignition matrix and the brightness detail layer ignition matrix are compared. The size of elements; where i and j are both integers; In the infrared detail layer ignition matrix elements greater than the first element in the intensity detail layer ignition matrix In the case of an element, determine the first The pixel value is the first pixel in the detail layer fusion image of the corresponding layer. pixel value; In the infrared detail layer ignition matrix elements are smaller than the first element in the intensity detail layer ignition matrix In the case of an element, determine the first The pixel value is the first pixel in the detail layer fusion image of the corresponding layer. pixel value; In the infrared detail layer ignition matrix The element is equal to the first element in the brightness detail layer ignition matrix In the case of an element, determine the brightness detail layer image and the infrared detail layer image. The mean of the pixels is the number of pixels in the detail layer fusion image of the corresponding layer. pixel values.

4. The image fusion method according to claim 1, characterized in that: Before extracting the brightness channel component of the visible light image to obtain the corresponding brightness component image, the method further includes: registering the infrared image and the visible light image; Performing color space transformation on the registered colored visible light image to obtain a visible light image in HSI color space; The step of superimposing the base layer fusion image and the plurality of detail layer fusion images to obtain a brightness component fusion image, and merging the brightness component fusion image with other channel components of the visible light image to obtain a fusion image includes: The base layer fusion image and the plurality of detail layer fusion images are superimposed to obtain a brightness component fusion image, and the brightness component fusion image is combined with other channel components of the visible light image to obtain an initial fusion image; The initial fused image is subjected to color space conversion to obtain a final fused image.

5. The image fusion method according to claim 1, characterized in that: The step of respectively decomposing the infrared image and the brightness component image to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image, respectively, comprises: The infrared image and the brightness component image are filtered respectively by sub-window variance filtering to obtain the detail layer image and the intermediate layer image corresponding to each other; The infrared image and the intermediate layer image of the brightness component image are filtered respectively by sub-window variance filtering to obtain another corresponding detail layer image and another intermediate layer image, and the filtering of the infrared image and the intermediate layer image of the brightness component image respectively by sub-window variance filtering is returned until the corresponding last detail layer image and base layer image are obtained.

6. An image fusion device, characterized in that: include: An image acquisition module, used to acquire infrared images and visible light images of a target scene collected synchronously; An image acquisition module is used to extract the brightness channel component of the visible light image to obtain a corresponding brightness component image; An image decomposition module, used to decompose the infrared image and the brightness component image respectively to obtain a base layer image and a plurality of detail layer images corresponding to the infrared image and the brightness component image respectively; A base layer image fusion module, used for fusing the base layer images of the infrared image and the brightness component image to obtain a base layer fused image; A detail layer image fusion module, used for fusing the infrared image and the plurality of base layer images of the brightness component image respectively to obtain a plurality of corresponding detail layer fused images; A fused image obtaining module, used for superimposing the base layer fused image and the plurality of detail layer fused images to obtain a brightness component fused image, and merging the brightness component fused image with other channel components of the visible light image to obtain a fused image; The base layer image includes a first base layer image of the infrared image and a second base layer image of the brightness component image, and the base layer image fusion module is used to perform singular value decomposition on the first base layer image and the second base layer image respectively to obtain respective singular value mean vectors in the first base layer image and the second base layer image; based on the respective singular value mean vectors, obtain a first pixel weight map of the first base layer image and a second pixel weight map of the second base layer image; based on the first pixel weight map and the second pixel weight map, perform linear fusion on the first base layer image and the second base layer image to obtain a base layer fused image; The multiple detail layer images include N layers of infrared detail layer images of the infrared image and N layers of brightness detail layer images of the brightness component image. The detail layer image fusion module is used to calculate the average gradient value of each pixel in the N layers of infrared detail layer images and the N layers of brightness detail layer images to obtain a first gradient map corresponding to the infrared detail layer image and a second gradient map corresponding to the brightness detail layer image; normalize the first gradient map and the second gradient map, and determine the synaptic link strength coefficient in the adaptive pulse coupled neural network based on the gradient values ​​of the processed first gradient map and the second gradient map; and fuse the N layers of infrared detail layer images and the N layers of brightness detail layer images one by one according to the matching relationship of the corresponding layers through the adaptive pulse coupled neural network to obtain an N-layer detail layer fused image; The performing singular value decomposition on the first base layer image and the second base layer image respectively to obtain the respective singular value mean vectors of the first base layer image and the second base layer image includes: For each pixel in the first base layer image and the second base layer image, perform the following steps: Obtain an M-order neighborhood matrix centered on the pixel, where M is a positive integer greater than 2; Performing singular value decomposition on the neighborhood matrix to obtain M singular values; An average value of the M singular values ​​is taken as the singular value corresponding to the pixel.

7. An image fusion device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, the image fusion method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the controller, the image fusion method according to any one of claims 1 to 5 is implemented.

Images