A system for registration fusion of infrared video images and visible light video images

Abstract

The application discloses a kind of infrared video image and visible light video image registration fusion system.Therein, system includes: image acquisition module, image registration module, image fusion module and image output module;Image registration module is used to carry out image registration to infrared video image and visible light video image, obtains registered visible light video image;Image fusion module is used to process infrared video image and registered visible light video image, obtains final fusion video image, and final fusion video image is sent to image output module;Image output module is used to provide final fusion video image to target user.The embodiment of the application can be based on image registration module and image fusion module, and infrared video image and visible light video image are registered and fused, and the imaging target of the fusion video image that is clear and imaging is significantly obtained, which sufficiently retains the detail information in infrared video image and visible light video image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a registration fusion system of infrared video images and visible light video images. BACKGROUND

[0002] The infrared camera imaging has the characteristics of not being interfered by low illumination and strong sunlight, strong penetration, and imaging target being obvious. The visible light camera has the characteristics of clear imaging and rich details. In order to accurately monitor the monitoring area, the enterprise can set the infrared camera and the visible light camera for synchronously shooting the monitoring area on the periphery of the monitoring area, synchronously shoot the monitoring area through the infrared camera and the visible light camera, obtain the infrared video and the visible light video of the monitoring area, and fuse each set of synchronously shot infrared video images and visible light video images to obtain the fused video images with obvious imaging target and clear imaging.

[0003] In the related art, a common fusion scheme of infrared video images and visible light video images is: a method based on multi-scale decomposition directly fuses each set of synchronously shot infrared video images and visible light video images to obtain the fused video images. The fusion scheme of infrared video images and visible light video images in the related art directly fuses each set of synchronously shot infrared video images and visible light video images based on the method based on multi-scale decomposition, and details in the infrared video images and the visible light video images are easily lost in the fusion process. SUMMARY

[0004] The present application provides a registration fusion system of infrared video images and visible light video images to solve the problem that the fusion scheme of infrared video images and visible light video images in the related art easily loses details in the infrared video images and the visible light video images in the fusion process.

[0005] According to an aspect of the present application, a registration fusion system of infrared video images and visible light video images is provided, comprising: an image acquisition module, an image registration module, an image fusion module, and an image output module.

[0006] The image acquisition module is configured to send the infrared video images and the visible light video images to the image registration module after obtaining the synchronously shot infrared video images and visible light video images corresponding to the target area.

[0007] The image registration module is configured to perform image registration on the received infrared video images and visible light video images to obtain registered visible light video images registered with the infrared video images, and send the infrared video images and the registered visible light video images to the image fusion module.

[0008] The image fusion module is configured to process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, a base layer image and a detail layer image of the registered visible light video image; determine a base layer image fusion weight map according to the base layer image of the infrared video image; fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image; determine a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image; fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image; add the base layer fusion image and the detail layer fusion image to obtain a final fusion video image; and send the final fusion video image to the image output module.

[0009] The image output module is configured to provide the received final fusion video image to a target user.

[0010] According to another aspect of the present application, there is provided an infrared video image and visible light video image registration fusion method, which is applied to an image fusion module in an infrared video image and visible light video image registration fusion system, and includes the following steps:

[0011] processing the received infrared video image and registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image;

[0012] determining a base layer image fusion weight map according to the base layer image of the infrared video image;

[0013] fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image;

[0014] determining a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image;

[0015] fusing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image;

[0016] Add the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and send the final fusion video image to an image output module.

[0017] According to another aspect of the present application, there is provided an infrared video image and visible light video image registration fusion device configured in an image fusion module in an infrared video image and visible light video image registration fusion system, comprising:

[0018] An image processing unit configured to process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image;

[0019] A first weight map determination unit configured to determine a base layer image fusion weight map according to the base layer image of the infrared video image;

[0020] A base layer image fusion unit configured to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image;

[0021] A second weight map determination unit configured to determine a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image;

[0022] A detail layer image fusion unit configured to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image;

[0023] A fusion video image determination unit configured to add the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and send the final fusion video image to an image output module.

[0024] According to another aspect of the present application, there is provided an electronic device, comprising:

[0025] at least one processor;

[0026] and a memory in communication connection with the at least one processor;

[0027] wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the infrared video image and visible light video image registration fusion method according to any one of the embodiments of the present application.

[0028] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement the method of registering and fusing an infrared video image and a visible light video image according to any of the embodiments of the present application when executed.

[0029] In the technical scheme of the embodiment, after the image acquisition module obtains the infrared video image and the visible light video image corresponding to the target region, the infrared video image and the visible light video image are sent to the image registration module; the image registration module performs image registration on the received infrared video image and visible light video image to obtain a registered visible light video image registered with the infrared video image, and sends the infrared video image and the registered visible light video image to the image fusion module; the image fusion module processes the received infrared video image and registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image and a base layer image and a detail layer image of the registered visible light video image; the image fusion module determines a base layer image fusion weight map according to the base layer image of the infrared video image; the image fusion module fuses the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image; the image fusion module determines a detail layer image fusion weight map according to the detail layer saliency image of the infrared video image and the registered visible light video image; the image fusion module fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image; the image fusion module adds the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and sends the final fusion video image to the image output module.The image output module provides the received final fusion video image to a target user, solves the problem that the fusion scheme of the infrared video image and the visible light video image in the prior art is easy to lose the detail information in the infrared video image and the visible light video image in the fusion process, can register the infrared video image and the visible light video image corresponding to the target region which are synchronously shot based on the image registration module, obtain the registered visible light video image which can be spatially aligned with the infrared video image, and ensure the quality of the subsequent image fusion process, can process the infrared video image and the registered visible light video image after image registration based on the image fusion module, obtain a base layer image containing large-scale information in the infrared video image, a base layer image containing large-scale information in the registered visible light video image, a detail layer image containing texture information and detail information and other small-scale information in the infrared video image, a detail layer image containing texture information and detail information and other small-scale information in the registered visible light video image, then fuse the base layer image containing large-scale information in the infrared video image and the base layer image containing large-scale information in the registered visible light video image according to a base layer image fusion weight map, obtain a base layer fusion image containing large-scale information in the infrared video image and the registered visible light video image and having overall contrast close to the infrared video image, and the imaging target is prominent, and fuse the detail layer image containing texture information and detail information and other small-scale information in the infrared video image and the detail layer image containing texture information and detail information and other small-scale information in the registered visible light video image according to a detail layer image fusion weight map, obtain a detail layer fusion image containing texture information and detail information in the infrared video image and the registered visible light video image, and finally add the base layer fusion image and the detail layer fusion image to obtain a final fusion video image which sufficiently retains the detail information in the infrared video image and the visible light video image, the imaging target is prominent and the imaging is clear, the image fusion module can provide the final fusion video image to the target user, and the detail information in the infrared video image and the visible light video image can be sufficiently retained in the fusion process, and the quality of the image fusion process is improved.

[0030] It should be understood that the matters described in this section are not intended to identify key or important features of the embodiments of the present application, nor are they used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only show some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0032] Figure 1 A structural schematic diagram of a registration fusion system of infrared video images and visible light video images provided for the embodiment one of the present application.

[0033] Figure 2 A flow chart of a registration fusion method of infrared video images and visible light video images provided for the embodiment two of the present application.

[0034] Figure 3 A structural schematic diagram of a registration fusion device of infrared video images and visible light video images provided for the embodiment three of the present application.

[0035] Figure 4 A structural schematic diagram of an electronic device for implementing the registration fusion method of infrared video images and visible light video images of the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the person skilled in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the protection scope of the present application.

[0037] It should be noted that the terms "target", "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise", "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment one

[0039] Figure 1A structural schematic diagram of a registration fusion system of an infrared video image and a visible light video image provided for an embodiment of the present application is shown in the figure. The embodiment can be applied to the fusion of the infrared video image and the visible light video image which are synchronously photographed, to obtain a fused video image. As shown in the figure, the registration fusion system of the infrared video image and the visible light video image can specifically include an image acquisition module 101, an image registration module 102, an image fusion module 103 and an image output module 104, and the structure and functions thereof are described below. Figure 1

[0040] The image acquisition module 101 is configured to, after obtaining the infrared video image and the visible light video image corresponding to the target region which are synchronously photographed, send the infrared video image and the visible light video image to the image registration module 102.

[0041] The image registration module 102 is configured to perform image registration on the received infrared video image and visible light video image, to obtain a registered visible light video image registered with the infrared video image, and send the infrared video image and the registered visible light video image to the image fusion module 103.

[0042] The image fusion module 103 is configured to perform processing on the received infrared video image and registered visible light video image, to obtain a base layer image and a detail layer image of the infrared video image, a base layer image and a detail layer image of the registered visible light video image; determine a base layer image fusion weight map according to the base layer image of the infrared video image; perform fusion on the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map, to obtain a base layer fusion image; determine a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image; perform fusion on the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map, to obtain a detail layer fusion image; add the base layer fusion image and the detail layer fusion image, to obtain a final fused video image; and send the final fused video image to the image output module 104.

[0043] The image output module 104 is configured to provide the received fused video image to a target user.

[0044] ​Optionally, the target area can be an area in an enterprise that needs video monitoring. The periphery of the target area is provided with an infrared camera and a visible light camera. The front ends of the lenses of the infrared camera and the visible light camera are fixed by a metal panel, so that the optical axes of the infrared camera and the visible light camera are parallel, and the fields of view of the infrared camera and the visible light camera coincide. The infrared camera provided at the periphery of the target area is an infrared camera for shooting the target area to obtain a video of the target area. The infrared video of the target area is a video containing the target area shot by the infrared camera provided at the periphery of the target area. The infrared video of the target area is composed of multiple video images. Each video image in the infrared video of the target area is an infrared video image. The infrared camera provided at the periphery of the target area can shoot the target area at a preset frequency to obtain each infrared video image, and send each infrared video image to the image acquisition module 101. The preset frequency can be a frequency preset for shooting the target area to obtain a video image. The visible light camera provided at the periphery of the target area is a visible light camera for shooting the target area to obtain a video of the target area. The visible light video of the target area is a video containing the target area shot by the visible light camera provided at the periphery of the target area. The visible light video of the target area is composed of multiple video images. Each video image in the visible light video of the target area is a visible light video image. The visible light camera provided at the periphery of the target area can shoot the target area at a preset frequency to obtain each visible light video image, and send each visible light video image to the image acquisition module 101.

[0045] Optionally, the image acquisition module 101 can be a hardware module or a software module for sending the synchronously photographed infrared video images and visible light video images corresponding to the target region to the image registration module 102 after the synchronously photographed infrared video images and visible light video images corresponding to the target region are acquired. The synchronously photographed infrared video images and visible light video images corresponding to the target region can refer to a set of synchronously photographed infrared video images and visible light video images of the target region. The infrared camera and the visible light camera arranged at the periphery of the target region are in communication connection with the image acquisition module 101, and information interaction can be performed based on the established communication connection. The infrared camera and the visible light camera arranged at the periphery of the target region synchronously and timely photograph the target region at a preset frequency, obtain each frame of infrared video image and each frame of visible light video image, and send each frame of infrared video image and each frame of visible light video image to the image acquisition module 101. The image acquisition module 101 will receive the synchronously photographed infrared video images and visible light video images sent by the infrared camera and the visible light camera arranged at the periphery of the target region in a timely manner. The infrared video images and visible light video images received by the image acquisition module 101 at the same time are a set of synchronously photographed infrared video images and visible light video images of the target region. After the image acquisition module 101 receives the synchronously photographed infrared video images and visible light video images sent by the infrared camera and the visible light camera arranged at the periphery of the target region each time, it is determined that the synchronously photographed infrared video images and visible light video images corresponding to the target region are acquired, and the received infrared video images and visible light video images are the synchronously photographed infrared video images and visible light video images corresponding to the target region acquired this time. The synchronously photographed infrared video images and visible light video images corresponding to the target region acquired this time are sent to the image registration module 102.

[0046] Optionally, the image registration module 102 can be a hardware module or a software module for image registration of the received infrared video image and the visible light video image corresponding to the target region, to obtain a registered visible light video image registered with the infrared video image, and send the infrared video image and the registered visible light video image to the image fusion module 103. Ideally, the pixel points at the same position in the infrared video image and the visible light video image corresponding to the target region are corresponding to the same position point in the target region. However, due to the installation position of the infrared camera and the visible light camera or other factors, the pixel points in the visible light video image received by the image registration module 102 will have a certain translation, rotation or scaling relative to the pixel points in the received infrared video image, resulting in that the pixel points at the same position in the infrared video image and the visible light video image are not corresponding to the same position point in the target region. The image registration of the infrared video image and the visible light video image can mean adjusting the infrared video image and the visible light video image with pixel points at the same position not corresponding to the same position point in the target region to the infrared video image and the visible light video image with pixel points at the same position corresponding to the same position point in the target region. The preset affine transformation algorithm can be an affine transformation algorithm preset in the image registration module 102 for translation correction, rotation correction or scaling correction of the pixel points in the visible light video image having a certain translation, rotation or scaling relative to the pixel points in the infrared video image, to obtain the visible light video image with pixel points at the same position corresponding to the same position point in the target region as the pixel points in the infrared video image. The image registration module 102 can use the preset affine transformation algorithm to perform translation correction, rotation correction or scaling correction on the pixel points in the visible light video image having a certain translation, rotation or scaling relative to the pixel points in the infrared video image, to obtain the visible light video image with pixel points at the same position corresponding to the same position point in the target region as the pixel points in the infrared video image, so as to adjust the infrared video image and the visible light video image with pixel points at the same position not corresponding to the same position point in the target region to the infrared video image and the visible light video image with pixel points at the same position corresponding to the same position point in the target region. The visible light video image obtained after translation correction, rotation correction or scaling correction of the pixel points in the visible light video image, with pixel points at the same position corresponding to the same position point in the target region as the pixel points in the infrared video image, is the registered visible light video image registered with the infrared video image. The infrared video image and the registered visible light video image are infrared video images and visible light video images with pixel points at the same position corresponding to the same position point in the target region.

[0047] Optionally, the image registration module 102 can use a preset affine transformation algorithm to perform translation correction, rotation correction or scaling correction on the pixel points in the visible light video image after receiving the infrared video image and the visible light video image corresponding to the target region synchronously each time, to obtain a visible light video image of a same pixel point at a same position in the target region, thereby performing image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and then sending the infrared video image and the registered visible light video image to the image fusion module 103.

[0048] Optionally, the image fusion module 103 can be a hardware module or a software module for fusing the infrared video image and the registered visible light video image to obtain a final fusion video image, and sending the final fusion video image to the image output module 104. The fusion video image is an image obtained by registering and fusing the infrared video image and the visible light video image corresponding to the target region synchronously.

[0049] Optionally, the image fusion module 103 can process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image and a base layer image and a detail layer image of the registered visible light video image after receiving the infrared video image and the registered visible light video image each time. The base layer image of the infrared video image can refer to an image containing large-scale information in the infrared video image. The detail layer image of the infrared video image can refer to an image containing texture information and small-scale information such as detail information in the infrared video image. The base layer image of the registered visible light video image can refer to an image containing large-scale information in the registered visible light video image. The detail layer image of the registered visible light video image can refer to an image containing texture information and small-scale information such as detail information in the registered visible light video image.

[0050] Optionally, for each frame of image, Gaussian filtering processing of the image by a Gaussian filter can obtain a base layer image of the image. However, the Gaussian filtering processing can blur all edges of the image, resulting in artifacts and halos near the edges of the base layer image of the image. After obtaining the base layer image of the image, rolling guide filtering processing of the base layer image of the image by a rolling guide filter can restore the edges of the base layer image of the image to be clear, solve the problem of artifacts and halos near the edges of the base layer image of the image, and obtain a base layer image of the image with clear edges, thereby avoiding the risk of edge blurring and over-smoothing in the subsequent fusion process and avoiding abnormal situations such as artifacts and halos in the fusion video image.

[0051] Optionally, for each frame of image, the detail layer image of the image can be obtained by subtracting the image from the base layer image of the image. Subtracting the image from the base layer image of the image means that for each pixel point in the image, the pixel value of the pixel point is subtracted from the pixel value of the pixel point in the same position in the base layer image of the image. The image in which the pixel value of each pixel point is subtracted from the pixel value of the pixel point in the same position in the base layer image of the image is the detail layer image of the image.

[0052] Optionally, the received infrared video image and the registered visible light video image are processed to obtain the base layer image and the detail layer image of the infrared video image and the base layer image and the detail layer image of the registered visible light video image, including: performing Gaussian filtering on the received infrared video image by using a preset Gaussian filter to obtain the base layer image of the infrared video image; performing edge recovery on the base layer image of the infrared video image by using a preset rolling guide filter to obtain the final base layer image of the infrared video image; subtracting the infrared video image from the final base layer image of the infrared video image to obtain the detail layer image of the infrared video image; performing Gaussian filtering on the received registered visible light video image by using a preset Gaussian filter to obtain the base layer image of the registered visible light video image; performing edge recovery on the base layer image of the registered visible light video image by using a preset rolling guide filter to obtain the final base layer image of the registered visible light video image; and subtracting the registered visible light video image from the final base layer image of the registered visible light video image to obtain the detail layer image of the registered visible light video image.

[0053] Optionally, the preset Gaussian filter can be a Gaussian filter provided in the image fusion module 103 and used for performing Gaussian filtering on an image to obtain the base layer image of the image. The preset rolling guide filter can be a rolling guide filter provided in the image fusion module 103 and used for performing rolling guide filtering on the base layer image of the image, so as to restore the edge of the base layer image of the image to be clear, solve the problems of artifacts and halos near the edge of the base layer image of the image, and obtain the base layer image of the image with clear edge.

[0054] Optionally, the image fusion module 103 can perform Gaussian filtering on the received infrared video image through a preset Gaussian filter to obtain a base layer image of the infrared video image. Then the image fusion module 103 can perform rolling guide filtering on the base layer image of the infrared video image through a preset rolling guide filter to restore the edges of the base layer image of the infrared video image to be clear, thereby performing edge restoration on the base layer image of the infrared video image to obtain a final base layer image of the infrared video image. The final base layer image of the infrared video image is the base layer image of the infrared video image after edge restoration and edge clear. Finally, the image fusion module 103 can subtract the infrared video image from the final base layer image of the infrared video image to obtain a detail layer image of the infrared video image. For each pixel point in the infrared video image, the pixel value of the pixel point is subtracted from the pixel value of the pixel point in the same position in the final base layer image of the infrared video image. The infrared video image containing the pixel values of each pixel point subtracted from the pixel values of the pixel points in the same position in the final base layer image of the infrared video image is the detail layer image of the infrared video image.

[0055] Optionally, the base layer image of the infrared video image obtained after performing Gaussian filtering on the received infrared video image through a preset Gaussian filter can be represented by the following formula:

[0056] G I =Gaussian(I I ,σ sI ),

[0057] wherein I I is the infrared video image, G I is the base layer image of the infrared video image obtained after performing Gaussian filtering on the received infrared video image through a preset Gaussian filter, Gaussian() represents the Gaussian filtering operation performed by the preset Gaussian filter, σ sI is the standard deviation of the pixel values of each pixel point in the infrared video image.

[0058] Optionally, the final base layer image of the infrared video image obtained after performing edge restoration on the base layer image of the infrared video image through a preset rolling guide filter can be represented by the following formula:

[0059] B I =GuidedFilter(G I ,σ sGl ,σ r ,m),

[0060] wherein, G I is the base layer image of the infrared video image, B I is the final base layer image of the infrared video image obtained by performing edge restoration on the base layer image of the infrared video image through a preset guided filter, GuidedFilter() represents a guided filtering operation performed by the preset guided filter, σ sGI is the standard deviation of the pixel value of each pixel point in the base layer image of the infrared video image, σ r is a preset control range weight, and m is a preset iteration number. The m is an integer greater than 2. The guided filter can gradually restore the edge of the base layer image of the infrared video image to be clear by performing m times of guided filtering operation on the base layer image of the infrared video image, thereby performing edge restoration on the base layer image of the infrared video image to obtain the final base layer image of the infrared video image.

[0061] Optionally, the detail layer image of the infrared video image obtained by subtracting the final base layer image of the infrared video image from the infrared video image can be represented by the following formula:

[0062] D I = I I -B I ,

[0063] wherein, I I is the infrared video image, B I is the final base layer image of the infrared video image, D I is the detail layer image of the infrared video image.

[0064] Optionally, the image fusion module 103 can perform Gaussian filtering on the received registered visible light video image through a preset Gaussian filter to obtain a base layer image of the registered visible light video image. Then the image fusion module 103 can perform edge recovery on the base layer image of the registered visible light video image through a preset edge-directed filter to restore the edges of the base layer image of the registered visible light video image to be clear, thereby obtaining a final base layer image of the registered visible light video image. The final base layer image of the registered visible light video image is the base layer image of the registered visible light video image after edge recovery. Finally, the image fusion module 103 can subtract the final base layer image of the registered visible light video image from the registered visible light video image to obtain a detail layer image of the registered visible light video image. For each pixel point in the registered visible light video image, the pixel value of the pixel point is subtracted from the pixel value of the pixel point in the same position in the final base layer image of the registered visible light video image. The registered visible light video image in which the pixel value of each pixel point is subtracted from the pixel value of the pixel point in the same position in the final base layer image of the registered visible light video image is the detail layer image of the registered visible light video image.

[0065] Optionally, the base layer image of the registered visible light video image obtained by performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter can be represented by the following formula:

[0066] G v = Gaussian(I v , σ sV ),

[0067] wherein I V is the registered visible light video image, G V is the base layer image of the registered visible light video image obtained by performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter, Gaussian() represents the Gaussian filtering operation performed by the preset Gaussian filter, σ sV is the standard deviation of the pixel values of each pixel point in the registered visible light video image.

[0068] Optionally, the final base layer image of the registered visible light video image obtained by performing edge recovery on the base layer image of the registered visible light video image through a preset edge-directed filter can be represented by the following formula:

[0069] BV = GuidedFilter(G V , σ sGV , σ r , m),

[0070] wherein, G V is the base layer image of the registered visible light video image, B V is the final base layer image of the registered visible light video image obtained by performing edge restoration on the base layer image of the registered visible light video image through a preset guided filter, GuidedFilter() represents a guided filter processing operation performed by the preset guided filter, σ sGV is the standard deviation of the pixel value of each pixel point in the base layer image of the registered visible light video image, σ r is a preset control range weight, and m is a preset iteration number. The m is an integer greater than 2. The guided filter can gradually restore the edge of the base layer image of the registered visible light video image to be clear by performing m times of guided filter processing on the base layer image of the registered visible light video image, thereby performing edge restoration on the base layer image of the registered visible light video image to obtain the final base layer image of the registered visible light video image.

[0071] Optionally, the detail layer image of the registered visible light video image obtained by subtracting the final base layer image of the registered visible light video image from the registered visible light video image can be represented by the following formula:

[0072] D V = I V - B V ,

[0073] wherein, I V is the registered visible light video image, B V is the final base layer image of the registered visible light video image, and D V is the detail layer image of the registered visible light video image.

[0074] Optionally, the image fusion module 103, after obtaining the base layer image and the detail layer image of the infrared video image and the base layer image and the detail layer image of the registered visible light video image, determines a base layer image fusion weight map according to the base layer image of the infrared video image. The base layer image fusion weight map can be an image containing pixel values of each pixel point between 0 and 1, and the size of the pixel value of each pixel point is used to represent the importance of the pixel point at the same position in the base layer image of the infrared video image. According to the base layer image fusion weight map, the weighted sum of the final base layer image of the infrared video image and the final base layer image of the registered visible light video image can obtain a base layer fusion image containing large-scale information in the infrared video image and the registered visible light video image and having a whole contrast close to the infrared video image and a significant imaging target.

[0075] Optionally, determining the base layer image fusion weight map according to the base layer image of the infrared video image comprises: performing normalization processing on the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image; determining a feature distribution map of the base layer image of the infrared video image according to the normalized base layer image of the infrared video image; and determining the base layer image fusion weight map according to the feature distribution map of the base layer image of the infrared video image.

[0076] Optionally, the normalization processing on the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image comprises: calculating the mean value of the pixel values of each pixel point of the infrared video image; calculating the standard deviation of the pixel values of each pixel point of the infrared video image; calculating the standard deviation of the pixel values of each pixel point of the final base layer image of the infrared video image; and updating the pixel value of each pixel point in the final base layer image of the infrared video image to wherein i is the pixel value of the pixel point before updating, μ B is the mean value of the pixel values of each pixel point of the infrared video image, σ B is the standard deviation of the pixel values of each pixel point of the infrared video image, σ sBI is the standard deviation of the pixel values of each pixel point of the final base layer image of the infrared video image. The final base layer image of the infrared video image after the pixel value updating operation is the normalized base layer image of the infrared video image.

[0077] Optionally, the normalized base layer image of the infrared video image can be represented by the following formula:

[0078]

[0079] wherein B I is the base layer image of the final infrared video image, B I ′ is the base layer image of the normalized infrared video image, μ B is the mean value of the pixel values of each pixel point of the infrared video image, σ B is the standard deviation of the pixel values of each pixel point of the infrared video image, σ sBI is the standard deviation of the pixel values of each pixel point of the base layer image of the final infrared video image.

[0080] Optionally, the feature distribution map of the base layer image of the infrared video image can be an image containing the size of the pixel value of each pixel point, which is used to represent the importance size of the pixel point at the same position in the base layer image of the infrared video image.

[0081] Optionally, according to the base layer image of the normalized infrared video image, the feature distribution map of the base layer image of the infrared video image is determined, including: counting the minimum value and the maximum value in the pixel values of each pixel point in the base layer image of the normalized infrared video image; for each pixel point in the base layer image of the normalized infrared video image, updating the pixel value of the pixel point to wherein i is the pixel value of the pixel point before updating, min(B′ I ) is the minimum value in the pixel values of each pixel point in the base layer image of the normalized infrared video image, max(B′ I ) is the maximum value in the pixel values of each pixel point in the base layer image of the normalized infrared video image. The base layer image of the normalized infrared video image after the pixel value updating operation is the feature distribution map of the base layer image of the infrared video image.

[0082] Optionally, the feature distribution map of the base layer image of the infrared video image can be represented by the following formula:

[0083]

[0084] wherein B′ I is the base layer image of the normalized infrared video image, L is the feature distribution map of the base layer image of the infrared video image, min(B′ I ) is the minimum value in the pixel values of each pixel point in the base layer image of the normalized infrared video image, max(B′ I ) is the maximum value in the pixel values of each pixel point in the base layer image of the normalized infrared video image.

[0085] Optionally, the base layer image fusion weight map is determined according to the feature distribution map of the base layer image of the infrared video image, including: for each pixel point in the feature distribution map of the base layer image of the infrared video image, updating the pixel value of the pixel point to Thus, the pixel value of each pixel point in the feature distribution map of the base layer image of the infrared video image is adjusted to between 0 and 1; wherein i is the pixel value of the pixel point before updating, and λ is a preset parameter, λ is between 0 and 1. The feature distribution map of the base layer image of the infrared video image after the pixel value updating operation is completed is the base layer image fusion weight map.

[0086] Optionally, the base layer image fusion weight map can be represented by the following formula:

[0087]

[0088] Wherein, L is the feature distribution map of the base layer image of the infrared video image, W B is the base layer image fusion weight map, and λ is a preset parameter, λ is between 0 and 1.

[0089] Optionally, after the image fusion module 103 determines the base layer image fusion weight map according to the base layer image of the infrared video image, it will fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image. The base layer fusion image can refer to an image containing large-scale information in the infrared video image and the registered visible light video image and having a whole contrast close to the infrared video image, which is obtained by fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image. Generally, the image with a whole contrast close to the infrared video image will be a significant image of the imaging target.

[0090] Optionally, the base layer image fusion weight map is determined according to the feature distribution map of the base layer image of the infrared video image, including: for each pixel point in the feature distribution map of the base layer image of the infrared video image, updating the pixel value of the pixel point to

[0091] Optionally, the base layer image fusion weight map is determined according to the feature distribution map of the base layer image of the infrared video image, including: for each pixel point in the feature distribution map of the base layer image of the infrared video image, updating the pixel value of the pixel point to WB ×iBI + (1 - i WB ) x i BV ; wherein, i BI is a pixel value of a pixel point, i WB is a pixel value of a pixel point located at the same position as the pixel point in the base layer image fusion weight map, i BV is a pixel value of a pixel point located at the same position as the pixel point in the base layer image of the registered visible light video image. The base layer image of the infrared video image after the pixel value updating operation is completed is the base layer fusion image. The base layer fusion image obtained contains large-scale information in the infrared video image and the registered visible light video image and has overall contrast close to the infrared video image, and the imaging target is prominent.

[0092] Optionally, the base layer fusion image can be represented by the following formula:

[0093] B F = W B B I + (1 - W B ) B V ,

[0094] wherein, W B is the base layer image fusion weight map, B I is the base layer image of the infrared video image, B v is the base layer image of the registered visible light video image, and B F is the base layer fusion image.

[0095] Optionally, after obtaining the base layer image and the detail layer image of the infrared video image, the base layer image and the detail layer image of the registered visible light video image, the image fusion module 103 can determine a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image.

[0096] Optionally, the detail layer saliency images of the infrared video image and the registered visible light video image can be images containing salient features in the detail layer images of the infrared video image and the registered visible light video image, which are generated according to the detail layer images of the infrared video image and the registered visible light video image.

[0097] Optionally, the detail layer image fusion weight map comprises a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image. The fusion weight map corresponding to the detail layer image of the infrared video image can be generated according to the detail layer saliency image of the infrared video image and the registered visible light video image, and the pixel value of each pixel point contained in the fusion weight map is between 0 and 1, and the size of the pixel value of each pixel point is used to represent the importance of the pixel point at the same position in the detail layer image of the infrared video image. The fusion weight map corresponding to the detail layer image of the registered visible light video image can be generated according to the detail layer saliency image of the infrared video image and the registered visible light video image, and the pixel value of each pixel point contained in the fusion weight map is between 0 and 1, and the size of the pixel value of each pixel point is used to represent the importance of the pixel point at the same position in the detail layer image of the registered visible light video image. According to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, the weighted sum of the detail layer image of the infrared video image and the detail layer image of the registered visible light video image can obtain a detail layer fusion image containing the texture information and the detail information in the infrared video image and the registered visible light video image.

[0098] Optionally, the detail layer image fusion weight map is determined according to the detail layer saliency image of the infrared video image and the registered visible light video image, comprising: processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain the detail layer saliency image of the infrared video image and the registered visible light video image; performing edge window guided filter processing on the detail layer image of the infrared video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as a guide image through a preset edge window guided filter to obtain the saliency image of the detail layer image of the infrared video image; performing edge window guided filter processing on the detail layer image of the registered visible light video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as a guide image through a preset edge window guided filter to obtain the saliency image of the registered visible light video image; and performing normalization processing on the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image to obtain the detail layer image fusion weight map; wherein the detail layer image fusion weight map comprises a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0099] Optionally, the saliency detection function can be a function set in the image fusion module 103 for processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image to obtain the detail layer saliency image of the infrared video image and the registered visible light video image. The image fusion module 103 can call the saliency detection function to process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through the saliency detection function to obtain the detail layer saliency image of the infrared video image and the registered visible light video image.

[0100] Optionally, processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through the saliency detection function to obtain the detail layer saliency image of the infrared video image and the registered visible light video image includes: calling the saliency detection function, and executing the following operations for each pixel point in the detail layer image of the infrared video image through the saliency detection function: judging whether the absolute value of the pixel value of the pixel point is greater than the absolute value of the pixel value of the pixel point in the same position in the detail layer image of the registered visible light video image; if yes, updating the pixel value of the pixel point as the absolute value of the difference between the pixel value of the pixel point and the pixel value of the pixel point in the same position in the detail layer image of the registered visible light video image; if no, updating the pixel value of the pixel point as 0. The detail layer image of the infrared video image after the pixel value updating operation is completed is the detail layer saliency image of the infrared video image and the registered visible light video image.

[0101] Optionally, the detail layer saliency image of the infrared video image and the registered visible light video image can be represented by the following formula:

[0102]

[0103] wherein, P is the detail layer saliency image of the infrared video image and the registered visible light video image, D I is the detail layer image of the infrared video image, D v is the detail layer image of the registered visible light video image.

[0104] Optionally, the saliency image of the detail layer image of the infrared video image can be an image containing salient features in the detail layer image of the infrared video image, which is generated according to the detail layer image of the infrared video image. The preset edge window guided filter can be a filter set in the image fusion module 103 for performing edge window guided filtering on an image. The image fusion module 103 can perform edge window guided filtering on the detail layer image of the infrared video image by taking the infrared video image and the detail layer saliency image of the registered visible light video image as the guide images through the preset edge window guided filter, to obtain an edge window guided filtered detail layer image of the infrared video image. The edge window guided filtered detail layer image of the infrared video image is an image containing salient features in the detail layer image of the infrared video image, i.e., a saliency image of the detail layer image of the infrared video image.

[0105] Optionally, the saliency image of the detail layer image of the infrared video image can be represented by the following formula:

[0106]

[0107] wherein, is the saliency image of the detail layer image of the infrared video image, D I is the detail layer image of the infrared video image, P is the detail layer saliency image of the infrared video image and the registered visible light video image, GuidedFilter() represents an edge window guided filtering operation performed by the preset edge window guided filter, σ SDI is a standard deviation of pixel values of each pixel point in the detail layer image of the infrared video image, σ r is a preset control range weight.

[0108] Optionally, the saliency image of the detail layer image of the registered visible light video image can be an image containing salient features in the detail layer image of the registered visible light video image, which is generated according to the detail layer image of the registered visible light video image. The image fusion module 103 can perform edge window guided filtering on the detail layer image of the registered visible light video image by taking the registered visible light video image and the detail layer saliency image of the registered visible light video image as the guide images through the preset edge window guided filter, to obtain an edge window guided filtered detail layer image of the registered visible light video image. The edge window guided filtered detail layer image of the registered visible light video image is an image containing salient features in the detail layer image of the registered visible light video image, i.e., a saliency image of the detail layer image of the registered visible light video image.

[0109] Optionally, the saliency image of the detail layer image of the registered visible light video image can be represented by the following formula:

[0110]

[0111] wherein, is a saliency image of the detail layer image of the registered visible light video image, D V is the detail layer image of the registered visible light video image, P is the detail layer saliency image of the infrared video image and the registered visible light video image, GuidedFilter() represents an edge window guided filtering processing operation performed by a preset edge window guided filter, σ SDV is a standard deviation of pixel values of each pixel point in the detail layer image of the registered visible light video image, σ r is a preset control range weight.

[0112] Optionally, the image fusion module 103 can perform normalization processing on the saliency image of the detail layer image of the infrared video image to obtain a fusion weight map corresponding to the detail layer image of the infrared video image.

[0113] Optionally, performing normalization processing on the saliency image of the detail layer image of the infrared video image to obtain a fusion weight map corresponding to the detail layer image of the infrared video image includes: performing the following operation on each pixel point in the saliency image of the detail layer image of the infrared video image: updating the pixel value of the pixel point to wherein, is the pixel value of the pixel point before updating, is a pixel value of a pixel point located at the same position as the pixel point in the saliency image of the detail layer image of the registered visible light video image. The saliency image of the detail layer image of the infrared video image after the pixel value updating operation is completed is the fusion weight map corresponding to the detail layer image of the infrared video image.

[0114] Optionally, the fusion weight map corresponding to the detail layer image of the infrared video image can be represented by the following formula:

[0115]

[0116] wherein, is the fusion weight map corresponding to the detail layer image of the infrared video image, is the saliency image of the detail layer image of the infrared video image, is the saliency image of the detail layer image of the registered visible light video image.

[0117] Optionally, the image fusion module 103 can normalize the saliency images of the detail layer images of the registered visible light video images to obtain the fusion weight maps corresponding to the detail layer images of the registered visible light video images.

[0118] Optionally, the normalization of the saliency images of the detail layer images of the registered visible light video images to obtain the fusion weight maps corresponding to the detail layer images of the registered visible light video images comprises: for each pixel point in the saliency images of the detail layer images of the registered visible light video images, updating the pixel value of the pixel point to wherein, is the pixel value of the pixel point before the updating, is the pixel value of the pixel point in the saliency images of the detail layer images of the infrared video images that is located at the same position as the pixel point. The saliency images of the detail layer images of the registered visible light video images after the pixel value updating operation are the fusion weight maps corresponding to the detail layer images of the registered visible light video images.

[0119] Optionally, the fusion weight maps corresponding to the detail layer images of the registered visible light video images can be represented by the following formula:

[0120]

[0121] wherein, is the fusion weight map corresponding to the detail layer images of the registered visible light video images, is the saliency images of the detail layer images of the infrared video images, is the saliency images of the detail layer images of the registered visible light video images.

[0122] Optionally, after the image fusion module 103 determines the detail layer image fusion weight maps according to the detail layer saliency images of the infrared video images and the registered visible light video images, it will fuse the detail layer images of the infrared video images and the registered visible light video images according to the detail layer image fusion weight maps to obtain the detail layer fusion images. The detail layer fusion images can refer to the images containing the texture information and the detail information in the infrared video images and the registered visible light video images obtained by fusing the base layer images of the infrared video images and the detail layer images of the registered visible light video images.

[0123] Optionally, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fusion image, including: the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are weightedly summed according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain the detail layer fusion image.

[0124] Optionally, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are weightedly summed according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain a detail layer fusion image, including: the following operations are performed for each pixel point in the detail layer image of the infrared video image: the pixel value of the pixel point is updated as wherein, i DI is a pixel value of the pixel point, is a pixel value of a pixel point in the fusion weight map corresponding to the detail layer image of the infrared video image and located at the same position as the pixel point, i DV is a pixel value of a pixel point in the detail layer image of the registered visible light video image and located at the same position as the pixel point, is a pixel value of a pixel point in the fusion weight map corresponding to the detail layer image of the registered visible light video image and located at the same position as the pixel point. The detail layer image of the infrared video image after the pixel value updating operation is the detail layer fusion image. The obtained detail layer fusion image contains texture information and detail information in the infrared video image and the registered visible light video image.

[0125] Optionally, the detail layer fusion image can be represented by the following formula:

[0126]

[0127] wherein, is a fusion weight map corresponding to the detail layer image of the infrared video image, is a fusion weight map corresponding to the detail layer image of the registered visible light video image, D I is a detail layer image of the infrared video image, D V is a detail layer image of the registered visible light video image, D F is a detail layer fusion image.

[0128] Optionally, after obtaining the base layer fusion image and the detail layer fusion image, the image fusion module 103 adds the base layer fusion image and the detail layer fusion image to obtain a final fusion video image. The final fusion video image obtained by adding the base layer fusion image and the detail layer fusion image contains large-scale information in the infrared video image and the registered visible light video image, contains texture information and detail information in the infrared video image and the registered visible light video image, and has an overall contrast close to the infrared video image, so that the imaging target is significant and clear.

[0129] Optionally, adding the base layer fusion image and the detail layer fusion image to obtain a final fusion video image includes: for each pixel point in the base layer fusion image, performing the following operation: updating the pixel value of the pixel point to wherein, is the pixel value of the pixel point, is the pixel value of a pixel point in the detail layer fusion image at the same position as the pixel point. The base layer fusion image after the pixel value updating operation is the final fusion video image. The final fusion video image obtained sufficiently retains detail information in the infrared video image and the visible light video image, and the imaging target is significant and clear.

[0130] Optionally, the final fusion video image can be represented by the following formula:

[0131] F = B F + D F ,

[0132] wherein, B F is the base layer fusion image, D F is the detail layer fusion image, and F is the final fusion video image.

[0133] Optionally, after adding the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, the image fusion module 103 sends the final fusion video image to the image output module 104.

[0134] Optionally, the image output module 104 can be a hardware module or a software module for providing each final fusion video image obtained to a target user. The target user can be a technician for managing a target area. Providing the final fusion video image to the target user includes sending the final fusion video image to a terminal device of the target user. The terminal device of the target user can be a terminal device used by the target user.

[0135] In the technical scheme of the embodiment, after the image acquisition module obtains the infrared video image and the visible light video image corresponding to the target region, the infrared video image and the visible light video image are sent to the image registration module; the image registration module performs image registration on the received infrared video image and visible light video image to obtain a registered visible light video image registered with the infrared video image, and sends the infrared video image and the registered visible light video image to the image fusion module; the image fusion module processes the received infrared video image and registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image and a base layer image and a detail layer image of the registered visible light video image; the image fusion module determines a base layer image fusion weight map according to the base layer image of the infrared video image; the image fusion module fuses the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image; the image fusion module determines a detail layer image fusion weight map according to the detail layer saliency image of the infrared video image and the registered visible light video image; the image fusion module fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image; the image fusion module adds the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and sends the final fusion video image to the image output module.The image output module provides the received final fused video image to the target user, solving the problem that fusion schemes for infrared and visible light video images in related technologies easily lose detailed information during the fusion process. It can register synchronously captured infrared and visible light video images corresponding to the target area based on the image registration module, obtaining a registered visible light video image that is spatially aligned with the infrared video image, ensuring the quality of subsequent image fusion. Based on the image fusion module, the registered infrared and visible light video images are first processed to obtain a base layer image containing large-scale information from the infrared video image, a base layer image containing large-scale information from the registered visible light video image, a detail layer image containing small-scale information such as texture and detail from the infrared video image, and a detail layer image containing small-scale information such as texture and detail from the registered visible light video image. Then, based on the fusion weight map of the base layer image, the large-scale information from the infrared video image is processed... The image fusion process involves fusing a base layer image containing large-scale information from both the infrared and visible light video images. This results in a fused base layer image with large-scale information from both images, an overall contrast close to that of the infrared video image, and a clearly visible target. Then, based on the detail layer image fusion weight map, the image fusion process further involves fusing a detail layer image containing small-scale information such as texture and detail from both the infrared and visible light video images. This results in a fused detail layer image containing texture and detail from both images. Finally, the base layer fused image and the detail layer fused image are added together to obtain a final fused video image that fully preserves the detail information from both the infrared and visible light video images, resulting in a clearly visible and sharp target. This final fused video image can be provided to the target user based on the image fusion module, effectively preserving the detail information from both images and improving the quality of the image fusion process.

[0136] Example 2

[0137] Figure 2 This is a flowchart illustrating a registration and fusion method for infrared video images and visible light video images provided in Embodiment 2 of the present invention. This embodiment is applicable to the fusion of simultaneously captured infrared video images and visible light video images to obtain a fused video image. This method can be applied to the image fusion module of the infrared video image and visible light video image registration and fusion system provided in the above embodiments of the present invention. Figure 2 As shown, the method in this embodiment specifically includes:

[0138] Step 201, processing the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image.

[0139] Optionally, processing the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image comprises: performing Gaussian filtering on the received infrared video image by using a preset Gaussian filter to obtain a base layer image of the infrared video image; performing edge restoration on the base layer image of the infrared video image by using a preset rolling guide filter to obtain a final base layer image of the infrared video image; subtracting the final base layer image of the infrared video image from the infrared video image to obtain a detail layer image of the infrared video image; performing Gaussian filtering on the received registered visible light video image by using a preset Gaussian filter to obtain a base layer image of the registered visible light video image; performing edge restoration on the base layer image of the registered visible light video image by using a preset rolling guide filter to obtain a final base layer image of the registered visible light video image; and subtracting the final base layer image of the registered visible light video image from the registered visible light video image to obtain a detail layer image of the registered visible light video image.

[0140] Step 202, determining a base layer image fusion weight map according to the base layer image of the infrared video image.

[0141] Optionally, determining a base layer image fusion weight map according to the base layer image of the infrared video image comprises: performing normalization processing on the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image; determining a feature distribution map of the base layer image of the infrared video image according to the normalized base layer image of the infrared video image; and determining the base layer image fusion weight map according to the feature distribution map of the base layer image of the infrared video image.

[0142] Step 203, fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image.

[0143] Optionally, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused according to the base layer image fusion weight map to obtain a base layer fusion image, including: the base layer image of the final infrared video image and the base layer image of the final registered visible light video image are weighted and summed according to the base layer image fusion weight map to obtain the base layer fusion image.

[0144] Step 204, determining a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image.

[0145] Optionally, the detail layer image fusion weight map is determined according to the detail layer saliency images of the infrared video image and the registered visible light video image, including: the detail layer images of the infrared video image and the registered visible light video image are processed by a saliency detection function to obtain the detail layer saliency images of the infrared video image and the registered visible light video image; the detail layer image of the infrared video image is edge window guided filter processed by a preset edge window guided filter with the detail layer saliency images of the infrared video image and the registered visible light video image as the guided images to obtain the saliency image of the detail layer image of the infrared video image; the detail layer image of the registered visible light video image is edge window guided filter processed by a preset edge window guided filter with the detail layer saliency images of the infrared video image and the registered visible light video image as the guided images to obtain the saliency image of the registered visible light video image; the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image are normalized to obtain the detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0146] Step 205, fusing the detail layer images of the infrared video image and the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image.

[0147] Optionally, the detail layer images of the infrared video image and the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fusion image, including: the detail layer images of the infrared video image and the registered visible light video image are weighted and summed according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain the detail layer fusion image.

[0148] Step 206, adding the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and sending the final fusion video image to an image output module.

[0149] The technical scheme of the embodiment of the application, by processing the received infrared video image and the registered visible light video image, obtains the base layer image and the detail layer image of the infrared video image, the base layer image and the detail layer image of the registered visible light video image; then according to the base layer image of the infrared video image, determines a base layer image fusion weight map; according to the base layer image fusion weight map, fuses the base layer image of the infrared video image and the base layer image of the registered visible light video image to obtain a base layer fusion image; according to the detail layer saliency images of the infrared video image and the registered visible light video image, determines a detail layer image fusion weight map; according to the detail layer image fusion weight map, fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image to obtain a detail layer fusion image; finally, adds the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and sends the final fusion video image to an image output module. The infrared video image and the registered visible light video image can be processed first to obtain the base layer image containing large-scale information in the infrared video image, the base layer image containing large-scale information in the registered visible light video image, the detail layer image containing texture information and small-scale information such as detail information in the infrared video image, and the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image; then the base layer image containing large-scale information in the infrared video image and the base layer image containing large-scale information in the registered visible light video image are fused according to the base layer image fusion weight map to obtain the base layer fusion image containing large-scale information in the infrared video image and the registered visible light video image and having overall contrast close to the infrared video image and the imaging target saliency; and the detail layer image containing texture information and small-scale information such as detail information in the infrared video image and the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image are fused according to the detail layer image fusion weight map to obtain the detail layer fusion image containing texture information and detail information in the infrared video image and the registered visible light video image; finally, the base layer fusion image and the detail layer fusion image are added to obtain the final fusion video image fully retaining the detail information in the infrared video image and the visible light video image, the imaging target being salient and the imaging being clear. The detail information in the infrared video image and the visible light video image can be fully retained in the fusion process, and the quality of the image fusion process is improved.

[0150] Embodiment three

[0151] Figure 3 A structural schematic diagram of a registration fusion device for infrared video images and visible light video images is provided for Embodiment Three of the present application. The device can be configured in the image fusion module in the registration fusion system for infrared video images and visible light video images provided by the above-mentioned embodiments of the present application. As shown in the figure, the device comprises an image processing unit 301, a first weight map determination unit 302, a base layer image fusion unit 303, a second weight map determination unit 304, a detail layer image fusion unit 305, and a fused video image determination unit 306. Figure 3

[0152] The image processing unit 301 is configured to process the received infrared video image and the registered visible light video image to obtain the base layer image and the detail layer image of the infrared video image, and the base layer image and the detail layer image of the registered visible light video image. The first weight map determination unit 302 is configured to determine a base layer image fusion weight map according to the base layer image of the infrared video image. The base layer image fusion unit 303 is configured to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image. The second weight map determination unit 304 is configured to determine a detail layer image fusion weight map according to the detail layer saliency images of the infrared video image and the registered visible light video image. The detail layer image fusion unit 305 is configured to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fused image. The fused video image determination unit 306 is configured to add the base layer fused image and the detail layer fused image to obtain a final fused video image, and send the final fused video image to an image output module.

[0153] ​The technical scheme of the embodiment of the present application processes the received infrared video image and the registered visible light video image to obtain the base layer image and the detail layer image of the infrared video image, the base layer image and the detail layer image of the registered visible light video image; then, according to the base layer image of the infrared video image, the base layer image fusion weight map is determined; according to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain the base layer fusion image; according to the detail layer saliency image of the infrared video image and the registered visible light video image, the detail layer image fusion weight map is determined; according to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain the detail layer fusion image; finally, the base layer fusion image and the detail layer fusion image are added to obtain the final fusion video image, which is sent to the image output module. The infrared video image and the registered visible light video image are processed to obtain the base layer image containing large-scale information in the infrared video image, the base layer image containing large-scale information in the registered visible light video image, the detail layer image containing texture information and small-scale information such as detail information in the infrared video image, and the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image. Then, according to the base layer image fusion weight map, the base layer image containing large-scale information in the infrared video image and the base layer image containing large-scale information in the registered visible light video image are fused to obtain the base layer fusion image containing large-scale information in the infrared video image and the registered visible light video image and having overall contrast close to the infrared video image and the imaging target saliency. According to the detail layer image fusion weight map, the detail layer image containing texture information and small-scale information such as detail information in the infrared video image and the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image are fused to obtain the detail layer fusion image containing texture information and detail information in the infrared video image and the registered visible light video image. Finally, the base layer fusion image and the detail layer fusion image are added to obtain the final fusion video image fully retaining the detail information in the infrared video image and the visible light video image, which is clear and has the imaging target saliency. The detail information in the infrared video image and the visible light video image can be fully retained in the fusion process, and the quality of the image fusion process is improved.

[0154] In an optional implementation of the embodiment of the present application, optionally, the image processing unit 301 is specifically configured to: perform Gaussian filtering on the received infrared video image by using a preset Gaussian filter to obtain a base layer image of the infrared video image; perform edge recovery on the base layer image of the infrared video image by using a preset rolling guide filter to obtain a final base layer image of the infrared video image; subtract the infrared video image from the final base layer image of the infrared video image to obtain a detail layer image of the infrared video image; perform Gaussian filtering on the received registered visible light video image by using the preset Gaussian filter to obtain a base layer image of the registered visible light video image; perform edge recovery on the base layer image of the registered visible light video image by using the preset rolling guide filter to obtain a final base layer image of the registered visible light video image; and subtract the registered visible light video image from the final base layer image of the registered visible light video image to obtain a detail layer image of the registered visible light video image.

[0155] In an optional implementation of the embodiment of the present application, optionally, the first weight map determination unit 302 is specifically configured to: perform normalization processing on the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image; determine a feature distribution map of the base layer image of the infrared video image according to the normalized base layer image of the infrared video image; and determine a base layer image fusion weight map according to the feature distribution map of the base layer image of the infrared video image.

[0156] In an optional implementation of the embodiment of the present application, optionally, the base layer image fusion unit 303 is specifically configured to: perform weighted summation on the final base layer image of the infrared video image and the final base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image.

[0157] In an optional implementation of the embodiment of the present application, the second weight map determination unit 304 is specifically configured to: process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain a detail layer saliency image of the infrared video image and the registered visible light video image; perform edge window guided filter processing on the detail layer image of the infrared video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as guide images through a preset edge window guided filter to obtain a saliency image of the detail layer image of the infrared video image; perform edge window guided filter processing on the detail layer image of the registered visible light video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as guide images through a preset edge window guided filter to obtain a saliency image of the registered visible light video image; and perform normalization processing on the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0158] In an optional implementation of the embodiment of the present application, the detail layer image fusion unit 305 is specifically configured to: perform weighted summation on the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain a detail layer fusion image.

[0159] As to the apparatus in the above-mentioned embodiments, the specific manners in which various units perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0160] Embodiment Four

[0161] Figure 4 A structural schematic diagram of an electronic device 10 that can be used to implement the method for registering and fusing infrared video images and visible light video images according to the embodiments of the present application is shown. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0162] As Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs to be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or constructed from the storage unit 18 to the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the random access memory (RAM) 13. The processor 11, the read-only memory (ROM) 12, and the random access memory (RAM) 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0163] Various components in the electronic device 10 are connected to the input / output (I / O) interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0164] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the registration fusion method of infrared video images and visible light video images.

[0165] In some embodiments, the registration fusion method of infrared video images and visible light video images can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the read-only memory (ROM) 12 and / or the communication unit 19. When the computer program is constructed to the random access memory (RAM) 13 and executed by the processor 11, one or more steps of the registration fusion method of infrared video images and visible light video images described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the registration fusion method of infrared video images and visible light video images by any other appropriate means, such as by means of firmware.

[0166] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0167] Computer programs used to implement the method of registration fusion of infrared video images and visible light video images of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or block diagrams. The computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0168] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0170] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0171] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0172] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0173] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A system for registered fusion of infrared video images and visible light video images, characterized by The image acquisition module, the image registration module, the image fusion module, and the image output module are included. The image acquisition module is configured to, after obtaining infrared video images and visible light video images corresponding to a target region that are synchronously captured, send the infrared video images and the visible light video images to the image registration module. The image registration module is configured to perform image registration on the received infrared video images and visible light video images to obtain registered visible light video images registered with the infrared video images, and send the infrared video images and the registered visible light video images to the image fusion module. The image fusion module is configured to process the received infrared video images and registered visible light video images to obtain base layer images and detail layer images of the infrared video images, base layer images and detail layer images of the registered visible light video images, determine a base layer image fusion weight map according to the base layer images of the infrared video images, fuse the base layer images of the infrared video images and the base layer images of the registered visible light video images according to the base layer image fusion weight map to obtain a base layer fusion image, determine a detail layer image fusion weight map according to the detail layer significant images of the infrared video images and the registered visible light video images, fuse the detail layer images of the infrared video images and the detail layer images of the registered visible light video images according to the detail layer image fusion weight map to obtain a detail layer fusion image, add the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and send the final fusion video image to the image output module. The image output module is configured to provide the received final fusion video image to a target user. The image fusion module is specifically configured to: perform Gaussian filtering on the received infrared video images by using a preset Gaussian filter to obtain base layer images of the infrared video images; perform edge restoration on the base layer images of the infrared video images by using a preset rolling guide filter to obtain final base layer images of the infrared video images; subtract the infrared video images from the final base layer images of the infrared video images to obtain detail layer images of the infrared video images; perform Gaussian filtering on the received registered visible light video images by using a preset Gaussian filter to obtain base layer images of the registered visible light video images; perform edge restoration on the base layer images of the registered visible light video images by using a preset rolling guide filter to obtain final base layer images of the registered visible light video images; subtract the registered visible light video images from the final base layer images of the registered visible light video images to obtain detail layer images of the registered visible light video images. The image fusion module is further configured to: perform normalization on the final base layer images of the infrared video images to obtain normalized base layer images of the infrared video images. ​ According to the normalized infrared video image, a feature distribution map of the base layer image of the infrared video image is determined; wherein the feature distribution map is an image generated according to the normalized infrared video image, and the size of the pixel value of each pixel point contained in the image is used to represent the importance of the pixel point at the same position in the base layer image of the infrared video image; According to the feature distribution map of the base layer image of the infrared video image, a base layer image fusion weight map is determined; The image fusion module is further configured to: process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain a detail layer saliency image of the infrared video image and the registered visible light video image; perform edge window guided filter processing on the detail layer image of the infrared video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as the guide image through a preset edge window guided filter to obtain a saliency image of the detail layer image of the infrared video image; perform edge window guided filter processing on the detail layer image of the registered visible light video image by taking the detail layer saliency image of the infrared video image and the registered visible light video image as the guide image through a preset edge window guided filter to obtain a saliency image of the registered visible light video image; perform normalization processing on the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

2. A method for registering and fusing infrared video images and visible light video images, applied to an image fusion module in a system for registering and fusing infrared video images and visible light video images as claimed in claim 1, characterized in that, comprises: processing the received infrared video image and the registered visible light video image to obtain the base layer image and the detail layer image of the infrared video image, the base layer image and the detail layer image of the registered visible light video image; determining a base layer image fusion weight map according to the base layer image of the infrared video image; fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image; determining a detail layer image fusion weight map according to the detail layer saliency image of the infrared video image and the registered visible light video image; fusing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image; adding the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, and sending the final fusion video image to an image output module; The processing of the received infrared video image and the registered visible light video image obtains the base layer image and the detail layer image of the infrared video image, and the base layer image and the detail layer image of the registered visible light video image, and comprises the following steps. The received infrared video image is processed by a preset Gaussian filter to obtain the base layer image of the infrared video image. The base layer image of the infrared video image is processed by a preset rolling guide filter to obtain the final base layer image of the infrared video image. The infrared video image and the final base layer image of the infrared video image are subtracted to obtain the detail layer image of the infrared video image. The received registered visible light video image is processed by a preset Gaussian filter to obtain the base layer image of the registered visible light video image. The base layer image of the registered visible light video image is processed by a preset rolling guide filter to obtain the final base layer image of the registered visible light video image. The registered visible light video image and the final base layer image of the registered visible light video image are subtracted to obtain the detail layer image of the registered visible light video image. Correspondingly, the base layer image fusion weight map is determined according to the base layer image of the infrared video image, and comprises the following steps. The final base layer image of the infrared video image is normalized to obtain the normalized base layer image of the infrared video image. The feature distribution map of the base layer image of the infrared video image is determined according to the normalized base layer image of the infrared video image; wherein the feature distribution map is an image containing the size of the pixel value of each pixel point generated according to the normalized base layer image of the infrared video image, and is used to represent the importance size of the pixel point at the same position in the base layer image of the infrared video image. The base layer image fusion weight map is determined according to the feature distribution map of the base layer image of the infrared video image. The detail layer image fusion weight map is determined according to the detail layer saliency images of the infrared video image and the registered visible light video image, and comprises the following steps. The detail layer images of the infrared video image and the registered visible light video image are processed by a saliency detection function to obtain the detail layer saliency images of the infrared video image and the registered visible light video image. The detail layer image of the infrared video image is processed by a preset edge window guide filter to obtain the saliency image of the detail layer image of the infrared video image, taking the detail layer saliency images of the infrared video image and the registered visible light video image as guide images. The detail layer image of the registered visible light video image is processed by a preset edge window guide filter to obtain the saliency image of the registered visible light video image, taking the detail layer saliency images of the infrared video image and the registered visible light video image as guide images. normalizing the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map comprises a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

3. The method of registered fusion of infrared video images and visible light video images according to claim 2, wherein, fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image, comprising: performing weighted summation on the final base layer image of the infrared video image and the final base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image.

4. The method of registered fusion of infrared video images and visible light video images of claim 2, wherein, fusing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image, comprising: performing weighted summation on the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain a detail layer fusion image.

5. An infrared video image and visible light video image registration fusion apparatus configured in an image fusion module in the infrared video image and visible light video image registration fusion system according to claim 1, characterized by comprising: an image processing unit, configured to process the received infrared video image and registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, a base layer image and a detail layer image of the registered visible light video image; a first weight map determination unit, configured to determine a base layer image fusion weight map according to the base layer image of the infrared video image; a base layer image fusion unit, configured to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fusion image; a second weight map determination unit, configured to determine a detail layer image fusion weight map according to the detail layer saliency image of the infrared video image and the registered visible light video image; a detail layer image fusion unit, configured to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fusion image; a fused video image determination unit, configured to add the base layer fusion image and the detail layer fusion image to obtain a final fused video image, and send the final fused video image to an image output module; The image processing unit is specifically configured to: perform Gaussian filtering on the received infrared video image by using a preset Gaussian filter to obtain a base layer image of the infrared video image; perform edge restoration on the base layer image of the infrared video image by using a preset rolling guide filter to obtain a final base layer image of the infrared video image; subtract the infrared video image from the final base layer image of the infrared video image to obtain a detail layer image of the infrared video image; perform Gaussian filtering on the received registered visible light video image by using a preset Gaussian filter to obtain a base layer image of the registered visible light video image; perform edge restoration on the base layer image of the registered visible light video image by using a preset rolling guide filter to obtain a final base layer image of the registered visible light video image; and subtract the registered visible light video image from the final base layer image of the registered visible light video image to obtain a detail layer image of the registered visible light video image. The first weight map determination unit is specifically configured to: perform normalization processing on the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image; determine a feature distribution map of the base layer image of the infrared video image according to the normalized base layer image of the infrared video image; and determine a base layer image fusion weight map according to the feature distribution map of the base layer image of the infrared video image, wherein the feature distribution map is an image containing the size of the pixel value of each pixel point and is used to represent the importance of the pixel point at the same position in the base layer image of the infrared video image. The second weight map determination unit is specifically configured to: process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image by using a saliency detection function to obtain detail layer saliency images of the infrared video image and the registered visible light video image; perform edge window guide filtering processing on the detail layer image of the infrared video image by using the detail layer saliency images of the infrared video image and the registered visible light video image as guide images by using a preset edge window guide filter to obtain a saliency image of the detail layer image of the infrared video image; perform edge window guide filtering processing on the detail layer image of the registered visible light video image by using the detail layer saliency images of the infrared video image and the registered visible light video image as guide images by using a preset edge window guide filter to obtain a saliency image of the registered visible light video image; and perform normalization processing on the saliency image of the detail layer image of the infrared video image and the saliency image of the registered visible light video image to obtain a detail layer image fusion weight map, wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

6. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method of registering and fusing the infrared video image and the visible light video image according to any one of claims 2-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the method of registering and fusing the infrared video image and the visible light video image according to any one of claims 2-4 when executed.

Images