Registration fusion system for infrared video image and visible light video image

Abstract

The invention discloses a registration fusion system for an infrared video image and a visible light video image. The system comprises an image acquisition module, an image registration module, an image fusion module and an image output module, the image registration module is used for performing image registration on the infrared video image and the visible light video image to obtain a registered visible light video image; the image fusion module is used for processing the infrared video image and the registered visible light video image to obtain a final fusion video image and sending the final fusion video image to the image output module; and the image output module is used for providing the final fused video image for the target user. According to the embodiment of the invention, on the basis of the image registration module and the image fusion module, the infrared video image and the visible light video image are registered and fused, and the fused video image which fully retains detail information in the infrared video image and the visible light video image and is remarkable in imaging target and clear in imaging is obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a registration and fusion system for infrared video images and visible light video images. Background Art

[0002] Infrared camera imaging is not affected by low illumination and strong sunlight, has strong penetration ability, can penetrate smoke, and has obvious imaging targets. Visible light cameras have the characteristics of clear imaging and rich details. In order to accurately monitor the monitoring area, the company can set up infrared cameras and visible light cameras around the monitoring area to synchronously shoot the monitoring area. The infrared camera and visible light camera synchronously shoot the monitoring area to obtain infrared video and visible light video of the monitoring area, and fuse each set of synchronously shot infrared video images and visible light video images to obtain a fused video image with obvious imaging targets and clear imaging.

[0003] In the related art, the commonly used fusion scheme of infrared video images and visible light video images is: based on the multi-scale decomposition method, each group of infrared video images and visible light video images taken synchronously are directly fused to obtain a fused video image. In the related art, the fusion scheme of infrared video images and visible light video images is based on the multi-scale decomposition method, each group of infrared video images and visible light video images taken synchronously are directly fused, which is easy to lose the detail information in the infrared video images and visible light video images during the fusion process. Summary of the invention

[0004] The present invention provides a registration and fusion system for infrared video images and visible light video images to solve the problem that the fusion scheme of infrared video images and visible light video images in the related art is prone to lose detail information in the infrared video images and visible light video images during the fusion process.

[0005] According to one aspect of the present invention, there is provided a registration and fusion system for infrared video images and visible light video images, comprising: an image acquisition module, an image registration module, an image fusion module and an image output module;

[0006] The image acquisition module is used to send the infrared video image and the visible light video image to the image registration module after acquiring the infrared video image and the visible light video image corresponding to the target area shot synchronously;

[0007] The image registration module is used to perform image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and send the infrared video image and the registered visible light video image to the image fusion module;

[0008] The image fusion module is used to process the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image; determine the base layer image fusion weight map according to the base layer image of the infrared video image; fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fused image; determine the detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain the detail layer fused image; add the base layer fused image and the detail layer fused image to obtain the final fused video image, and send the final fused video image to the image output module;

[0009] The image output module is used to provide the received final fused video image to a target user.

[0010] According to another aspect of the present invention, a registration and fusion method for infrared video images and visible light video images is provided, which is applied to an image fusion module in a registration and fusion system for infrared video images and visible light video images, and comprises:

[0011] Processing the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image;

[0012] Determining a base layer image fusion weight map according to the base layer image of the infrared video image;

[0013] According to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image;

[0014] Determining a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image;

[0015] According to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain a detail layer fused image;

[0016] The base layer fused image and the detail layer fused image are added to obtain a final fused video image, and the final fused video image is sent to an image output module.

[0017] According to another aspect of the present invention, there is provided a registration and fusion device for infrared video images and visible light video images, which is configured in an image fusion module in a registration and fusion system for infrared video images and visible light video images, and comprises:

[0018] An image processing unit, configured to process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image;

[0019] A first weight map determining unit, configured to determine a base layer image fusion weight map according to the base layer image of the infrared video image;

[0020] A base layer image fusion unit, configured to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image;

[0021] A second weight map determining unit, configured to determine a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image;

[0022] A detail layer image fusion unit, configured to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fused image;

[0023] The fused video image determination unit is used to add the base layer fused image and the detail layer fused image to obtain a final fused video image, and send the final fused video image to an image output module.

[0024] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0025] at least one processor;

[0026] and a memory communicatively coupled to the at least one processor;

[0027] The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for registration and fusion of infrared video images and visible light video images described in any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for registering and fusing infrared video images and visible light video images described in any embodiment of the present invention when executed.

[0029] In the technical solution of the embodiment of the present invention, after the image acquisition module acquires the infrared video image and the visible light video image corresponding to the target area shot synchronously, the infrared video image and the visible light video image are sent to the image registration module; the image registration module performs image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and sends the infrared video image and the registered visible light video image to the image fusion module; the image fusion module processes the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; the image fusion module processes the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; the image fusion module performs image registration on the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the registered visible light video image; the image fusion module performs image registration on the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the registered visible light video image. The image fusion module determines the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fusion image; the image fusion module determines the detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; the image fusion module fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain the detail layer fusion image; the image fusion module adds the base layer fusion image and the detail layer fusion image to obtain the final fusion video image, and sends the final fusion video image to the image output module;The image output module provides the received final fused video image to the target user, which solves the problem that the fusion scheme of infrared video images and visible light video images in the related art is easy to lose the detail information in the infrared video images and visible light video images during the fusion process. The infrared video image and the visible light video image corresponding to the target area taken synchronously can be registered based on the image registration module to obtain a registered visible light video image that can be spatially aligned with the infrared video image, thereby ensuring the quality of the subsequent image fusion process. Based on the image fusion module, the infrared video image and the registered visible light video image after image registration can be processed first to obtain a base layer image containing large-scale information in the infrared video image, a base layer image containing large-scale information in the registered visible light video image, a detail layer image containing small-scale information such as texture information and detail information in the infrared video image, and a detail layer image containing small-scale information such as texture information and detail information in the registered visible light video image. Then, the base layer image fusion weight map is used to process the large-scale information in the infrared video image. The base layer image containing the information and the base layer image containing the large-scale information in the registered visible light video image are fused to obtain a base layer fused image containing the large-scale information in the infrared video image and the registered visible light video image and having an overall contrast close to the infrared video image, and having a significant imaging target. The detail layer image containing the small-scale information such as texture information and detail information in the infrared video image and the detail layer image containing the small-scale information such as texture information and detail information in the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image containing the texture information and detail information in the infrared video image and the registered visible light video image. Finally, the base layer fused image and the detail layer fused image are added to obtain a final fused video image with sufficient retention of the detail information in the infrared video image and the visible light video image, with significant imaging targets and clear imaging. Based on the image fusion module, the final fused video image can be provided to the target user. The detail information in the infrared video image and the visible light video image can be fully retained during the fusion process, thereby improving the quality of the image fusion process. ;

[0030] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A structural schematic diagram of a system for registering and fusing infrared video images and visible light video images provided in Embodiment 1 of the present invention.

[0033] Figure 2 A flowchart of a method for registering and fusing infrared video images and visible light video images provided in Embodiment 2 of the present invention.

[0034] Figure 3 A schematic diagram of the structure of a device for registering and fusing infrared video images and visible light video images provided in Embodiment 3 of the present invention.

[0035] Figure 4 A schematic structural diagram of an electronic device for implementing the registration and fusion method of infrared video images and visible light video images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "target", "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise", "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment 1

[0039] Figure 1The structure diagram of a system for registering and fusing infrared video images and visible light video images provided in the first embodiment of the present invention is shown in FIG. This embodiment is applicable to the case where infrared video images and visible light video images taken synchronously are fused to obtain a fused video image. Figure 1 As shown, the registration and fusion system of infrared video images and visible light video images may specifically include: an image acquisition module 101, an image registration module 102, an image fusion module 103 and an image output module 104, and its structure and function are described below.

[0040] The image acquisition module 101 is used to send the infrared video image and the visible light video image to the image registration module 102 after acquiring the infrared video image and the visible light video image corresponding to the target area shot synchronously.

[0041] The image registration module 102 is used to perform image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and send the infrared video image and the registered visible light video image to the image fusion module 103.

[0042] The image fusion module 103 is used to process the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image; determine the base layer image fusion weight map according to the base layer image of the infrared video image; fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fused image; determine the detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain the detail layer fused image; add the base layer fused image and the detail layer fused image to obtain the final fused video image, and send the final fused video image to the image output module 104.

[0043] The image output module 104 is used to provide the received fused video image to the target user.

[0044] Optionally, the target area may be an area in an enterprise that requires video surveillance. An infrared camera and a visible light camera are arranged around the target area. The front ends of the lenses of the infrared camera and the visible light camera are fixed by a metal panel so that the optical axes of the infrared camera and the visible light camera are parallel, and the fields of view of the infrared camera and the visible light camera overlap. The infrared camera arranged around the target area is an infrared camera used to shoot the target area and obtain a video of the target area. The infrared video of the target area is a video of the target area taken by the infrared camera arranged around the target area. The infrared video of the target area is composed of multiple frames of video images. Each frame of video image in the infrared video of the target area is a frame of infrared video image. The infrared camera arranged around the target area can shoot the target area at a preset frequency, obtain each frame of infrared video image, and send each frame of infrared video image to the image acquisition module 101. The preset frequency may be a pre-set frequency for shooting the target area and obtaining a video image. The visible light camera arranged around the target area is a visible light camera used to shoot the target area and obtain a video of the target area. The visible light video of the target area is a video of the target area captured by a visible light camera disposed around the target area. The visible light video of the target area is composed of multiple frames of video images. Each frame of video image in the visible light video of the target area is a frame of visible light video image. The visible light camera disposed around the target area can capture the target area at a preset frequency and at a fixed time to obtain each frame of visible light video image, and send each frame of visible light video image to the image acquisition module 101.

[0045] Optionally, the image acquisition module 101 may be a hardware module or software module for sending the infrared video image and the visible light video image corresponding to the target area taken synchronously to the image registration module 102 after acquiring the infrared video image and the visible light video image corresponding to the target area taken synchronously. The infrared video image and the visible light video image corresponding to the target area taken synchronously may refer to a group of infrared video images and visible light video images obtained by synchronously shooting the target area. The infrared camera and the visible light camera set at the periphery of the target area establish a communication connection with the image acquisition module 101, and information exchange can be performed based on the established communication connection. The infrared camera and the visible light camera set at the periphery of the target area synchronously and regularly shoot the target area according to a preset frequency, obtain each frame of infrared video image and each frame of visible light video image, and send each frame of infrared video image and each frame of visible light video image to the image acquisition module 101. The image acquisition module 101 will receive the infrared video image and the visible light video image taken synchronously and sent by the infrared camera and the visible light camera set at the periphery of the target area at a regular time. The infrared video image and visible light video image received by the image acquisition module 101 at the same time are a set of infrared video images and visible light video images obtained by synchronously shooting the target area. After each time the image acquisition module 101 receives the infrared video image and visible light video image synchronously shot and sent by the infrared camera and visible light camera set around the target area, it determines that the infrared video image and visible light video image corresponding to the target area shot synchronously are obtained, and the received infrared video image and visible light video image are the infrared video image and visible light video image corresponding to the target area shot synchronously obtained this time, and sends the infrared video image and visible light video image corresponding to the target area shot synchronously obtained this time to the image registration module 102.

[0046] Optionally, the image registration module 102 may be a hardware module or software module for performing image registration on the received infrared video image and visible light video image corresponding to the target area that are taken synchronously, obtaining a registered visible light video image that is registered with the infrared video image, and sending the infrared video image and the registered visible light video image to the image fusion module 103. Ideally, the pixels at the same position in the infrared video image and the visible light video image that are taken synchronously and correspond to the target area correspond to the same position point in the target area. However, due to the installation position of the infrared camera and the visible light camera or other factors, the pixels in the visible light video image received by the image registration module 102 may have a certain translation, rotation or scaling relative to the pixels in the received infrared video image, resulting in the pixels at the same position in the infrared video image and the visible light video image not corresponding to the same position point in the target area. Performing image registration on the infrared video image and the visible light video image may refer to adjusting the infrared video image and the visible light video image whose pixels at the same position are not corresponding to the same position in the target area to the infrared video image and the visible light video image whose pixels at the same position correspond to the same position in the target area. The preset affine transformation algorithm may be a pre-set algorithm in the image registration module 102 for performing translation correction, rotation correction or scaling correction on the pixels in the visible light video image whose pixels are translated, rotated or scaled to a certain extent relative to the pixels in the infrared video image, so as to obtain the affine transformation algorithm of the visible light video image whose pixels at the same position in the infrared video image correspond to the same position in the target area. The image registration module 102 can use a preset affine transformation algorithm to perform translation correction, rotation correction or scaling correction on the pixels in the visible light video image where the pixels are translated, rotated or scaled to a certain extent relative to the pixels in the infrared video image, so as to obtain a visible light video image where the pixels and the pixels at the same position in the infrared video image correspond to the same position in the target area, thereby adjusting the infrared video image and the visible light video image where the pixels at the same position do not correspond to the same position in the target area to the infrared video image and the visible light video image where the pixels at the same position correspond to the same position in the target area. The visible light video image where the pixels and the pixels at the same position in the infrared video image correspond to the same position in the target area after the translation correction, rotation correction or scaling correction is performed on the pixels in the visible light video image is the registered visible light video image registered with the infrared video image. The infrared video image and the registered visible light video image are the infrared video image and the visible light video image where the pixels at the same position correspond to the same position in the target area.

[0047] Optionally, the image registration module 102 may use a preset affine transformation algorithm to perform translation correction, rotation correction or scaling correction on the pixel points in the visible light video image each time after receiving the infrared video image and the visible light video image corresponding to the target area taken synchronously, so as to obtain a visible light video image of the same position in the target area corresponding to the pixel point at the same position in the infrared video image, thereby performing image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and then sending the infrared video image and the registered visible light video image to the image fusion module 103.

[0048] Optionally, the image fusion module 103 may be a hardware module or a software module for fusing the infrared video image and the registered visible light video image to obtain a final fused video image, and sending the final fused video image to the image output module 104. The fused video image is an image obtained by registering and fusing the infrared video image and the visible light video image corresponding to the target area that are shot synchronously.

[0049] Optionally, after receiving the infrared video image and the registered visible light video image each time, the image fusion module 103 processes the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image. The base layer image of the infrared video image may refer to an image containing large-scale information in the infrared video image. The detail layer image of the infrared video image may refer to an image containing small-scale information such as texture information and detail information in the infrared video image. The base layer image of the registered visible light video image may refer to an image containing large-scale information in the registered visible light video image. The detail layer image of the registered visible light video image may refer to an image containing small-scale information such as texture information and detail information in the registered visible light video image.

[0050] Optionally, for each frame of the image, the image is subjected to Gaussian filtering processing through a Gaussian filter to obtain a base layer image of the image. However, the Gaussian filtering processing may blur all the edges of the image, resulting in artifacts and halos near the edges of the base layer image of the image. After obtaining the base layer image of the image, the base layer image of the image is subjected to rolling guided filtering processing through a rolling guided filter, which can restore the edges of the base layer image of the image to clarity, solve the problem of artifacts and halos near the edges of the base layer image of the image, and obtain a base layer image of the image with clear edges, thereby avoiding the risk of edge blur and over-smoothing in the subsequent fusion process, and avoiding the abnormal situation of artifacts and halos in the fused video image.

[0051] Optionally, for each frame of the image, the detail layer image of the image can be obtained by subtracting the image from the base layer image of the image. Subtracting the image from the base layer image of the image means that for each pixel in the image, the pixel value of the pixel is subtracted from the pixel value of the pixel in the base layer image of the image that is located at the same position as the pixel. The image containing the pixel value of each pixel minus the pixel value of the pixel in the base layer image of the image that is located at the same position is the detail layer image of the image.

[0052] Optionally, the received infrared video image and the registered visible light video image are processed to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image, including: performing Gaussian filtering on the received infrared video image through a preset Gaussian filter to obtain the base layer image of the infrared video image; performing edge restoration on the base layer image of the infrared video image through a preset rolling guided filter to obtain the final base layer image of the infrared video image; subtracting the infrared video image from the final base layer image of the infrared video image to obtain the detail layer image of the infrared video image; performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter to obtain the base layer image of the registered visible light video image; performing edge restoration on the base layer image of the registered visible light video image through a preset rolling guided filter to obtain the final base layer image of the registered visible light video image; subtracting the registered visible light video image from the final base layer image of the registered visible light video image to obtain the detail layer image of the registered visible light video image.

[0053] Optionally, the preset Gaussian filter may be a Gaussian filter set in the image fusion module 103 for performing Gaussian filtering on the image to obtain a base layer image of the image. The preset rolling guide filter may be a rolling guide filter set in the image fusion module 103 for performing rolling guide filtering on the base layer image of the image, thereby restoring the edge of the base layer image of the image to clarity, solving the problem of artifacts and halos near the edge of the base layer image of the image, and obtaining a base layer image of the image with clear edges.

[0054] Optionally, the image fusion module 103 may perform Gaussian filtering on the received infrared video image through a preset Gaussian filter to obtain the base layer image of the infrared video image. Then the image fusion module 103 may perform rolling guided filtering on the base layer image of the infrared video image through a preset rolling guided filter to restore the edge of the base layer image of the infrared video image to be clear, thereby performing edge restoration on the base layer image of the infrared video image to obtain the final base layer image of the infrared video image. The final base layer image of the infrared video image is the base layer image of the infrared video image with clear edges after edge restoration. Finally, the image fusion module 103 may subtract the infrared video image from the base layer image of the final infrared video image to obtain the detail layer image of the infrared video image. For each pixel point in the infrared video image, the pixel value of the pixel point is subtracted from the pixel value of the pixel point in the base layer image of the final infrared video image that is located at the same position as the pixel point. The infrared video image that includes the pixel values ​​of each pixel point minus the pixel values ​​of the pixel points located at the same position in the base layer image of the final infrared video image is the detail layer image of the infrared video image.

[0055] Optionally, the base layer image of the infrared video image obtained by performing Gaussian filtering on the received infrared video image through a preset Gaussian filter can be expressed by the following formula:

[0056] G I =Gaussian(I I , σ sI ),

[0057] Among them, I I is the infrared video image, G I is a base layer image of the infrared video image obtained by performing Gaussian filtering on the received infrared video image through a preset Gaussian filter, Gaussian() represents the Gaussian filtering operation performed by the preset Gaussian filter, σ sI is the standard deviation of the pixel values ​​of each pixel in the infrared video image.

[0058] Optionally, the final base layer image of the infrared video image obtained by performing edge restoration on the base layer image of the infrared video image through a preset rolling guided filter can be expressed by the following formula:

[0059] B I =GuidedFilter(G I , σ sGl , σ r , m),

[0060] Among them, G I is the base layer image of the infrared video image, B I is the final base layer image of the infrared video image obtained by edge restoration of the base layer image of the infrared video image through a preset rolling guided filter, GuidedFilter() represents the rolling guided filtering operation performed by the preset rolling guided filter, σ sGI is the standard deviation of the pixel values ​​of each pixel in the base layer image of the infrared video image, σ r is a preset control range weight, and m is a preset number of iterations. m is an integer greater than 2. The rolling guided filter can gradually restore the edge of the base layer image of the infrared video image to clarity by performing m rolling guided filtering processes on the base layer image of the infrared video image, thereby performing edge restoration on the base layer image of the infrared video image to obtain the final base layer image of the infrared video image.

[0061] Optionally, the detail layer image of the infrared video image obtained by subtracting the infrared video image from the final base layer image of the infrared video image can be expressed by the following formula:

[0062] D I =I I -B I ,

[0063] Among them, I I is the infrared video image, B I is the base layer image of the final infrared video image, D I It is a detail layer image of the infrared video image.

[0064] Optionally, the image fusion module 103 may perform Gaussian filtering on the received registered visible light video image through a preset Gaussian filter to obtain the base layer image of the registered visible light video image. Then the image fusion module 103 may perform rolling guided filtering on the base layer image of the registered visible light video image through a preset rolling guided filter to restore the edge of the base layer image of the registered visible light video image to be clear, thereby performing edge restoration on the base layer image of the registered visible light video image to obtain the final base layer image of the registered visible light video image. The final base layer image of the registered visible light video image is the base layer image of the registered visible light video image with clear edges after edge restoration. Finally, the image fusion module 103 may subtract the registered visible light video image from the final base layer image of the registered visible light video image to obtain the detail layer image of the registered visible light video image. For each pixel point in the registered visible light video image, the pixel value of the pixel point is subtracted from the pixel value of the pixel point located at the same position as the pixel point in the final base layer image of the registered visible light video image. The registered visible light video image, which includes the pixel values ​​of each pixel point subtracted from the pixel values ​​of the pixel points located at the same position in the final base layer image of the registered visible light video image, is the detail layer image of the registered visible light video image.

[0065] Optionally, the base layer image of the registered visible light video image obtained by performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter can be expressed by the following formula:

[0066] G v =Gaussian(I v , σ sV ),

[0067] Among them, I V is the registered visible light video image, G V is the base layer image of the registered visible light video image obtained by performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter, Gaussian() represents the Gaussian filtering operation performed by the preset Gaussian filter, σ sV is the standard deviation of the pixel values ​​of each pixel in the registered visible light video image.

[0068] Optionally, the final base layer image of the registered visible light video image obtained by performing edge restoration on the base layer image of the registered visible light video image through a preset rolling guided filter can be expressed by the following formula:

[0069] BV =GuidedFilter(G V , σ sGV , σ r , m),

[0070] Among them, G V is the base layer image of the registered visible light video image, B V The base layer image of the registered visible light video image is obtained by edge restoration of the base layer image of the registered visible light video image through a preset rolling guided filter. GuidedFilter() represents the rolling guided filtering operation performed by the preset rolling guided filter. σ sGV is the standard deviation of the pixel values ​​of each pixel in the base layer image of the registered visible light video image, σ r is a preset control range weight, and m is a preset number of iterations. m is an integer greater than 2. The rolling guided filter can gradually restore the edge of the base layer image of the registered visible light video image to clarity by performing m rolling guided filtering processes on the base layer image of the registered visible light video image, thereby performing edge restoration on the base layer image of the registered visible light video image to obtain the final base layer image of the registered visible light video image.

[0071] Optionally, the detail layer image of the registered visible light video image obtained by subtracting the registered visible light video image from the final base layer image of the registered visible light video image can be expressed by the following formula:

[0072] D V =I V -B V ,

[0073] Among them, I V is the registered visible light video image, B V is the final base layer image of the registered visible light video image, D V It is a detail layer image of the registered visible light video image.

[0074] Optionally, after obtaining the base layer image and detail layer image of the infrared video image and the base layer image and detail layer image of the registered visible light video image, the image fusion module 103 will determine a base layer image fusion weight map according to the base layer image of the infrared video image. The base layer image fusion weight map can be an image generated according to the base layer image of the infrared video image, in which the pixel value of each pixel point included is between 0 and 1, and the size of the pixel value of each pixel point included is used to characterize the importance of the pixel points located at the same position in the base layer image of the infrared video image. According to the base layer image fusion weight map, the final base layer image of the infrared video image and the final base layer image of the registered visible light video image are weighted and summed, and a base layer fusion image containing large-scale information in the infrared video image and the registered visible light video image and having an overall contrast close to the infrared video image and with a prominent imaging target can be obtained.

[0075] Optionally, based on the base layer image of the infrared video image, a base layer image fusion weight map is determined, including: normalizing the base layer image of the final infrared video image to obtain a normalized base layer image of the infrared video image; determining a feature distribution map of the base layer image of the infrared video image based on the base layer image of the normalized infrared video image; and determining a base layer image fusion weight map based on the feature distribution map of the base layer image of the infrared video image.

[0076] Optionally, the base layer image of the final infrared video image is normalized to obtain a normalized base layer image of the infrared video image, including: calculating the mean of the pixel values ​​of each pixel point of the infrared video image; calculating the standard deviation of the pixel values ​​of each pixel point of the infrared video image; calculating the standard deviation of the pixel values ​​of each pixel point of the base layer image of the final infrared video image; for each pixel point in the base layer image of the final infrared video image, updating the pixel value of the pixel point to Among them, i is the pixel value before updating, μ B is the mean value of the pixel values ​​of each pixel point of the infrared video image, σ B is the standard deviation of the pixel values ​​of each pixel point of the infrared video image, σ sBI is the standard deviation of the pixel values ​​of each pixel point of the base layer image of the final infrared video image. The base layer image of the final infrared video image after completing the pixel value update operation is the normalized base layer image of the infrared video image.

[0077] Optionally, the base layer image of the normalized infrared video image can be expressed by the following formula:

[0078]

[0079] Among them, B I is the base layer image of the final infrared video image, B I ′ is the base layer image of the normalized infrared video image, μ B is the mean value of the pixel values ​​of each pixel point of the infrared video image, σ B is the standard deviation of the pixel values ​​of each pixel point of the infrared video image, σ sBI It is the standard deviation of the pixel values ​​of each pixel point of the base layer image of the final infrared video image.

[0080] Optionally, the feature distribution map of the base layer image of the infrared video image can be an image generated according to the normalized base layer image of the infrared video image, which contains the pixel value of each pixel point and is used to characterize the importance of pixels located at the same position in the base layer image of the infrared video image.

[0081] Optionally, according to the base layer image of the normalized infrared video image, determining the feature distribution map of the base layer image of the infrared video image, including: counting the minimum and maximum values ​​of the pixel values ​​of each pixel point in the base layer image of the normalized infrared video image; for each pixel point in the base layer image of the normalized infrared video image, updating the pixel value of the pixel point to Among them, i is the pixel value before updating, min(B′ I ) is the minimum value of the pixel values ​​of each pixel in the base layer image of the normalized infrared video image, max(B′ I ) is the maximum value of the pixel values ​​of each pixel point in the base layer image of the normalized infrared video image. The base layer image of the normalized infrared video image after completing the pixel value update operation is the feature distribution map of the base layer image of the infrared video image.

[0082] Optionally, the feature distribution map of the base layer image of the infrared video image can be expressed by the following formula:

[0083]

[0084] Among them, B′ I is the normalized base layer image of the infrared video image, L is the characteristic distribution map of the base layer image of the infrared video image, min(B′ I ) is the minimum value of the pixel values ​​of each pixel in the base layer image of the normalized infrared video image, max(B′ I ) is the maximum value of the pixel values ​​of each pixel in the base layer image of the normalized infrared video image.

[0085] Optionally, determining the base layer image fusion weight map according to the feature distribution map of the base layer image of the infrared video image includes: for each pixel point in the feature distribution map of the base layer image of the infrared video image, updating the pixel value of the pixel point to Thus, the pixel value of each pixel in the feature distribution map of the base layer image of the infrared video image is adjusted to between 0 and 1; wherein i is the pixel value of the pixel before updating, and λ is a preset parameter, which is between 0 and 1. The feature distribution map of the base layer image of the infrared video image after completing the pixel value update operation is the base layer image fusion weight map.

[0086] Optionally, the base layer image fusion weight map can be expressed by the following formula:

[0087]

[0088] Where L is the characteristic distribution map of the base layer image of the infrared video image, W B is the base layer image fusion weight map, λ is a preset parameter, and λ is between 0 and 1.

[0089] Optionally, after determining the base layer image fusion weight map according to the base layer image of the infrared video image, the image fusion module 103 will fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image. The base layer fused image may refer to an image obtained by fusing the base layer image of the infrared video image and the base layer image of the registered visible light video image, which contains large-scale information in the infrared video image and the registered visible light video image and has an overall contrast close to that of the infrared video image. Typically, an image with an overall contrast close to that of the infrared video image will be an image in which the imaging target is prominent.

[0090] Optionally, according to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image, including: according to the base layer image fusion weight map, weighted summing up the final base layer image of the infrared video image and the final base layer image of the registered visible light video image to obtain a base layer fused image.

[0091] Optionally, according to the base layer image fusion weight map, weighted sum is performed on the base layer image of the final infrared video image and the base layer image of the final registered visible light video image to obtain a base layer fusion image, including: performing the following operations for each pixel point in the base layer image of the final infrared video image: updating the pixel value of the pixel point to i WB ×iBI +(1-i WB )×i BV ; where i BI is the pixel value of the pixel, i WB is the pixel value of the pixel point located at the same position as the pixel point in the base layer image fusion weight map, i BV is the pixel value of the pixel point in the base layer image of the final registered visible light video image that is located at the same position as the pixel point. The base layer image of the final infrared video image after completing the pixel value update operation is the base layer fusion image. The obtained base layer fusion image contains the large-scale information in the infrared video image and the registered visible light video image, and the overall contrast is close to the infrared video image, and the imaging target is prominent.

[0092] Optionally, the base layer fused image can be expressed by the following formula:

[0093] B F =W B B I +(1-W B )B V ,

[0094] Among them, W B is the base layer image fusion weight map, B I is the base layer image of the final infrared video image, B v is the final base layer image of the registered visible light video image, B F Fuse the image to the base layer.

[0095] Optionally, after obtaining the base layer image and detail layer image of the infrared video image and the base layer image and detail layer image of the registered visible light video image, the image fusion module 103 will determine the detail layer image fusion weight map based on the infrared video image and the detail layer salient image of the registered visible light video image.

[0096] Optionally, the detail layer salient images of the infrared video image and the registered visible light video image may be images generated based on the detail layer images of the infrared video image and the detail layer images of the registered visible light video image, and contain salient features in the detail layer images of the infrared video image and the detail layer images of the registered visible light video image.

[0097] Optionally, the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image. The fusion weight map corresponding to the detail layer image of the infrared video image may be an image generated based on the detail layer saliency image of the infrared video image and the registered visible light video image, and each pixel value of the infrared video image is between 0 and 1, and the size of the pixel value of each pixel is used to characterize the importance of the pixel points located at the same position in the detail layer image of the infrared video image. The fusion weight map corresponding to the detail layer image of the registered visible light video image may be an image generated based on the detail layer saliency image of the infrared video image and the registered visible light video image, and each pixel value of the registered visible light video image is between 0 and 1, and the size of the pixel value of each pixel is used to characterize the importance of the pixel points located at the same position in the registered visible light video image. According to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are weightedly summed to obtain a detail layer fused image containing texture information and detail information in the infrared video image and the registered visible light video image.

[0098] Optionally, a detail layer image fusion weight map is determined based on the detail layer saliency image of the infrared video image and the registered visible light video image, including: processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain the detail layer saliency image of the infrared video image and the registered visible light video image; performing edge window guided filtering processing on the detail layer image of the infrared video image using a preset edge window guided filter, taking the infrared video image and the detail layer saliency image of the registered visible light video image as guided images, to obtain the saliency image of the detail layer image of the infrared video image. ; Through a preset edge window guided filter, with the infrared video image and the detail layer salient image of the registered visible light video image as guided images, the detail layer image of the registered visible light video image is subjected to edge window guided filtering processing to obtain the salient image of the registered visible light video image; the salient image of the detail layer image of the infrared video image and the salient image of the registered visible light video image are normalized to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0099] Optionally, the saliency detection function may be a function set in the image fusion module 103 for processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image to obtain the detail layer salient image of the infrared video image and the registered visible light video image. The image fusion module 103 may call the saliency detection function, and process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through the saliency detection function to obtain the detail layer salient image of the infrared video image and the registered visible light video image.

[0100] Optionally, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are processed by a saliency detection function to obtain the detail layer saliency images of the infrared video image and the registered visible light video image, including: calling the saliency detection function, and performing the following operation on each pixel point in the detail layer image of the infrared video image by the saliency detection function: judging whether the absolute value of the pixel value of the pixel point is greater than the absolute value of the pixel value of the pixel point in the detail layer image of the registered visible light video image located at the same position as the pixel point; if so, updating the pixel value of the pixel point to the absolute value of the difference between the pixel value of the pixel point and the pixel value of the pixel point in the detail layer image of the registered visible light video image located at the same position as the pixel point; if not, updating the pixel value of the pixel point to 0. The detail layer image of the infrared video image after completing the pixel value update operation is the detail layer saliency image of the infrared video image and the registered visible light video image.

[0101] Optionally, the detail layer salient image of the infrared video image and the registered visible light video image may be expressed by the following formula:

[0102]

[0103] Wherein, P is the detail layer salient image of the infrared video image and the registered visible light video image, D I is the detail layer image of the infrared video image, D v It is a detail layer image of the registered visible light video image.

[0104] Optionally, the salient image of the detail layer image of the infrared video image may refer to an image generated based on the detail layer image of the infrared video image and containing salient features in the detail layer image of the infrared video image. The preset side window guided filter may be a filter provided in the image fusion module 103 for performing side window guided filtering on the image. The image fusion module 103 may use a preset side window guided filter to perform side window guided filtering on the detail layer image of the infrared video image with the infrared video image and the salient image of the detail layer of the registered visible light video image as guided images, to obtain the detail layer image of the infrared video image after the side window guided filtering. The detail layer image of the infrared video image after the side window guided filtering is an image containing salient features in the detail layer image of the infrared video image, that is, the salient image of the detail layer image of the infrared video image.

[0105] Optionally, the salient image of the detail layer image of the infrared video image can be expressed by the following formula:

[0106]

[0107] in, is the salient image of the detail layer image of the infrared video image, D I is the detail layer image of the infrared video image, P is the detail layer salient image of the infrared video image and the registered visible light video image, GuidedFilter() represents the edge window guided filtering operation performed by the preset edge window guided filter, σ SDI is the standard deviation of the pixel values ​​of each pixel in the detail layer image of the infrared video image, σ r is the preset control range weight.

[0108] Optionally, the salient image of the detail layer image of the registered visible light video image may refer to an image generated based on the detail layer image of the registered visible light video image and containing salient features in the detail layer image of the registered visible light video image. The image fusion module 103 may use a preset side window guided filter to perform side window guided filtering on the detail layer image of the registered visible light video image with the registered visible light video image and the detail layer salient image of the registered visible light video image as guided images, to obtain the detail layer image of the registered visible light video image after the side window guided filtering. The detail layer image of the registered visible light video image after the side window guided filtering is an image containing salient features in the detail layer image of the registered visible light video image, that is, the salient image of the detail layer image of the registered visible light video image.

[0109] Optionally, the salient image of the detail layer image of the registered visible light video image may be expressed by the following formula:

[0110]

[0111] in, is the salient image of the detail layer image of the registered visible light video image, D V is the detail layer image of the registered visible light video image, P is the detail layer salient image of the infrared video image and the registered visible light video image, GuidedFilter() represents the edge window guided filtering operation performed by the preset edge window guided filter, σ SDV is the standard deviation of the pixel values ​​of each pixel in the detail layer image of the registered visible light video image, σ r is the preset control range weight.

[0112] Optionally, the image fusion module 103 may perform normalization processing on the salient image of the detail layer image of the infrared video image to obtain a fusion weight map corresponding to the detail layer image of the infrared video image.

[0113] Optionally, normalizing the salient image of the detail layer image of the infrared video image to obtain a fusion weight map corresponding to the detail layer image of the infrared video image includes: performing the following operations on each pixel point in the salient image of the detail layer image of the infrared video image: updating the pixel value of the pixel point to in, is the pixel value before updating. is the pixel value of the pixel point in the salient image of the detail layer image of the registered visible light video image, which is located at the same position as the pixel point. The salient image of the detail layer image of the infrared video image after completing the pixel value update operation is the fusion weight map corresponding to the detail layer image of the infrared video image.

[0114] Optionally, the fusion weight map corresponding to the detail layer image of the infrared video image can be expressed by the following formula:

[0115]

[0116] in, is a fusion weight map corresponding to the detail layer image of the infrared video image, is a salient image of the detail layer image of the infrared video image, The salient image is a detail layer image of the registered visible light video image.

[0117] Optionally, the image fusion module 103 may perform normalization processing on the salient image of the detail layer image of the registered visible light video image to obtain a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0118] Optionally, normalizing the salient image of the detail layer image of the registered visible light video image to obtain a fusion weight map corresponding to the detail layer image of the registered visible light video image includes: performing the following operations for each pixel point in the salient image of the detail layer image of the registered visible light video image: updating the pixel value of the pixel point to in, is the pixel value before updating. is the pixel value of the pixel point in the salient image of the detail layer image of the infrared video image that is located at the same position as the pixel point. After completing the pixel value update operation, the salient image of the detail layer image of the registered visible light video image is the fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0119] Optionally, the fusion weight map corresponding to the detail layer image of the registered visible light video image can be expressed by the following formula:

[0120]

[0121] in, is a fusion weight map corresponding to the detail layer image of the registered visible light video image, is a salient image of the detail layer image of the infrared video image, The salient image is a detail layer image of the registered visible light video image.

[0122] Optionally, after determining the detail layer image fusion weight map based on the detail layer saliency image of the infrared video image and the registered visible light video image, the image fusion module 103 fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fused image. The detail layer fused image may refer to an image obtained by fusing the base layer image of the infrared video image and the detail layer image of the registered visible light video image, which contains texture information and detail information in the infrared video image and the registered visible light video image.

[0123] Optionally, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image, including: according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, weighted summation of the detail layer image of the infrared video image and the detail layer image of the registered visible light video image is performed to obtain a detail layer fused image.

[0124] Optionally, according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, weighted summation is performed on the detail layer image of the infrared video image and the detail layer image of the registered visible light video image to obtain a detail layer fused image, including: performing the following operations for each pixel point in the detail layer image of the infrared video image: updating the pixel value of the pixel point to Among them, i DI is the pixel value of the pixel, is the pixel value of the pixel point located at the same position as the pixel point in the fusion weight map corresponding to the detail layer image of the infrared video image, i DV is the pixel value of a pixel point located at the same position as the pixel point in the detail layer image of the registered visible light video image, is the pixel value of the pixel point located at the same position as the pixel point in the fusion weight map corresponding to the detail layer image of the registered visible light video image. The detail layer image of the infrared video image after completing the pixel value update operation is the detail layer fusion image. The obtained detail layer fusion image contains the texture information and detail information in the infrared video image and the registered visible light video image.

[0125] Optionally, the detail layer fused image can be expressed by the following formula:

[0126]

[0127] in, is a fusion weight map corresponding to the detail layer image of the infrared video image, is the fusion weight map corresponding to the detail layer image of the registered visible light video image, D I is the detail layer image of the infrared video image, D V is the detail layer image of the registered visible light video image, D F Fuse images for detail layers.

[0128] Optionally, after obtaining the base layer fusion image and the detail layer fusion image, the image fusion module 103 will add the base layer fusion image to the detail layer fusion image to obtain a final fused video image. The final fused video image obtained by adding the base layer fusion image to the detail layer fusion image will contain the large-scale information in the infrared video image and the registered visible light video image, and will contain the texture information and detail information in the infrared video image and the registered visible light video image, and the overall contrast is close to that of the infrared video image, and is an image with a prominent imaging target and clear imaging that fully retains the detail information in the infrared video image and the visible light video image.

[0129] Optionally, the base layer fusion image is added to the detail layer fusion image to obtain a final fusion video image, including: performing the following operations for each pixel point in the base layer fusion image: updating the pixel value of the pixel point to in, is the pixel value of the pixel, is the pixel value of the pixel point in the detail layer fused image that is located at the same position as the pixel point. The base layer fused image after completing the pixel value update operation is the final fused video image. The final fused video image obtained fully retains the detail information in the infrared video image and the visible light video image, and the imaging target is prominent and the imaging is clear.

[0130] Optionally, the final fused video image can be expressed by the following formula:

[0131] F=B F +D F ,

[0132] Among them, B F is the base layer fusion image, D F is the detail layer fused image, and F is the final fused video image.

[0133] Optionally, after the image fusion module 103 adds the base layer fusion image and the detail layer fusion image to obtain a final fusion video image, the image fusion module 103 sends the final fusion video image to the image output module 104 .

[0134] Optionally, the image output module 104 may be a hardware module or a software module for providing the final fused video image obtained each time to a target user. The target user may be a technician for managing the target area. Providing the final fused video image to the target user includes: sending the final fused video image to a terminal device of the target user. The terminal device of the target user may be a terminal device used by the target user.

[0135] In the technical solution of the embodiment of the present invention, after the image acquisition module acquires the infrared video image and the visible light video image corresponding to the target area shot synchronously, the infrared video image and the visible light video image are sent to the image registration module; the image registration module performs image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and sends the infrared video image and the registered visible light video image to the image fusion module; the image fusion module processes the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; the image fusion module processes the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; the image fusion module performs image registration on the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the registered visible light video image; the image fusion module performs image registration on the infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the registered visible light video image. The image fusion module determines the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fusion image; the image fusion module determines the detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; the image fusion module fuses the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain the detail layer fusion image; the image fusion module adds the base layer fusion image and the detail layer fusion image to obtain the final fusion video image, and sends the final fusion video image to the image output module;The image output module provides the received final fused video image to the target user, which solves the problem that the fusion scheme of infrared video images and visible light video images in the related art is easy to lose the detail information in the infrared video images and visible light video images during the fusion process. The infrared video image and the visible light video image corresponding to the target area taken synchronously can be registered based on the image registration module to obtain a registered visible light video image that can be spatially aligned with the infrared video image, thereby ensuring the quality of the subsequent image fusion process. Based on the image fusion module, the infrared video image and the registered visible light video image after image registration can be processed first to obtain a base layer image containing large-scale information in the infrared video image, a base layer image containing large-scale information in the registered visible light video image, a detail layer image containing small-scale information such as texture information and detail information in the infrared video image, and a detail layer image containing small-scale information such as texture information and detail information in the registered visible light video image. Then, the base layer image fusion weight map is used to process the large-scale information in the infrared video image. The base layer image containing the information and the base layer image containing the large-scale information in the registered visible light video image are fused to obtain a base layer fused image containing the large-scale information in the infrared video image and the registered visible light video image and having an overall contrast close to the infrared video image, and having a significant imaging target. The detail layer image containing the small-scale information such as texture information and detail information in the infrared video image and the detail layer image containing the small-scale information such as texture information and detail information in the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image containing the texture information and detail information in the infrared video image and the registered visible light video image. Finally, the base layer fused image and the detail layer fused image are added to obtain a final fused video image with sufficient retention of the detail information in the infrared video image and the visible light video image, with significant imaging targets and clear imaging. Based on the image fusion module, the final fused video image can be provided to the target user. The detail information in the infrared video image and the visible light video image can be fully retained during the fusion process, thereby improving the quality of the image fusion process. ;

[0136] Embodiment 2

[0137] Figure 2 The flowchart of a method for registering and fusing infrared video images and visible light video images provided in the second embodiment of the present invention is applicable to the case where infrared video images and visible light video images taken synchronously are fused to obtain a fused video image. The method can be applied to the image fusion module in the registration and fusion system of infrared video images and visible light video images provided in the above embodiment of the present invention. Figure 2 As shown, the method of this embodiment specifically includes:

[0138] Step 201 : Process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image and a base layer image and a detail layer image of the registered visible light video image.

[0139] Optionally, the received infrared video image and the registered visible light video image are processed to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image, including: performing Gaussian filtering on the received infrared video image through a preset Gaussian filter to obtain the base layer image of the infrared video image; performing edge restoration on the base layer image of the infrared video image through a preset rolling guided filter to obtain the final base layer image of the infrared video image; subtracting the infrared video image from the final base layer image of the infrared video image to obtain the detail layer image of the infrared video image; performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter to obtain the base layer image of the registered visible light video image; performing edge restoration on the base layer image of the registered visible light video image through a preset rolling guided filter to obtain the final base layer image of the registered visible light video image; subtracting the registered visible light video image from the final base layer image of the registered visible light video image to obtain the detail layer image of the registered visible light video image.

[0140] Step 202: Determine a base layer image fusion weight map according to the base layer image of the infrared video image.

[0141] Optionally, based on the base layer image of the infrared video image, a base layer image fusion weight map is determined, including: normalizing the base layer image of the final infrared video image to obtain a normalized base layer image of the infrared video image; determining a feature distribution map of the base layer image of the infrared video image based on the base layer image of the normalized infrared video image; and determining a base layer image fusion weight map based on the feature distribution map of the base layer image of the infrared video image.

[0142] Step 203: According to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image.

[0143] Optionally, according to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image, including: according to the base layer image fusion weight map, weighted summing up the final base layer image of the infrared video image and the final base layer image of the registered visible light video image to obtain a base layer fused image.

[0144] Step 204: Determine a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image.

[0145] Optionally, a detail layer image fusion weight map is determined based on the detail layer saliency image of the infrared video image and the registered visible light video image, including: processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain the detail layer saliency image of the infrared video image and the registered visible light video image; performing edge window guided filtering processing on the detail layer image of the infrared video image using a preset edge window guided filter, taking the infrared video image and the detail layer saliency image of the registered visible light video image as guided images, to obtain the saliency image of the detail layer image of the infrared video image. ; Through a preset edge window guided filter, with the infrared video image and the detail layer salient image of the registered visible light video image as guided images, the detail layer image of the registered visible light video image is subjected to edge window guided filtering processing to obtain the salient image of the registered visible light video image; the salient image of the detail layer image of the infrared video image and the salient image of the registered visible light video image are normalized to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0146] Step 205 : According to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain a detail layer fused image.

[0147] Optionally, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image, including: according to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, weighted summation of the detail layer image of the infrared video image and the detail layer image of the registered visible light video image is performed to obtain a detail layer fused image.

[0148] Step 206: Add the base layer fused image and the detail layer fused image to obtain a final fused video image, and send the final fused video image to an image output module.

[0149] The technical solution of the embodiment of the present invention processes the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image; then, according to the base layer image of the infrared video image, a base layer image fusion weight map is determined; according to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain the base layer fused image; according to the infrared video image and the detail layer salient image of the registered visible light video image, the detail layer image fusion weight map is determined. Figure; according to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain the detail layer fusion image; finally, the base layer fusion image is added to the detail layer fusion image to obtain the final fused video image, and the final fused video image is sent to the image output module. The infrared video image after image registration and the registered visible light video image can be processed first to obtain the base layer image containing large-scale information in the infrared video image, the base layer image containing large-scale information in the registered visible light video image, the texture information in the infrared video image and The detail layer image containing small-scale information such as detail information, the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image, and then the base layer image containing large-scale information in the infrared video image and the base layer image containing large-scale information in the registered visible light video image are fused according to the base layer image fusion weight map to obtain a base layer fused image containing large-scale information in the infrared video image and the registered visible light video image and with an overall contrast close to that of the infrared video image, and with a significant imaging target. The detail layer image containing small-scale information such as texture information and detail information in the infrared video image and the detail layer image containing small-scale information such as texture information and detail information in the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image containing texture information and detail information in the infrared video image and the registered visible light video image. Finally, the base layer fused image is added to the detail layer fused image to obtain a final fused video image with significant imaging targets and clear imaging that fully retains the detail information in the infrared video image and the visible light video image. The detail information in the infrared video image and the visible light video image can be fully retained during the fusion process, thereby improving the quality of the image fusion process.

[0150] Embodiment 3

[0151] Figure 3 The structure diagram of a device for registering and fusing infrared video images and visible light video images provided in the third embodiment of the present invention is shown. The device can be configured in the image fusion module of the system for registering and fusing infrared video images and visible light video images provided in the above embodiment of the present invention. Figure 3 As shown, the device includes: an image processing unit 301, a first weight map determining unit 302, a base layer image fusion unit 303, a second weight map determining unit 304, a detail layer image fusion unit 305 and a fused video image determining unit 306.

[0152] The image processing unit 301 is used to process the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image and the base layer image and detail layer image of the registered visible light video image; the first weight map determination unit 302 is used to determine the base layer image fusion weight map according to the base layer image of the infrared video image; the base layer image fusion unit 303 is used to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fusion image ; A second weight map determining unit 304 is used to determine a detail layer image fusion weight map according to the detail layer salient image of the infrared video image and the registered visible light video image; a detail layer image fusion unit 305 is used to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fused image; a fused video image determining unit 306 is used to add the base layer fused image and the detail layer fused image to obtain a final fused video image, and send the final fused video image to the image output module.

[0153] The technical solution of the embodiment of the present invention processes the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image; then, according to the base layer image of the infrared video image, a base layer image fusion weight map is determined; according to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain the base layer fused image; according to the infrared video image and the detail layer salient image of the registered visible light video image, the detail layer image fusion weight map is determined. Figure; according to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain the detail layer fusion image; finally, the base layer fusion image is added to the detail layer fusion image to obtain the final fused video image, and the final fused video image is sent to the image output module. The infrared video image after image registration and the registered visible light video image can be processed first to obtain the base layer image containing large-scale information in the infrared video image, the base layer image containing large-scale information in the registered visible light video image, the texture information in the infrared video image and The detail layer image containing small-scale information such as detail information, the detail layer image containing texture information and small-scale information such as detail information in the registered visible light video image, and then the base layer image containing large-scale information in the infrared video image and the base layer image containing large-scale information in the registered visible light video image are fused according to the base layer image fusion weight map to obtain a base layer fused image containing large-scale information in the infrared video image and the registered visible light video image and with an overall contrast close to that of the infrared video image, and with a significant imaging target. The detail layer image containing small-scale information such as texture information and detail information in the infrared video image and the detail layer image containing small-scale information such as texture information and detail information in the registered visible light video image are fused according to the detail layer image fusion weight map to obtain a detail layer fused image containing texture information and detail information in the infrared video image and the registered visible light video image. Finally, the base layer fused image is added to the detail layer fused image to obtain a final fused video image with significant imaging targets and clear imaging that fully retains the detail information in the infrared video image and the visible light video image. The detail information in the infrared video image and the visible light video image can be fully retained during the fusion process, thereby improving the quality of the image fusion process.

[0154] In an optional implementation of the embodiment of the present invention, optionally, the image processing unit 301 is specifically used to: perform Gaussian filtering processing on the received infrared video image through a preset Gaussian filter to obtain a base layer image of the infrared video image; perform edge restoration on the base layer image of the infrared video image through a preset rolling guided filter to obtain a final base layer image of the infrared video image; subtract the infrared video image from the base layer image of the final infrared video image to obtain a detail layer image of the infrared video image; perform Gaussian filtering processing on the received registered visible light video image through a preset Gaussian filter to obtain a base layer image of the registered visible light video image; perform edge restoration on the base layer image of the registered visible light video image through a preset rolling guided filter to obtain a final base layer image of the registered visible light video image; subtract the registered visible light video image from the base layer image of the final registered visible light video image to obtain a detail layer image of the registered visible light video image.

[0155] In an optional implementation of the embodiment of the present invention, optionally, the first weight map determination unit 302 is specifically used to: normalize the base layer image of the final infrared video image to obtain the base layer image of the normalized infrared video image; determine the feature distribution map of the base layer image of the infrared video image based on the base layer image of the normalized infrared video image; determine the base layer image fusion weight map based on the feature distribution map of the base layer image of the infrared video image.

[0156] In an optional implementation of an embodiment of the present invention, optionally, the base layer image fusion unit 303 is specifically used to: perform weighted summation of the final base layer image of the infrared video image and the final base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image.

[0157] In an optional implementation of an embodiment of the present invention, optionally, the second weight map determination unit 304 is specifically used to: process the detail layer image of the infrared video image and the detail layer image of the registered visible light video image through a saliency detection function to obtain the detail layer saliency images of the infrared video image and the registered visible light video image; perform edge window guided filtering processing on the detail layer image of the infrared video image using the infrared video image and the detail layer saliency images of the registered visible light video image as guide images through a preset edge window guided filter to obtain the saliency image of the detail layer image of the infrared video image; An edge window guided filter is used, which uses the infrared video image and the detail layer salient image of the registered visible light video image as guiding images, performs edge window guided filtering on the detail layer image of the registered visible light video image, and obtains the salient image of the registered visible light video image; normalizes the salient image of the detail layer image of the infrared video image and the salient image of the registered visible light video image, and obtains a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

[0158] In an optional implementation of an embodiment of the present invention, optionally, the detail layer image fusion unit 305 is specifically used to: perform weighted summation of the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image to obtain a detail layer fused image.

[0159] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0160] Embodiment 4

[0161] Figure 4 The schematic diagram of the structure of the electronic device 10 that can be used to implement the registration and fusion method of the infrared video image and the visible light video image according to the embodiment of the present invention is shown. The components shown herein, their connections and relationships, and their functions are only for example, and are not intended to limit the implementation of the present invention described and / or required herein.

[0162] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program built from the storage unit 18 to the random access memory (RAM) 13. In the random access memory (RAM) 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory (ROM) 12 and the random access memory (RAM) 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0163] A number of components in the electronic device 10 are connected to an input / output (I / O) interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0164] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the registration and fusion method of infrared video images and visible light video images.

[0165] In some embodiments, the registration and fusion method of infrared video images and visible light video images can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via a read-only memory (ROM) 12 and / or a communication unit 19. When the computer program is built into a random access memory (RAM) 13 and executed by the processor 11, one or more steps of the registration and fusion method of infrared video images and visible light video images described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the registration and fusion method of infrared video images and visible light video images in any other appropriate manner (e.g., by means of firmware).

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The computer program for implementing the registration and fusion method of infrared video images and visible light video images of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, partially on the machine and partially on a remote machine as an independent software package, or completely on a remote machine or server.

[0168] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0171] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A registration and fusion system for infrared video images and visible light video images, characterized in that: include: Image acquisition module, image registration module, image fusion module and image output module; The image acquisition module is used to send the infrared video image and the visible light video image to the image registration module after acquiring the infrared video image and the visible light video image corresponding to the target area shot synchronously; The image registration module is used to perform image registration on the received infrared video image and the visible light video image to obtain a registered visible light video image registered with the infrared video image, and send the infrared video image and the registered visible light video image to the image fusion module; The image fusion module is used to process the received infrared video image and the registered visible light video image to obtain the base layer image and detail layer image of the infrared video image, and the base layer image and detail layer image of the registered visible light video image; determine the base layer image fusion weight map according to the base layer image of the infrared video image; fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain the base layer fused image; determine the detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain the detail layer fused image; add the base layer fused image and the detail layer fused image to obtain the final fused video image, and send the final fused video image to the image output module; The image output module is used to provide the received final fused video image to a target user.

2. A registration and fusion method for infrared video images and visible light video images, applied to the image fusion module in the registration and fusion system for infrared video images and visible light video images as claimed in claim 1, characterized in that: include: Processing the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; Determining a base layer image fusion weight map according to the base layer image of the infrared video image; According to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image; Determining a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; According to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain a detail layer fused image; The base layer fused image and the detail layer fused image are added to obtain a final fused video image, and the final fused video image is sent to an image output module.

3. The registration and fusion method of infrared video images and visible light video images according to claim 2 is characterized in that: The received infrared video image and the registered visible light video image are processed to obtain a base layer image and a detail layer image of the infrared video image and a base layer image and a detail layer image of the registered visible light video image, including: Performing Gaussian filtering on the received infrared video image through a preset Gaussian filter to obtain a base layer image of the infrared video image; Performing edge restoration on the base layer image of the infrared video image by using a preset rolling guided filter to obtain a final base layer image of the infrared video image; Subtracting the infrared video image from a final base layer image of the infrared video image to obtain a detail layer image of the infrared video image; Performing Gaussian filtering on the received registered visible light video image through a preset Gaussian filter to obtain a base layer image of the registered visible light video image; Performing edge restoration on the base layer image of the registered visible light video image by using a preset rolling guided filter to obtain a final base layer image of the registered visible light video image; The registered visible light video image and a final base layer image of the registered visible light video image are subtracted to obtain a detail layer image of the registered visible light video image.

4. The registration and fusion method of infrared video images and visible light video images according to claim 3 is characterized in that: Determining a base layer image fusion weight map according to the base layer image of the infrared video image includes: Normalizing the final base layer image of the infrared video image to obtain a normalized base layer image of the infrared video image; Determine a feature distribution map of the base layer image of the infrared video image according to the normalized base layer image of the infrared video image; A base layer image fusion weight map is determined according to the feature distribution map of the base layer image of the infrared video image.

5. The registration and fusion method of infrared video images and visible light video images according to claim 3 is characterized in that: According to the base layer image fusion weight map, the base layer image of the infrared video image and the base layer image of the registered visible light video image are fused to obtain a base layer fused image, including: According to the base layer image fusion weight map, a weighted sum is performed on the final base layer image of the infrared video image and the final base layer image of the registered visible light video image to obtain a base layer fused image.

6. The registration and fusion method of infrared video images and visible light video images according to claim 2, characterized in that: Determining a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image, comprising: Processing the detail layer image of the infrared video image and the detail layer image of the registered visible light video image by using a saliency detection function to obtain detail layer saliency images of the infrared video image and the registered visible light video image; By using a preset edge window guided filter, the infrared video image and the detail layer salient image of the registered visible light video image are used as guided images, and an edge window guided filtering process is performed on the detail layer image of the infrared video image to obtain a salient image of the detail layer image of the infrared video image; By using a preset edge window guided filter, the infrared video image and the detail layer salient image of the registered visible light video image are used as guide images, and an edge window guided filtering process is performed on the detail layer image of the registered visible light video image to obtain a salient image of the registered visible light video image; The salient image of the detail layer image of the infrared video image and the salient image of the registered visible light video image are normalized to obtain a detail layer image fusion weight map; wherein the detail layer image fusion weight map includes a fusion weight map corresponding to the detail layer image of the infrared video image and a fusion weight map corresponding to the detail layer image of the registered visible light video image.

7. The registration and fusion method of infrared video images and visible light video images according to claim 6, characterized in that: According to the detail layer image fusion weight map, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are fused to obtain a detail layer fused image, including: According to the fusion weight map corresponding to the detail layer image of the infrared video image and the fusion weight map corresponding to the detail layer image of the registered visible light video image, the detail layer image of the infrared video image and the detail layer image of the registered visible light video image are weightedly summed to obtain a detail layer fused image.

8. A registration and fusion device for infrared video images and visible light video images, configured in an image fusion module in the registration and fusion system for infrared video images and visible light video images as claimed in claim 1, characterized in that: include: An image processing unit, configured to process the received infrared video image and the registered visible light video image to obtain a base layer image and a detail layer image of the infrared video image, and a base layer image and a detail layer image of the registered visible light video image; A first weight map determining unit, configured to determine a base layer image fusion weight map according to the base layer image of the infrared video image; A base layer image fusion unit, configured to fuse the base layer image of the infrared video image and the base layer image of the registered visible light video image according to the base layer image fusion weight map to obtain a base layer fused image; A second weight map determining unit, configured to determine a detail layer image fusion weight map according to the infrared video image and the detail layer salient image of the registered visible light video image; A detail layer image fusion unit, configured to fuse the detail layer image of the infrared video image and the detail layer image of the registered visible light video image according to the detail layer image fusion weight map to obtain a detail layer fused image; The fused video image determination unit is used to add the base layer fused image and the detail layer fused image to obtain a final fused video image, and send the final fused video image to an image output module.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the registration and fusion method of infrared video images and visible light video images described in any one of claims 2-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the registration and fusion method of infrared video images and visible light video images according to any one of claims 2 to 7 when executed.

Images