Real-time dual-spectrum infrared and visible light fusion method, device, equipment and medium

Abstract

The invention provides a real-time dual-spectrum infrared and visible light fusion method, device and equipment and a medium. The method comprises the following steps: acquiring an input initial infrared light image and an input initial visible light image in real time; distortion correction is carried out on the initial infrared light image and the initial visible light image; registering the infrared light image to be registered according to the spatial geometrical relationship between the infrared camera and the visible light camera established in the off-line calibration stage to generate an infrared image to be fused; decomposing the to-be-fused infrared image and the to-be-fused visible light image by adopting a Laplacian pyramid, generating a detail layer fusion image for the detail layer image by adopting an adaptive weight fusion mode, and generating a base layer fusion image for the base layer image by adopting an average value mode; and carrying out inverse transformation reconstruction on the detail layer fusion image and the base layer fusion image by adopting a Laplacian pyramid to generate a final fusion image. According to the method, the calculation efficiency of the algorithm is deeply optimized, and real-time and high-quality fusion of infrared and visible light videos is realized.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image processing technology, and in particular to a method, apparatus, device and medium for real-time dual-spectrum infrared and visible light fusion. Background Technology

[0002] In numerous applications such as military, security, autonomous driving, and medical diagnostics, single-spectral images (such as visible light or infrared) often have information limitations. Visible light images typically provide rich texture details and high spatial resolution, but their image quality deteriorates significantly under harsh environmental conditions such as low light, fog, rain, and snow. Conversely, infrared images capture the thermal radiation information of objects themselves, are unaffected by visible light illumination, are robust to camouflage, and can effectively penetrate obstacles such as smoke and darkness. However, their spatial resolution is relatively low, they lack contour details, and they have difficulty distinguishing objects at the same temperature. Therefore, effectively fusing infrared and visible light images can combine the advantages of both to generate a more comprehensive, robust, and human-visually-compatible fused image, significantly improving the system's perception capabilities in complex environments.

[0003] However, infrared and visible light cameras have inherent differences in optical characteristics, imaging principles, field of view size, and installation location, resulting in geometric distortion and field-of-view mismatch in the images they acquire. Direct pixel-level or feature-level fusion easily leads to artifacts such as ghosting, blurring, and information loss in the fusion results, severely impacting fusion quality and subsequent applications. Furthermore, existing image fusion methods generally face significant computational challenges when processing real-time video streams, making it difficult to meet real-time requirements.

[0004] Existing image fusion techniques are mainly divided into pixel-level, feature-level, and decision-level fusion. Pixel-level fusion (such as weighted averaging, wavelet transform, and Laplacian pyramid) typically operates directly on image pixels, but most methods are sensitive to geometric inconsistencies in the image and have high computational complexity, making them difficult to apply directly to real-time video streams. Feature-level fusion (such as SIFT and SURF feature point matching) can handle certain geometric transformations, but its feature extraction and matching processes are computationally intensive, and its performance is limited in low-texture regions, making it difficult to guarantee real-time performance and global consistency. Decision-level fusion operates at a higher level of abstraction and does not directly process image pixels.

[0005] To address the geometric differences in bispectral images, some methods attempt to perform image registration before fusion. However, these registration methods are often independent of the fusion algorithm and tend to focus on homography or affine transformations, failing to adequately consider the inherent nonlinear distortions of the camera. Furthermore, existing solutions lack integrated, high-efficiency optimization strategies across the entire chain of distortion correction, field-of-view matching, and image fusion. In particular, they fail to effectively utilize the parallel computing capabilities and vectorized instruction sets of modern processors (such as multi-core CPUs and GPUs), resulting in overall processing speeds that cannot meet the demands of real-time video fusion. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a method, apparatus, device, and medium for real-time dual-spectral infrared and visible light fusion. Camera distortion parameters and dual-spectral geometric transformation relationships are obtained through offline high-precision calibration. In the online processing stage, efficient distortion correction, geometric registration, and a fusion algorithm based on multi-scale decomposition and adaptive weighting are combined, while the computational efficiency of the algorithm is deeply optimized to meet the processing requirements of real-time video streams.

[0007] In a first aspect, the present invention proposes a method for real-time dual-spectrum infrared and visible light fusion, specifically including the following steps: The initial infrared and visible light images are acquired in real time. Based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, the initial infrared light image and the initial visible light image are distorted to generate the distorted infrared light image to be registered and the visible light image to be fused. The infrared image to be registered is registered based on the spatial geometric relationship between the infrared camera and the visible light camera established during the offline calibration phase to generate the infrared image to be fused. The infrared image and the visible light image to be fused are decomposed using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused, respectively. An adaptive weighted fusion method is used to generate a fused detail layer image from the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and an average value method is used to generate a fused base layer image from the base layer images in the first Laplacian pyramid and the second Laplacian pyramid. The detailed layer fused image and the basic layer fused image are reconstructed using the inverse Laplacian pyramid transform to generate the final fused image.

[0008] Furthermore, the process of performing distortion correction on the initial infrared and visible light images based on the distortion parameters of the infrared and visible light cameras obtained during the offline calibration phase, and generating the corrected infrared image to be registered and the visible light image to be fused, specifically includes: Distortion calibration was performed on both the infrared and visible light cameras during the offline calibration phase to obtain the distortion parameters of the infrared camera. and visible light camera distortion parameters ; Based on the distortion model function and the acquired initial infrared image and the initial visible light image The intensity value is used to generate the distortion-corrected image, represented as: in, This represents the corrected infrared image to be registered. This represents the corrected visible light image to be fused. This is the distortion model function for infrared cameras. For visible light camera distortion model functions, each represents the coordinates of a distortion-free pixel. Mapped to the corresponding coordinates in the distorted image Represents the initial infrared image coordinate , Represents the initial visible light image In coordinates Interpolation is performed at the location to obtain the intensity value. Infrared image to be registered pixel domain, Represents the visible light images to be fused The pixel field.

[0009] Furthermore, the step of registering the infrared image to be registered and generating the infrared image to be fused based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage specifically includes: In the offline calibration stage, image samples based on infrared and visible light are acquired, and the feature points in the image samples are distorted. The feature points in the distorted infrared image are then matched with the corresponding feature points in the visible light image to generate a set of matched distortion-free feature points. Based on the distortion-free feature point set pairs and the obtained infrared camera intrinsic parameter matrix and visible light camera intrinsic parameter matrix, the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system is estimated. A pixel coordinate mapping model between the infrared image and the visible light image is established based on the relative pose. The infrared image to be registered is registered according to the pixel coordinate mapping model to generate an infrared image to be fused.

[0010] Furthermore, the process of estimating the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system specifically includes a nonlinear optimization process that minimizes the reprojection error function to estimate the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system.

[0011] Furthermore, the process of decomposing the infrared image to be fused and the visible light image to be fused using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused specifically includes: decomposing the infrared image to be fused using a Laplacian pyramid to generate a first Laplacian pyramid based on a base layer image and multiple detail layer images of the infrared image to be fused; and decomposing the visible light image to be fused using a Laplacian pyramid to generate a second Laplacian pyramid based on a base layer image and multiple detail layer images of the visible light image to be fused.

[0012] Furthermore, the process of generating a detail layer fused image by using an adaptive weight fusion method on the detail layer images in the first and second Laplacian pyramids specifically includes: Obtain the local activity of detail layer images at any level of the first and second Laplacian pyramids, and generate adaptive fusion weights for infrared and visible light images; For each detail layer, the adaptive fusion weights are used to fuse and generate a detail layer fused image.

[0013] Furthermore, the process of reconstructing the detail layer fused image and the base layer fused image using the inverse Laplacian pyramid transformation to generate the final fused image also includes converting the floating-point fusion result of the final fused image into an 8-bit grayscale image as the image output result.

[0014] In a second aspect, the present invention also proposes a device for real-time dual-spectral infrared and visible light fusion to perform the method as described in the first aspect, the device comprising: The image acquisition module is used to acquire the initial infrared light image and the initial visible light image in real time; The distortion correction module is used to correct the distortion of the initial infrared light image and the initial visible light image based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, and generate the corrected infrared light image to be registered and the visible light image to be fused. The registration generation module is used to register the infrared light image to be registered and generate the infrared image to be fused based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage. The image decomposition module is used to decompose the infrared image to be fused and the visible light image to be fused using a Laplacian pyramid, so as to obtain the first Laplacian pyramid of the infrared image to be fused and the second Laplacian pyramid of the visible light image to be fused, respectively. The layered fusion module is used to generate a fused detail layer image by using an adaptive weighted fusion method on the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and to generate a fused base layer image by using an average value method on the base layer images in the first Laplacian pyramid and the second Laplacian pyramid. The fusion output module is used to reconstruct the detail layer fused image and the base layer fused image using the inverse Laplacian pyramid transform to generate the final fused image.

[0015] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in the first aspect.

[0016] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0017] The beneficial effects of this invention are: (1) High efficiency in real-time fusion performance: The fusion method of the present invention significantly improves the real-time fusion speed of infrared and visible light video.

[0018] (2) Enhanced visual effects and user comfort: From the perspective of visual effects, the fused video images have been significantly improved in terms of clarity, contrast and detail. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the real-time dual-spectrum infrared and visible light fusion method proposed in this invention; Figure 2 This is a schematic diagram of the device framework for real-time dual-spectrum infrared and visible light fusion proposed in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown in the figure, this embodiment proposes a method for real-time dual-spectrum infrared and visible light fusion, which specifically includes the following steps: S101. Real-time acquisition of the initial infrared light image and the initial visible light image; S102. Based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, the initial infrared light image and the initial visible light image are distorted to generate the distorted infrared light image to be registered and the visible light image to be fused. S103. Based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage, the infrared light image to be registered is registered to generate an infrared image to be fused. S104. The infrared image to be fused and the visible light image to be fused are decomposed using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused, respectively; S105. An adaptive weighted fusion method is used to generate a detail layer fusion image from the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and an average value method is used to generate a base layer fusion image from the base layer images in the first Laplacian pyramid and the second Laplacian pyramid; S106. The detail layer fused image and the base layer fused image are reconstructed using the inverse Laplacian pyramid transform to generate the final fused image.

[0022] In this embodiment, the image fusion method mainly includes two stages: an offline calibration stage and an online processing stage. It should be noted that the above steps in this embodiment combine camera distortion calibration and dual-spectral geometric calibration in the offline calibration stage, as well as the fusion processing of real-time acquired infrared and visible light images, which will be described in detail below.

[0023] The offline calibration stage in this embodiment serves to provide the distortion parameters and geometric transformation relationship between infrared and visible light images required during the online processing stage.

[0024] The offline calibration stage in this embodiment includes camera distortion calibration and dual-spectral geometric calibration.

[0025] (1) Camera distortion calibration In this embodiment, the camera distortion calibration stage includes distortion calibration of the infrared camera and the visible light camera. During the calibration process, multiple sets of images are acquired at different positions and orientations using a standard calibration board. By detecting feature points on the calibration board, the intrinsic parameters and distortion coefficients of each camera are estimated using a camera distortion model. This process can be represented as follows: For infrared cameras, obtain the set of distortion parameters. ; For visible light cameras, obtain the set of distortion parameters. ; in, It is the intrinsic parameter matrix of the infrared camera. It is the intrinsic parameter matrix of the visible light camera, which includes the focal length and principal point coordinates; These are the distortion parameters of the infrared camera. These are the distortion parameters of a visible light camera, which are used for subsequent distortion correction.

[0026] (2) Dual-spectral geometric calibration In this embodiment, a spatial geometric relationship between the infrared camera and the visible light camera is established through dual-spectral geometric calibration. This allows for precise registration of the infrared image to the coordinate system of the visible light image during online processing, resolving the issue of mismatched fields of view. Since the field of view of the visible light camera is typically larger than that of the infrared camera, this registration process projects the content of the infrared image into the common field of view of the visible light image.

[0027] In this embodiment, infrared and visible light image samples are acquired in the same scene, ensuring that the scene contains a sufficient number of identifiable feature points. Typically, to improve calibration accuracy, a high-contrast planar calibration board (such as a checkerboard or circular array) is used to acquire multiple sets of image pairs under different poses and positions.

[0028] For each acquired image pair, feature point extraction and matching are performed to generate a set of distortion-free feature points. During this process, the aforementioned camera distortion parameters are used to correct the distortion of the feature points, resulting in distortion-free pixel coordinates. For example, for a distorted pixel in an infrared image... , The corrected distortion-free pixels are ,in This represents the two-dimensional coordinates of the distorted pixel in the image coordinate system. Represents the horizontal coordinate. Represents vertical coordinates, This represents the two-dimensional coordinates of a distortion-free pixel in the image coordinate system. Represents the horizontal coordinate. This represents the vertical coordinate; the same applies to visible light images.

[0029] Infrared image after distortion correction and visible light images In the process, significant feature points are detected. For calibration boards, corner points are typically detected. Specifically, a matching relationship is established between feature points in the infrared image and corresponding feature points in the visible light image. For calibration boards, since their structure is known, the matching is intuitive. For unstructured scenes, feature descriptors (such as...) can be used. SURF (etc.) and matching algorithms (such as , (etc.) to find pairs of points with the same name.

[0030] In the relative pose estimation step, the matched N pairs of distortion-free feature points and the intrinsic parameter matrices of each camera are used. and This involves estimating the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system. This is typically accomplished by solving an optimization problem aimed at minimizing the reprojection error.

[0031] In this embodiment, a 3D spatial point is assumed. In the world coordinate system, its coordinates in the infrared camera coordinate system and the visible light camera coordinate system are respectively... and Their relationship is: in, It is a 3×3 rotation matrix representing the rotation from the infrared camera coordinate system to the visible light camera coordinate system. It is a 3×1 translation vector, representing the translation from the origin of the infrared camera coordinate system to the origin of the visible light camera coordinate system.

[0032] The projections of 3D points onto their respective camera image planes (represented using homogeneous coordinates) are: in , As a scale factor, and These are the homogeneous pixel coordinates on a distortion-free image. (Using known matching point pairs) and camera intrinsic parameter matrix and A nonlinear optimization method that minimizes the reprojection error can be used to calculate... and Reprojection error is usually defined as: in and It is a projection function. It refers to the pose of the infrared camera in the world coordinate system. It is the pose of the visible light camera in the world coordinate system. It is the translation of the infrared camera to the world coordinate system. It represents the translation from the visible light camera to the world coordinate system, where N is the number of matching points. It is the point of the i-th matching point in the world coordinate system. and These are the i-th matching point pairs on the infrared and visible light images, respectively. By solving this optimization problem, the poses of the infrared and visible light cameras relative to the same world coordinate system can be obtained, and their relative poses can be derived. and .

[0033] Obtaining relative pose and Afterwards, each pixel of the infrared image can be processed online. Mapped to the corresponding position in the visible light image .

[0034] In this embodiment, the mapping process is divided into the following steps: (1) Distortion correction: the pixels of the infrared image Convert to distortion-free pixel coordinates .

[0035] (2) Normalized planar coordinates: normalize the coordinates of undistorted pixels. Convert to infrared camera coordinate system Normalized plane coordinates under , where coordinates .

[0036] (3) Transform to the visible light camera coordinate system. Assume... Located on the unit depth plane, transform it to the visible light camera coordinate system: .

[0037] (4) Transform to the visible light camera coordinate system. Projecting the image onto the visible light image plane yields the corresponding pixel coordinates. : .

[0038] During online processing, there is no need to perform the aforementioned complex matrix operations in real time. Instead, the offline stage will build a mapping table based on this mapping relationship. For each target visible light image pixel Pre-calculate its corresponding source pixel in the original infrared image The results of this pre-calculation are typically stored as two mapping tables, one for each target pixel. coordinates and Coordinates. In online processing, geometric registration can be quickly completed by simply using these mapping tables in conjunction with interpolation algorithms to resample the infrared image.

[0039] The foregoing content describes the acquisition of distortion parameters and the establishment of geometric transformation relationships between infrared and visible light images during the offline calibration phase.

[0040] In step S101, the initial infrared light image input is acquired in real time. and the initial visible light image Then, through the above-mentioned offline calibration stage, distortion parameters are obtained to correct the distortion of the image, and the infrared light image to be registered is registered based on the pose obtained in the offline calibration stage to generate the infrared image to be fused.

[0041] Based on the distortion model function and the acquired initial infrared image and the initial visible light image The intensity value is used to generate the distortion-corrected image, represented as: in, This represents the corrected infrared image to be registered. This represents the corrected visible light image to be fused. This is the distortion model function for infrared cameras. For visible light camera distortion model functions, each represents the coordinates of a distortion-free pixel. Mapped to the corresponding coordinates in the distorted image Represents the initial infrared image coordinate , Represents the initial visible light image In coordinates Interpolation is performed at the location to obtain the intensity value. Infrared image to be registered pixel domain, Represents the visible light images to be fused The pixel field.

[0042] Based on the spatial geometric relationship between the infrared camera and the visible light camera established during the offline calibration phase, the infrared image to be registered is... Transform to visible light image In an aligned coordinate system, the infrared images to be fused are registered. , is represented as: Registered infrared image Size and visible light image same.

[0043] To achieve the registered infrared image and visible light images The process of fusing the infrared image to be fused and the visible light image to be fused by decomposing them using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused, specifically includes: decomposing the infrared image to be fused using a Laplacian pyramid to generate a first Laplacian pyramid based on a base layer image and multiple detail layer images of the infrared image to be fused; and decomposing the visible light image to be fused using a Laplacian pyramid to generate a second Laplacian pyramid based on a base layer image and multiple detail layer images of the visible light image to be fused.

[0044] In this embodiment, the decomposed infrared image is represented as: This represents the detail layer image of the k-th layer of an infrared image. This represents the Gaussian image, the highest layer of an infrared image, also known as the base layer image. This represents the detail layer image of the k-th layer of a visible light image. The Gaussian image representing the highest layer of the visible light image is the base layer image. The Gaussian filtering and downsampling operations in the pyramid decomposition process can be parallelized using OpenMP, and the filtering operation can be further accelerated using the SIMD instruction set.

[0045] For each Laplace pyramid level k, the detailed layer and The local activity (saliency) is calculated. In this embodiment, local gradient energy is used as the activity measure of the detail layer because it can effectively reflect the texture and edge information of the image.

[0046] Local gradient energy calculation: For a pixel p in image I, its local gradient energy E(p) can be defined as: in, and These represent gradient operators (such as Sobel or Prewitt operators) in the horizontal and vertical directions, respectively. This represents the grayscale value of image I at position i.

[0047] Activity calculation: This measure assesses the saliency of the fundamental components of an image in a local region. In this embodiment, the local energy gradient of the k-th layer after Laplacian pyramid decomposition is used as the activity measure.

[0048] Based on the activity level of infrared and visible light images at each level, the fusion weights for that level are calculated, as follows: in, Represents the fusion weights of infrared images. Represents the fusion weights of visible light images. It is a very small integer used to prevent the denominator from being zero and to keep the value stable.

[0049] Gradient calculation and activity map generation are pixel-level operations that can be efficiently computed using OpenMP parallelization and SIMD vectorized instruction sets (such as AVX / SSE).

[0050] For each detail layer, the adaptive fusion weights are used to generate a fused detail layer image, represented as follows: in, It controls the gain coefficient of the detail layer. and These are the coefficients of the infrared and visible light images at level k, respectively; and These are detail layer images after Laplacian transformation of infrared and visible light images, respectively.

[0051] The basic components are fused using a simple averaging method to generate a base layer fused image, represented as follows: in, This is the weighting coefficient for the infrared portion of the base layer, typically set to 0.5; and These are the base layer images after Laplacian transformation of the infrared and visible light images, respectively.

[0052] The merged base layer and all detail layers are reconstructed using the inverse transformation of the Laplacian pyramid to obtain the final merged image.

[0053] The upsampling and filtering operations in the pyramid reconstruction process can be parallelized using OpenMP, and the filtering operations can be further accelerated using the SIMD instruction set.

[0054] In practice, to improve calculation speed, the value of N should not be too large.

[0055] The result of merging floating-point numbers Convert to an 8-bit grayscale image suitable for display or subsequent processing. This includes cropping the result to the appropriate range and quantizing it.

[0056] This embodiment proposes and implements a complete solution integrating offline high-precision camera distortion calibration, dual-spectral geometric registration (through accurate relative pose estimation and pre-computed mapping tables), and online real-time distortion correction and field-of-view matching. This solution addresses the inherent geometric inconsistencies between infrared and visible light images, avoiding fusion artifacts. Using the Laplacian pyramid as a multi-scale decomposition framework and combining local gradient energy as a measure of activity in the detail layer, relative weights effectively preserve the salient information of each element, improving the detail richness and visual quality of the fused image.

[0057] This embodiment offloads complex geometric transformation calculations to an offline stage, while online processing only requires efficient table lookups and interpolation. It achieves full-process parallelization: all core steps, including image acquisition, distortion correction, geometric registration, pyramid decomposition and reconstruction, and pixel-level fusion rule calculation, deeply utilize multi-threaded parallel computing, fully leveraging the performance of multi-core CPUs.

[0058] This embodiment accelerates the process with vectorized instruction sets: SIMD vectorized instruction sets (such as AVX / SSE) are widely used in critical pixel-level array operations (such as filtering, gradient calculation, and weighted summation) to achieve parallel data processing, significantly improve computational throughput, and ensure that the entire fusion process can meet the real-time video frame rate requirements.

[0059] The embodiment uses activity measurement based on local gradient energy. By using relative weights, the fusion process can adaptively and dynamically allocate weights according to the image content. The addition of a detail layer relative weight can generate a more information-rich and robust fused image.

[0060] Figure 2 This is a schematic diagram of a device frame for real-time dual-spectrum infrared and visible light fusion according to one embodiment. The device 200 includes: Image acquisition module 201 is used to acquire the input initial infrared light image and initial visible light image in real time; The distortion correction module 202 is used to perform distortion correction on the initial infrared light image and the initial visible light image based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, and generate the corrected infrared light image to be registered and the visible light image to be fused. The registration generation module 203 is used to register the infrared light image to be registered and generate the infrared image to be fused based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage. The image decomposition module 204 is used to decompose the infrared image to be fused and the visible light image to be fused using a Laplacian pyramid, so as to obtain the first Laplacian pyramid of the infrared image to be fused and the second Laplacian pyramid of the visible light image to be fused, respectively. The layered fusion module 205 is used to generate a detail layer fusion image by using an adaptive weight fusion method on the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and to generate a base layer fusion image by using an average value method on the base layer images in the first Laplacian pyramid and the second Laplacian pyramid. The fusion output module 206 is used to reconstruct the detail layer fused image and the base layer fused image by applying the inverse Laplacian pyramid transform to generate the final fused image.

[0061] This embodiment also discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a method for real-time dual-spectrum infrared and visible light fusion.

[0062] Specifically, the electronic device includes a processor and a memory connected via a system bus. The processor provides computing and control capabilities to support the operation of the entire electronic device. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. These computer programs can be executed by the processor to implement a dual-spectrum fusion method provided in the following embodiments. The internal memory provides a cached runtime environment for the operating system computer programs in the non-volatile storage media.

[0063] The various modules in the fusion device provided in this application embodiment can be implemented in the form of computer programs. These computer programs can run on a terminal or server. The program modules constituted by these computer programs can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements the steps of the methods described in this application embodiment.

[0064] This embodiment also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of a method for real-time dual-spectrum infrared and visible light fusion. A computer program product containing instructions, which, when run on a computer, causes the computer to perform dual-spectrum fusion.

[0065] Any references to memory, storage, database, or other media used in the embodiments of this application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for real-time dual-spectral infrared and visible light fusion, characterized in that, Specifically, the steps include the following: The initial infrared and visible light images are acquired in real time. Based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, the initial infrared light image and the initial visible light image are distorted to generate the distorted infrared light image to be registered and the visible light image to be fused. The infrared image to be registered is registered based on the spatial geometric relationship between the infrared camera and the visible light camera established during the offline calibration phase to generate the infrared image to be fused. The infrared image and the visible light image to be fused are decomposed using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused, respectively. An adaptive weighted fusion method is used to generate a fused detail layer image from the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and an average value method is used to generate a fused base layer image from the base layer images in the first Laplacian pyramid and the second Laplacian pyramid. The detailed layer fused image and the basic layer fused image are reconstructed using the inverse Laplacian pyramid transform to generate the final fused image.

2. The method for real-time dual-spectral infrared and visible light fusion according to claim 1, characterized in that, The process of correcting the distortion parameters of the infrared camera and the visible light camera obtained during the offline calibration phase to generate the corrected infrared image to be registered and the visible light image to be fused specifically includes: Distortion calibration was performed on both the infrared and visible light cameras during the offline calibration phase to obtain the distortion parameters of the infrared camera. and visible light camera distortion parameters ; Based on the distortion model function and the acquired initial infrared image and the initial visible light image The intensity value is used to generate the distortion-corrected image, represented as: in, This represents the corrected infrared image to be registered. This represents the corrected visible light image to be fused. This is the distortion model function for infrared cameras. For visible light camera distortion model functions, each represents the coordinates of a distortion-free pixel. Mapped to the corresponding coordinates in the distorted image Represents the initial infrared image In coordinates Interpolation is performed at the location to obtain the intensity value. Represents the initial visible light image In coordinates Interpolation is performed at the location to obtain the intensity value. Indicates the infrared image to be registered. pixel domain, Represents the visible light images to be fused The pixel field.

3. The method for real-time dual-spectral infrared and visible light fusion according to claim 2, characterized in that, The steps for registering the infrared image to be registered and generating the infrared image to be fused based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage specifically include: In the offline calibration stage, image samples based on infrared and visible light are acquired, and the feature points in the image samples are distorted. The feature points in the distorted infrared image are then matched with the corresponding feature points in the visible light image to generate a set of matched distortion-free feature points. Based on the distortion-free feature point set pairs and the obtained infrared camera intrinsic parameter matrix and visible light camera intrinsic parameter matrix, the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system is estimated. A pixel coordinate mapping model between the infrared image and the visible light image is established based on the relative pose. The infrared image to be registered is registered according to the pixel coordinate mapping model to generate an infrared image to be fused.

4. The method for real-time dual-spectral infrared and visible light fusion according to claim 3, characterized in that, The process of estimating the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system specifically includes a nonlinear optimization process that minimizes the reprojection error function to estimate the relative pose of the infrared camera coordinate system with respect to the visible light camera coordinate system.

5. The method for real-time dual-spectral infrared and visible light fusion according to claim 1, characterized in that, The process of decomposing the infrared image to be fused and the visible light image to be fused using a Laplacian pyramid to obtain a first Laplacian pyramid of the infrared image to be fused and a second Laplacian pyramid of the visible light image to be fused specifically includes: decomposing the infrared image to be fused using a Laplacian pyramid to generate a first Laplacian pyramid based on a base layer image and multiple detail layer images of the infrared image to be fused; and decomposing the visible light image to be fused using a Laplacian pyramid to generate a second Laplacian pyramid based on a base layer image and multiple detail layer images of the visible light image to be fused.

6. The method for real-time dual-spectral infrared and visible light fusion according to claim 5, characterized in that, The process of generating a fused detail image by using an adaptive weighted fusion method for the detail layer images in the first and second Laplacian pyramids specifically includes: Obtain the local activity of detail layer images at any level of the first and second Laplacian pyramids, and generate adaptive fusion weights for infrared and visible light images; For each detail layer, the adaptive fusion weights are used to fuse and generate a detail layer fused image.

7. The method for real-time dual-spectral infrared and visible light fusion according to claim 1, characterized in that, The process of reconstructing the detail layer fused image and the base layer fused image using the inverse Laplacian pyramid transformation to generate the final fused image also includes converting the floating-point fused result of the final fused image into an 8-bit grayscale image as the image output result.

8. An apparatus for real-time dual-spectrum infrared and visible light fusion to perform the method as described in any one of claims 1-7, characterized in that, The device includes: The image acquisition module is used to acquire the initial infrared light image and the initial visible light image in real time; The distortion correction module is used to correct the distortion of the initial infrared light image and the initial visible light image based on the distortion parameters of the infrared camera and the visible light camera obtained in the offline calibration stage, and generate the corrected infrared light image to be registered and the visible light image to be fused. The registration generation module is used to register the infrared light image to be registered and generate the infrared image to be fused based on the spatial geometric relationship between the infrared camera and the visible light camera established in the offline calibration stage. The image decomposition module is used to decompose the infrared image to be fused and the visible light image to be fused using a Laplacian pyramid, so as to obtain the first Laplacian pyramid of the infrared image to be fused and the second Laplacian pyramid of the visible light image to be fused, respectively. The layered fusion module is used to generate a fused detail layer image by using an adaptive weighted fusion method on the detail layer images in the first Laplacian pyramid and the second Laplacian pyramid, and to generate a fused base layer image by using an average value method on the base layer images in the first Laplacian pyramid and the second Laplacian pyramid. The fusion output module is used to reconstruct the detail layer fused image and the base layer fused image using the inverse Laplacian pyramid transform to generate the final fused image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images