Method, system and device for infrared and visible light image fusion

Abstract

The application provides a method, system and device for infrared light image and visible light image fusion, comprising: acquiring an affine matrix between visible light images and infrared light images with a corresponding relationship through a calibration board, and acquiring an affine matrix set by counting a plurality of affine matrices with different distances; acquiring a corresponding relationship set of affine matrix and image information with different distances according to the affine matrix set; acquiring a to-be-fused affine matrix of a to-be-fused infrared light image and a to-be-fused visible light image according to the corresponding relationship set; and performing image fusion on the to-be-fused infrared light image and the to-be-fused visible light image according to the to-be-fused affine matrix. Thus, the to-be-fused images with different distances can be accurately fused according to the arithmetic model, and manpower and material resources are saved.

Description

Methods, systems, and devices for fusing infrared and visible light images Technical Field

[0001] This invention belongs to the field of image fusion, and particularly relates to methods, systems and devices for fusing infrared and visible light images. Background Technology

[0002] Image fusion technology is becoming increasingly common, especially the fusion of infrared and visible light images of the same target. It can simultaneously display details from both infrared and visible light images in the fused image. Existing image fusion technologies mainly include the following aspects:

[0003] 1. The method based on anisotropic diffusion fusion (ADF) decomposes the original image into a base layer and a detail layer, performs Karhunen-Loeve transformation, and then performs linear superposition to finally obtain a fused image. The algorithm is easy to implement and has small loss after fusion, but the final visual effect is not ideal.

[0004] 2. Methods based on convolutional neural networks (CNN) and residual neural networks (ResNet) are used to build convolutional neural networks and residual neural networks, optimize parameters through extensive training, and then fuse images. The performance of image fusion is better, but it requires the construction of datasets and consumes a lot of time for training, which is time-consuming.

[0005] 3. The gradient transfer fusion (GTF) method first constructs an objective function and solves the objective function iteratively to make the fused image closer to the infrared image and retain more appearance information. Its advantage is that the fused image has a good visual effect, but because it requires iterative solution, the algorithm cannot meet the requirements of real-time processing.

[0006] 4. The method based on latent low-rank representation (LatLRR) first decomposes the original image to obtain the low-rank part and the salient part, then uses two different strategies to fuse them, and finally reconstructs the image, but its visual effect is not very ideal. Summary of the Invention

[0007] The main objective of this invention is to provide a method, system, and device for fusing infrared and visible light images, enabling images to be fused at different distances to be accurately fused according to an arithmetic model, thus saving manpower and resources.

[0008] In a first aspect, a method for fusing infrared light images and visible light images is provided, the method comprising:

[0009] The affine matrix between the visible light image and the infrared light image with a corresponding relationship is obtained by using a calibration board, and the affine matrix set is obtained by counting multiple affine matrices at different distances. The correspondence is the same as that of taking pictures of the same object at the same position.

[0010] Based on the affine matrix set, obtain the correspondence set between the affine matrix and the relevant information of infrared light images at different distances;

[0011] The affine matrices of the infrared light image and the visible light image to be fused are obtained based on the set of correspondences.

[0012] Image fusion is performed on the infrared image and the visible image to be fused based on the affine matrix to be fused.

[0013] In one possible implementation, obtaining the affine matrix between the corresponding visible light image and infrared light image through a standard calibration board includes:

[0014] Visible light images and infrared images were acquired at a set calibration distance;

[0015] Extract and save the center of the calibration plate from the visible light image and infrared image at the set calibration distance, respectively. Calculate the affine matrix at the set calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at the set calibration distance.

[0016] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at the set calibration distance.

[0017] In another possible implementation, the step of calculating the affine matrix of the visible light image corresponding to the infrared image at each set distance based on the affine matrix at the set calibration distance specifically includes:

[0018] The average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are calculated. Using the center of the calibration plate in the visible light and infrared images at each calibration distance and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are corrected.

[0019] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated, and the affine matrix at each set distance is generated.

[0020] In another possible implementation, the correction of the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance specifically includes:

[0021] The nearest neighbor distance method is used to correct the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance.

[0022] In another possible implementation, the step of calculating the X-direction offset components and Y-direction offset components of the affine matrix at each set distance based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance specifically includes: calculating the X-direction offset components and Y-direction offset components of the affine matrix at each set distance using interpolation based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance.

[0023] In another possible implementation, obtaining the relevant information of the affine matrix and infrared light images at different distances based on the affine matrix set includes:

[0024] Based on the focusing principle, the relevant information of the infrared light image and the visible light image to be fused are obtained through the Hall element. The relevant information includes: x-direction component and y-direction component. The x-direction component is the length information of the infrared light image, and the y-direction component is the width information of the infrared light image.

[0025] In another possible implementation, the image fusion of the infrared image and the visible image to be fused based on the affine matrix to be fused includes:

[0026] Perform an affine transformation on the visible light image to be fused according to the affine matrix to be fused, and obtain the transformed visible light image;

[0027] Extract high-frequency image information from the transformed visible light image;

[0028] The high-frequency image information is superimposed onto the infrared light image to be fused to obtain a fused image.

[0029] In another possible implementation, obtaining the set of correspondences between the affine matrices and images at different distances based on the set of affine matrices includes:

[0030] Obtaining the average affine matrix from the set of affine matrices includes:

[0031] Obtain the average value of the rotation component and the average value of the scaling component of the affine matrices in the affine matrix set;

[0032] The average affine matrix is ​​obtained based on the average value of the rotation component and the average value of the scaling component.

[0033] Based on the average affine matrix and the affine matrix to be acquired, the offsets of the affine matrix to be acquired at different distances are obtained by the nearest neighbor offset distance algorithm. The offsets include: X-direction offset t1 and Y-direction offset t2.

[0034] Interpolation calculations are performed on multiple offsets to obtain a set of correspondences between the offsets and affine matrices.

[0035] Secondly, a system for fusing infrared and visible light images is provided, the system comprising:

[0036] The affine matrix set acquisition module is used to acquire the affine matrix between the visible light image and the infrared light image that have a corresponding relationship through the calibration plate, and to count multiple affine matrices at different distances to obtain the affine matrix set, wherein the correspondence is the same as that of taking pictures of the same object at the same position;

[0037] The correspondence set acquisition module is used to acquire the correspondence set between the affine matrix and the relevant information of infrared light images at different distances based on the affine matrix set;

[0038] The module for obtaining the affine matrix to be fused is used to obtain the affine matrix to be fused of the infrared light image and the visible light image to be fused according to the set of correspondences.

[0039] An image fusion module is used to perform image fusion on the infrared light image and the visible light image to be fused according to the affine matrix to be fused.

[0040] Thirdly, an apparatus is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for fusing infrared and visible light images as provided in the first aspect. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0042] Figure 1 is a flowchart of a method for fusing infrared and visible light images according to an embodiment of the present invention;

[0043] Figure 2 is a flowchart of a method for fusing infrared and visible light images according to another embodiment of the present invention;

[0044] Figure 3 is a structural diagram of a system for fusing infrared and visible light images according to an embodiment of the present invention;

[0045] Figure 4 is a structural diagram of a system for fusing infrared and visible light images according to another embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of the calibration plate provided in an embodiment of the present invention;

[0047] Figure 6 is a cross-sectional view of the calibration plate provided in an embodiment of the present invention;

[0048] Figure 7 is an enlarged view of part P in Figure 6;

[0049] Figure 8 is a schematic diagram of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0050] Figure 9 is an exploded view of the heating target box of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0051] Figure 10 is a schematic diagram of the heating unit of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0052] Figure 11 is an exploded view of the frame of the infrared dual-light fusion testing device provided in an embodiment of the present invention.

[0053] Figure 12 is a schematic diagram of the physical structure of an electronic device according to the present invention. Detailed Implementation

[0054] Specific implementation method

[0055] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0056] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, modules, components, and / or groups thereof. It should be understood that when we say a module is “connected” or “coupled” to another module, it can be directly connected or coupled to the other module, or there may be an intermediate module. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein includes all or any of the modules and all combinations thereof of one or more associated listed items.

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the implementation of the present invention will be described in further detail below with reference to the accompanying drawings.

[0058] The technical solutions of the present invention and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0059] Figure 1 shows a flowchart of a method for fusing infrared and visible light images according to an embodiment of the present invention. The method includes:

[0060] Step 101: Obtain the affine matrix between the visible light image and the infrared light image that have a corresponding relationship through the calibration plate, and count the multiple affine matrices at different distances to obtain the affine matrix set. The corresponding relationship is that the same object is photographed at the same position.

[0061] Step 102: Obtain the set of correspondences between the affine matrices and the relevant information of infrared light images at different distances based on the affine matrix set;

[0062] Step 103: Obtain the affine matrices of the infrared light image and the visible light image to be fused according to the set of correspondences;

[0063] Step 104: Perform image fusion on the infrared light image and the visible light image to be fused according to the affine matrix to be fused.

[0064] In this embodiment of the invention, infrared and visible light images of the same object are captured at the same location. The acquired infrared and visible light images form a set of corresponding images. These corresponding images are registered using existing technology to obtain their affine matrices. This process of obtaining affine matrices is repeated at different distances to obtain multiple affine matrices at different distances. These multiple affine matrices are then compiled into an affine matrix set. The affine matrix set is then processed to obtain a set of correspondences between the affine matrices and relevant information about the images at different distances. This relevant information includes distance and height information of the images. Relevant information about the image group to be fused is obtained. This image group includes infrared and visible light images captured at the same distance. This relevant information is input into the affine matrix set to obtain the corresponding affine matrices. Finally, the affine matrices are used to perform image fusion on the infrared and visible light images to be fused.

[0065] The step of obtaining the affine matrix between the corresponding visible light image and infrared light image through a standard calibration plate includes:

[0066] Visible light images and infrared images were acquired at a set calibration distance;

[0067] Extract and save the center of the calibration plate from the visible light image and infrared image at the set calibration distance, respectively. Calculate the affine matrix at the set calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at the set calibration distance.

[0068] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at the set calibration distance.

[0069] Specifically, the step of calculating the affine matrix of the visible light image corresponding to the infrared image at each set distance based on the affine matrix at the set calibration distance includes:

[0070] The average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are calculated. Using the center of the calibration plate in the visible light and infrared images at each calibration distance and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are corrected.

[0071] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated, and the affine matrix at each set distance is generated.

[0072] Specifically, the correction of the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance includes:

[0073] The nearest neighbor distance method is used to correct the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance.

[0074] The step of calculating the X-direction offset components and Y-direction offset components of the affine matrix at each set distance based on the corrected X-direction offset components and Y-direction offset components at each calibration distance includes:

[0075] Based on the X-direction and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction and Y-direction offset components of the affine matrix at each set distance are calculated using interpolation.

[0076] The principle of the nearest neighbor distance method of this invention is as follows: Under a single distance, the coordinates of each circle center in the visible calibration image are cyclically input, and the offset vectors in the X direction and Y direction corresponding to the circle center coordinates in the corresponding infrared calibration image can be calculated. For example, the first circle center after sorting in the visible calibration image corresponds to the first circle center after sorting in the infrared calibration image. The corresponding offset vectors in the X direction and Y direction are calculated and denoted as an, bn. Then, by traversing all the circle centers corresponding to the infrared calibration image circles under this distance, the circle centers in the visible calibration image are transformed into the circle centers in the infrared calibration image, and the straight-line distance dis between the circle centers extracted from the infrared calibration image and the circle centers is calculated. For example, if 25 circle centers are extracted from both the infrared calibration image and the visible calibration image, the loop is repeated 25 times, and the minimum straight-line distance value is accumulated and a set of an, bn is denoted as the offset vectors in the X direction and Y direction under the current distance.

[0077] Using the centers of the calibration plate in the visible and infrared images at various calibration distances, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X and Y offset components of the affine matrix at each calibration distance. Specifically, this includes: at a single calibration distance, obtaining the centers of the calibration plate in the visible and infrared images at that calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances; for each pair of corresponding visible and infrared calibration image centers, first obtaining the X and Y offset components, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, to form a new affine matrix (a pair of corresponding visible and infrared calibration image centers). Each new affine matrix corresponds to a new circle center. Using each new affine matrix, the coordinates of the circle center in the visible calibration image transformed to the center in the infrared calibration image are calculated and designated as the second circle center in the infrared calibration image. The offset vectors in the X and Y directions, as well as the straight-line distance dis, between the circle center and the first circle center coordinates in the corresponding infrared calibration image (i.e., the extracted circle center in the infrared calibration image) are calculated. The straight-line distances dis corresponding to all circle centers are summed to obtain the cumulative straight-line distance value corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X and Y directions, denoted as an and bn. The set of offset vectors in the X and Y directions corresponding to the minimum value of the cumulative straight-line distance is taken as the offset vectors in the X and Y directions under the current calibration distance.

[0078] The acquisition of the relevant information of the infrared light image and the visible light image to be fused includes:

[0079] Based on the focusing principle, the relevant information of the infrared light image and the visible light image to be fused are obtained through the Hall element. The relevant information includes: x-direction component and y-direction component. The x-direction component is the length information of the infrared light image, and the y-direction component is the width information of the infrared light image.

[0080] The step of performing image fusion on the infrared image and the visible image to be fused based on the affine matrix to be fused includes:

[0081] Perform an affine transformation on the visible light image to be fused according to the affine matrix to be fused, and obtain the transformed visible light image;

[0082] Extract high-frequency image information from the transformed visible light image;

[0083] The high-frequency image information is superimposed onto the infrared light image to be fused to obtain a fused image.

[0084] As an optional embodiment of the present invention, after the step of performing image fusion on the infrared image to be fused and the visible image to be fused according to the affine matrix to be fused, the method further includes:

[0085] The fused image is converted into an RGB fused image.

[0086] In this embodiment of the invention, an affine matrix between a visible light image and an infrared image with a corresponding relationship is obtained through a standard calibration board, and a set of affine matrices is obtained by statistically analyzing multiple affine matrices at different distances. Based on the set of affine matrices, a set of correspondences between the affine matrices and relevant information of images at different distances is obtained. Relevant information of the infrared light image and the visible light image to be fused is obtained. Based on the relevant information, the corresponding affine matrix is ​​obtained from an arithmetic model. Image fusion is then performed on the infrared light image and the visible light image to be fused based on the affine matrix. This allows images to be fused at different distances to be accurately fused according to the arithmetic model, saving manpower and resources.

[0087] Figure 2 shows a flowchart of a method for fusing infrared and visible light images according to another embodiment of the present invention. The step of obtaining a set of correspondences between the affine matrix and images at different distances based on the affine matrix set includes:

[0088] Step 201: Obtain the average affine matrix based on the set of affine matrices;

[0089] Step 202: Based on the average affine matrix and the affine matrix to be acquired, the offset of the affine matrix to be acquired at different distances is obtained by the nearest neighbor offset distance algorithm. The offset includes: X-direction offset t1 and Y-direction offset t2.

[0090] Step 203: Perform interpolation calculations on the multiple offsets to obtain a set of correspondences between the offsets and the affine matrix.

[0091] In this embodiment of the invention, the affine matrix can be expressed by the formula... Let a be represented as a function of a. 11 a 12 a 21 a 22 These represent the rotation and scaling components of the affine matrix. The affine matrix set is calculated according to the affine matrix formula to obtain the average affine matrix of the entire set. Based on the average affine matrix and the affine matrix to be acquired in the image, offsets of the affine matrix at different distances are obtained using the nearest neighbor offset distance algorithm. These offsets include an X-direction offset t1 and a Y-direction offset t2. However, due to the distance intervals in the obtained offsets, a continuous and smooth set of correspondences cannot be obtained. Therefore, interpolation calculations are needed for multiple offsets to obtain a continuous and smooth set of correspondences. The nearest neighbor offset distance algorithm and interpolation algorithm used here employ existing technologies and will not be elaborated upon here.

[0092] Wherein, obtaining the average affine matrix based on the set of affine matrices includes:

[0093] Obtain the average value of the rotation component and the average value of the scaling component of the affine matrices in the affine matrix set;

[0094] The average affine matrix is ​​obtained based on the average value of the rotation component and the average value of the scaling component.

[0095] In this embodiment of the invention, the average value of the rotation components of all affine matrices in the affine matrix set is obtained by averaging the rotation components, and the average value of the scaling components is obtained by averaging the scaling components. The average value of the rotation components and the average value of the scaling components are then substituted into the affine matrix formula to obtain the average affine matrix.

[0096] Figure 3 shows a structural diagram of a system for fusing infrared and visible light images according to an embodiment of the present invention. The system includes:

[0097] The affine matrix set acquisition module 301 is used to acquire the affine matrix between the visible light image and the infrared light image that have a corresponding relationship through the calibration plate, and to count multiple affine matrices at different distances to obtain the affine matrix set, wherein the correspondence is the same as taking pictures of the same object at the same position.

[0098] The correspondence set acquisition module 302 is used to acquire the correspondence set between the affine matrix and the infrared light images at different distances based on the affine matrix set;

[0099] The affine matrix acquisition module 303 is used to acquire the affine matrices of the infrared light image and the visible light image to be fused according to the correspondence set;

[0100] The image fusion module 304 is used to perform image fusion on the infrared light image to be fused and the visible light image to be fused according to the affine matrix to be fused.

[0101] In this embodiment of the invention, infrared and visible light images of the same object are captured at the same location. The acquired infrared and visible light images form a set of corresponding images. These corresponding images are registered using existing technology to obtain their affine matrices. This process of obtaining affine matrices is repeated at different distances to obtain multiple affine matrices at different distances. These multiple affine matrices are then compiled into an affine matrix set. The affine matrix set is then processed to obtain a set of correspondences between the affine matrices and relevant information about the images at different distances. This relevant information includes distance and height information of the images. Relevant information about the image group to be fused is obtained. This image group includes infrared and visible light images captured at the same distance. This relevant information is input into the affine matrix set to obtain the corresponding affine matrices. Finally, the affine matrices are used to perform image fusion on the infrared and visible light images to be fused.

[0102] Visible light images and infrared images were acquired at a set calibration distance;

[0103] Extract and save the center of the calibration plate from the visible light image and infrared image at the set calibration distance, respectively. Calculate the affine matrix at the set calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at the set calibration distance.

[0104] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at the set calibration distance.

[0105] Specifically, the step of calculating the affine matrix of the visible light image corresponding to the infrared image at each set distance based on the affine matrix at the set calibration distance includes:

[0106] The average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are calculated. Using the center of the calibration plate in the visible light and infrared images at each calibration distance and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are corrected.

[0107] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated, and the affine matrix at each set distance is generated.

[0108] Specifically, the correction of the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance includes:

[0109] The nearest neighbor distance method is used to correct the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance.

[0110] The step of calculating the X-direction offset components and Y-direction offset components of the affine matrix at each set distance based on the corrected X-direction offset components and Y-direction offset components at each calibration distance includes:

[0111] Based on the X-direction and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction and Y-direction offset components of the affine matrix at each set distance are calculated using interpolation.

[0112] The principle of the nearest neighbor distance method of this invention is as follows: Under a single distance, the coordinates of each circle center in the visible calibration image are cyclically input, and the offset vectors in the X direction and Y direction corresponding to the circle center coordinates in the corresponding infrared calibration image can be calculated. For example, the first circle center after sorting in the visible calibration image corresponds to the first circle center after sorting in the infrared calibration image. The corresponding offset vectors in the X direction and Y direction are calculated and denoted as an and bn. Then, by traversing all the circle centers corresponding to the infrared calibration image circles under this distance, the circle center in the visible calibration image is transformed into the circle center in the infrared calibration image, and the straight-line distance dis between the circle center extracted from the infrared calibration image and the circle center is calculated. For example, if 25 circle centers are extracted from both the infrared calibration image and the visible calibration image, the loop is repeated 25 times, and the minimum straight-line distance value is accumulated to obtain a set of an and bn, which are denoted as the offset vectors in the X direction and Y direction under the current distance.

[0113] Using the centers of the calibration plate in the visible and infrared images at various calibration distances, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X and Y offset components of the affine matrix at each calibration distance. Specifically, this includes: at a single calibration distance, obtaining the centers of the calibration plate in the visible and infrared images at that calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances; for each pair of corresponding visible and infrared calibration image centers, first obtaining the X and Y offset components, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, to form a new affine matrix (a pair of corresponding visible and infrared calibration image centers). Each new affine matrix corresponds to a new circle center. Using each new affine matrix, the coordinates of the circle center in the visible calibration image transformed to the center in the infrared calibration image are calculated and designated as the second circle center in the infrared calibration image. The offset vectors in the X and Y directions, as well as the straight-line distance dis, between the circle center and the first circle center coordinates in the corresponding infrared calibration image (i.e., the extracted circle center in the infrared calibration image) are calculated. The straight-line distances dis corresponding to all circle centers are summed to obtain the cumulative straight-line distance value corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X and Y directions, denoted as an and bn. The set of offset vectors in the X and Y directions corresponding to the minimum value of the cumulative straight-line distance is taken as the offset vectors in the X and Y directions under the current calibration distance.

[0114] The acquisition of the relevant information of the infrared light image and the visible light image to be fused includes:

[0115] Based on the focusing principle, the relevant information of the infrared light image and the visible light image to be fused are obtained through the Hall element. The relevant information includes: x-direction component and y-direction component. The x-direction component is the length information of the infrared light image, and the y-direction component is the width information of the infrared light image.

[0116] The step of performing image fusion on the infrared image and the visible image to be fused based on the affine matrix to be fused includes:

[0117] Perform an affine transformation on the visible light image to be fused according to the affine matrix to be fused, and obtain the transformed visible light image;

[0118] Extract high-frequency image information from the transformed visible light image;

[0119] The high-frequency image information is superimposed onto the infrared light image to be fused to obtain a fused image.

[0120] As an optional embodiment of the present invention, after the step of performing image fusion on the infrared image to be fused and the visible image to be fused according to the affine matrix to be fused, the method further includes:

[0121] The fused image is converted into an RGB fused image.

[0122] In this embodiment of the invention, an affine matrix between a visible light image and an infrared image with a corresponding relationship is obtained through a standard calibration board, and a set of affine matrices is obtained by statistically analyzing multiple affine matrices at different distances. Based on the set of affine matrices, a set of correspondences between the affine matrices and relevant information of images at different distances is obtained. Relevant information of the infrared light image and the visible light image to be fused is obtained. Based on the relevant information, the corresponding affine matrix is ​​obtained from an arithmetic model. Image fusion is then performed on the infrared light image and the visible light image to be fused based on the affine matrix. This allows images to be fused at different distances to be accurately fused according to the arithmetic model, saving manpower and resources.

[0123] Figure 4 shows a structural diagram of a system for fusing infrared and visible light images according to another embodiment of the present invention. The correspondence set acquisition module 302 includes:

[0124] The average affine matrix acquisition unit 401 is used to acquire the average affine matrix based on the set of affine matrices.

[0125] The offset acquisition unit 402 is used to acquire the offset of the affine matrix to be acquired at different distances by using the nearest neighbor offset distance algorithm based on the average affine matrix and the affine matrix to be acquired. The offset includes: X-direction offset t1 and Y-direction offset t2.

[0126] The correspondence set acquisition unit 403 is used to perform interpolation calculations on multiple offsets to obtain the correspondence set between the offsets and the affine matrix.

[0127] In this embodiment of the invention, the affine matrix can be expressed by the formula... Let a be represented as a function of a. 11 a 12 a 21 a 22These represent the rotation and scaling components of the affine matrix. The affine matrix set is calculated according to the affine matrix formula to obtain the average affine matrix of the entire set. Based on the average affine matrix and the affine matrix to be acquired in the image, offsets of the affine matrix at different distances are obtained using the nearest neighbor offset distance algorithm. These offsets include an X-direction offset t1 and a Y-direction offset t2. However, due to the distance intervals in the obtained offsets, a continuous and smooth set of correspondences cannot be obtained. Therefore, interpolation calculations are needed for multiple offsets to obtain a continuous and smooth set of correspondences. The nearest neighbor offset distance algorithm and interpolation algorithm used here employ existing technologies and will not be elaborated upon here.

[0128] The average affine matrix acquisition unit 401 includes:

[0129] The average value acquisition subunit is used to acquire the average value of the rotation component and the average value of the scaling component of the affine matrix in the affine matrix set;

[0130] The average affine matrix acquisition sub-unit is used to obtain the average affine matrix based on the average value of the rotation component and the average value of the scaling component.

[0131] In this embodiment of the invention, the average value of the rotation components of all affine matrices in the affine matrix set is obtained by averaging the rotation components, and the average value of the scaling components is obtained by averaging the scaling components. The average value of the rotation components and the average value of the scaling components are then substituted into the affine matrix formula to obtain the average affine matrix.

[0132] Figures 5 to 7 show a schematic diagram of a calibration plate provided in an embodiment of the present invention, including a calibration plate body 51. The calibration plate body 51 is provided with at least two sets of target holes. Each set of target holes contains multiple target holes. The target holes in the same set of target holes have the same diameter, while the target holes in different sets of target holes have different diameters.

[0133] Furthermore, each group of target holes corresponds to a region, and the area of ​​the region corresponding to each group of target holes is different. All target holes in each group of target holes must be located within the corresponding region.

[0134] Furthermore, when the area corresponding to the first set of target holes is larger than the area corresponding to the second set of target holes, the aperture of the first set of target holes is larger than the aperture of the second set of target holes. Areas can overlap; for example, a large area can completely cover a small area.

[0135] Furthermore, the area corresponding to each group of target holes is either a circular area or a polygonal area (polygonal areas include triangular areas, quadrilateral areas, pentagonal areas, etc.). That is, the area corresponding to each group of target holes is one of a circular area, a triangular area, a quadrilateral area, a pentagonal area, etc. In this embodiment, the area corresponding to each group of target holes is a rectangular area.

[0136] Furthermore, the centers of the regions corresponding to multiple sets of target holes are the same.

[0137] Furthermore, the areas corresponding to the multiple sets of target holes are the same as the center of calibration plate 5.

[0138] Furthermore, each group of target holes contains at least three target holes, and not all target holes in each group are on a straight line.

[0139] Furthermore, all target holes in each group are arranged in an array.

[0140] Furthermore, after all the target holes in each group are arranged in an array to form an array, the centers of each array are the same.

[0141] Furthermore, the spacing between the centers of two adjacent target holes in each row of each array is equal, and the spacing between the centers of two adjacent target holes in each column is equal.

[0142] Furthermore, the target hole is a circular hole, and the back opening of the target hole is chamfered. After the back opening of the target hole is chamfered, it has a connected circular hole segment and a conical hole segment. The circular hole segment is away from the heat source module, and the conical hole segment is close to the heat source module (the diameter of the conical hole segment gradually increases as it approaches the heat source module). The minimum diameter of the conical hole segment is equal to the diameter of the circular hole segment.

[0143] Furthermore, the front of the calibration plate body is treated with a matte white frosted finish, and the back of the calibration plate body is treated with a glossy black high-reflection finish.

[0144] In one embodiment, the calibration plate body 51 is provided with two sets of target holes, namely a first set of target holes and a second set of target holes. The first set of target holes includes a plurality of first target holes 52 (large round holes), and the second set of target holes includes a plurality of second target holes 53 (small round holes). The diameter of the first target holes 52 is larger than the diameter of the second target holes 53. All the first target holes 52 are located in a first region, and all the second target holes 53 are located in a second region.

[0145] Furthermore, the first region is a first rectangular region with a length of 800mm and a width of 600mm; the second region is a second rectangular region with a length of 70mm and a width of 50mm.

[0146] Furthermore, the centers of the first rectangular region and the second rectangular region are the same.

[0147] Furthermore, the first rectangular region and the second rectangular region are centered on the calibration plate 5.

[0148] Furthermore, the number of the first target hole 52 and the second target hole 53 is at least three, and not all of the first target holes 52 are on a straight line, nor are all of the second target holes 53 on a straight line.

[0149] The diameters of the first target hole 52 and the second target hole 53 are set as needed. In one embodiment, the diameter of the first target hole 52 is 100mm and the diameter of the second target hole 53 is 10mm.

[0150] Furthermore, multiple first target holes 52 are arranged in an array to form a first array, and multiple second target holes 53 are arranged in an array to form a second array, with the center of the first array being the same as the center of the second array.

[0151] Furthermore, in the first array, the distance between the centers of two adjacent first target holes 52 in each row is equal. The distance between the centers of two adjacent first target holes 52 in each column is also equal.

[0152] Furthermore, in the second array, the distance between the centers of two adjacent second target holes 53 in each row is equal. The distance between the centers of two adjacent second target holes 53 in each column is also equal.

[0153] Furthermore, the first target hole 52 and the second target hole 53 are round holes, and the back openings of both the first target hole 52 and the second target hole 53 are chamfered. In this embodiment, the calibration plate 5 is 3mm thick and made of aluminum alloy.

[0154] A specific embodiment of a calibration plate 5 is as follows: the calibration plate body 51 is machined with six first target holes 52 (the first target holes 52 are large round holes with a diameter of 100mm) and six second target holes 53 (the second target holes 53 are small round holes with a diameter of 10mm). The back opening of each hole has a chamfer of 1.5 x 60° mm. After the machining is completed, the front side of the calibration plate body 51, i.e. the outward side, is treated with a matte white frosted finish, and the back side of the calibration plate body 51, i.e. the inward side, is treated with a glossy black high-reflection finish.

[0155] In this embodiment, the six first target holes 52 are arranged in two rows and three columns in a first array. The distance between the centers of two adjacent first target holes 52 in each row of the first array is 200 mm. The distance between the centers of two adjacent first target holes 52 in each column of the first array is also 200 mm.

[0156] In this embodiment, the six second target holes 53 are arranged in a second array of two rows and three columns. The distance between the centers of two adjacent second target holes 53 in each row of the second array is 20 mm. The distance between the centers of two adjacent first target holes 52 in each column of the second array is also 20 mm.

[0157] In this embodiment, the center of the first array and the center of the second array are the same as the center of the calibration board 5.

[0158] Referring to Figures 5 to 11, an embodiment of the present invention provides an infrared dual-light fusion testing device, including a heat source module and a calibration plate 5, wherein the heat source module is used to provide a heat source for the calibration plate 5.

[0159] The method for fusing infrared and visible light images described in Example 1 can be calibrated using the calibration plate 5 of this example.

[0160] Each set of target holes corresponds to a distance range, obtaining the distance between the infrared thermal imaging device and the calibration plate 5. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with multiple set distance ranges to determine which distance range the distance between the infrared thermal imaging device and the calibration plate 5 falls within. When the distance between the infrared thermal imaging device and the calibration plate 5 falls within a certain distance range, the set of target holes corresponding to that distance range is used for calibration (i.e., infrared-visible light image registration) or testing (during calibration, the center of the target hole circle of the calibration plate in the visible light image and infrared image at the calibration distance is extracted and saved, and then calibration is performed).

[0161] When two sets of target holes are set on the calibration plate 5, namely the first set of target holes (a set of large circular holes) and the second set of target holes (a set of small circular holes), the infrared thermal imaging device acquires paired visible light images and infrared images. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with a set first distance (e.g., 2 meters). When the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to the first distance (e.g., 2 meters), the second target hole 53 (small circular hole) is used for calibration (i.e., the center of the second target hole of the calibration plate, i.e., the center of the small circle, is extracted from the visible light image and the infrared image at the calibration distance, and saved, and then calibration is performed). When the distance between the infrared thermal imaging device and the calibration plate 5 is greater than the first distance (e.g., 2 meters), the first target hole 52 (large circular hole) is used for calibration (i.e., the center of the first target hole of the calibration plate, i.e., the center of the large circle, is extracted from the visible light image and the infrared image at the calibration distance, and saved, and then calibration is performed). Generally, when the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to 2 meters, it is considered to be relatively close; when it is greater than 2 meters, it is considered to be relatively far.

[0162] Furthermore, the heat source module includes at least one heating unit, and each set of target holes corresponds to one or more heating units.

[0163] Furthermore, at least one set of target holes corresponds to multiple heating units.

[0164] Furthermore, at least one set of target holes corresponds to one heating unit.

[0165] Furthermore, when a set of target holes corresponds to multiple heating units, each target hole in that set of target holes corresponds to one heating unit.

[0166] Referring to Figure 5, the calibration plate 5 in this embodiment is provided with two sets of target holes, namely the first set of target holes and the second set of target holes. The first set of target holes includes a plurality of first target holes 52 (large round holes), and the second set of target holes includes a plurality of second target holes 53 (small round holes).

[0167] Furthermore, each first target hole 52 of the calibration plate 5 corresponds to one heating unit (i.e., the first heating unit), and all second target holes 53 of the calibration plate 5 correspond to one heating unit (i.e., the second heating unit). The sizes of the first heating unit and the second heating unit may differ as needed. In this embodiment, the size of the first heating unit is larger than the size of the second heating unit.

[0168] This embodiment has a total of seven heating units, including six first heating units and one second heating unit. The six first heating units correspond one-to-one with the six first target holes 52 on the calibration plate 5 and are arranged opposite each other. Each first heating unit provides a heat source to the corresponding first target hole 52 on the calibration plate 5. The second heating unit corresponds to the six second target holes 53 on the calibration plate 5 and provides a heat source to the six second target holes 53 on the calibration plate 5 through the second heating unit.

[0169] Furthermore, the heating unit includes a heat source 21, in which a heating element 22 and a temperature detection element 23 are installed. The heating element 22 is connected to the output terminal of the controller, and the temperature detection element 23 is connected to the input terminal of the controller. The controller is connected to an instruction input unit and / or a display unit.

[0170] Each heating unit is controlled independently.

[0171] Furthermore, the heating element 22 can be a heating rod, and the temperature sensing element 23 can be a thermocouple.

[0172] In this embodiment, the heat source 21 of the first heating unit uses two heating rods and one thermocouple.

[0173] In this embodiment, the heat source 21 of the second heating unit uses a heating rod and a thermocouple.

[0174] In this embodiment, the heat source 21 is a rectangular block. The heat source 21 is provided with a first mounting hole 211 for mounting a heating rod and a second mounting hole 212 for mounting a thermocouple. The lead-out ends of the heating rod and the thermocouple extend out of the heat source 21.

[0175] The rectangular block of the heat source 21 is made of a material with good thermal conductivity, such as aluminum in this embodiment. The surface of the heat source 21, i.e., the aluminum plate, is treated with a glossy black high-reflectivity finish to improve image quality.

[0176] Furthermore, the heating unit is connected to the heat source fixing plate 3 via a first heat insulation plate 24, and the heat source fixing plate 3 is installed on the back of the calibration plate 5. In this embodiment, the first heat insulation plate 24 is made of mica fiber board, which serves as heat insulation. The heat source body 21 is fixed to the heat source fixing plate 3 by the first heat insulation plate 24 and does not come into contact with the heat source fixing plate 3.

[0177] The heat source fixing plate 3 is provided with mounting holes for installing the heating unit. In this embodiment, the heat source body 21 is installed in the mounting holes of the heat source fixing plate 3.

[0178] A second heat insulation plate 4 is provided between the calibration plate 5 and the heat source fixing plate 3. The heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence. The second heat insulation plate 4 is provided with a through hole for the heat source of the heating unit to pass through.

[0179] In this embodiment, the second heat insulation board 4 is made of mica fiber board, which serves as a heat insulation agent.

[0180] Furthermore, each heating unit is connected to the first heat insulation plate 24 in a one-to-one correspondence, such as by bolt fixing. The first heat insulation plate 24 is connected to the heat source fixing plate 3, such as by bolt fixing.

[0181] Furthermore, the second heat insulation plate 4 is provided with a plurality of first through holes for the heat source of the first heating unit to pass through. The first through holes on the second heat insulation plate 4 correspond one-to-one with the first target holes 52 on the calibration plate 5 and are arranged opposite each other. The diameter of the first through holes on the second heat insulation plate 4 is set as needed so that the heat source of the first heating unit can reach the corresponding first target hole 52 through the corresponding first through hole.

[0182] Furthermore, the second heat insulation plate 4 is provided with one or more second through holes for the heat source of the second heating unit to pass through. In this embodiment, the second heat insulation plate 4 is provided with a second through hole for the heat source of the second heating unit to pass through, and this second through hole is arranged opposite to all the second target holes 53 on the calibration plate 5.

[0183] The number and size of the second through holes are set as needed so that the heat source of the second heating unit can reach the second target hole 53 through the second through holes.

[0184] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected by bolts. Bolt holes are provided on the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 respectively.

[0185] In this embodiment, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are all rectangular, and their lengths and widths are equal. The solution of this invention is not limited to the above embodiment.

[0186] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence to form a heating target assembly, which is fixed on a chassis 1 to form a heating target box. In this embodiment, the entire heating target assembly is fixed to the sheet metal chassis 1 with screws.

[0187] As shown in Figure 9, the entire chassis 1 (e.g., a sheet metal chassis) is machined from 2mm steel plate and then powder-coated with black paint. The heating unit uses a heating rod for heating, and the heat is conducted to the heat source 21. A sheet metal door assembly 6 with a fan is installed on chassis 1; the fan is used for heat dissipation. The calibration plate 5 serves as a side panel of chassis 1, and the heat source fixing plate 3 is located inside chassis 1. A human-machine interface 8, a power switch 9, a start switch 10, and indicator lights 7 for displaying the heating status are installed on chassis 1. The human-machine interface 8 and indicator lights 7 are connected to a controller. The controller contains a control unit and a temperature control module. The control unit is connected to the temperature detection element 23, the temperature control module, the human-machine interface 8, and the indicator lights 7. The temperature control module is connected to the heating element 22, and the temperature of each heat source 21 is controlled by the temperature control module. The temperature of each heating unit is controlled through the human-machine interface 8, and each target can be controlled individually as required. Two handles are installed on chassis 1 for transporting the heating targets. Its working principle is that the heating rod heats the material and transfers the heat to the heat source 21. The thermocouple has a built-in temperature sensor to sense the real-time temperature of the heat source 21 and displays the actual sensed temperature through the human-machine interface 8. If the actual measurement requires the temperature to be adjusted to rise or fall, the temperature can be set through the human-machine interface 8. Each temperature can be controlled individually (adjustment range 25°~120°).

[0188] Furthermore, the heating target box is mounted on the frame 11. In this embodiment, the heating target box is fixed to the frame 11 with screws.

[0189] Furthermore, the lower end of the frame 11 is provided with casters 116.

[0190] The frame 11 includes a body frame 111, a fixing plate 112, a double sheet metal door 113, a stainless steel cover plate 114, and a frame base plate 115. The body frame 111 is made of stainless steel square tubing welded around its perimeter. The rear and side fixing plates 112 are made of stainless steel sheet metal and fixed to the frame 11 with screws. The front of the body frame 111 has a stainless steel double door, and the top stainless steel cover plate 114 is fixed to the frame 11 for fixing the heating target box. The base plate of the frame 11 is made of stainless steel sheet metal and is equipped with a caster wheel with feet at each corner for movement and fixation.

[0191] The calibration plate of this invention has target holes of various diameters, which can be adapted to tests at various distances (such as short distance and long distance, where distance refers to the distance between the infrared thermal imaging device and the calibration plate). It has strong versatility, can meet the optical axis testing needs of existing products, and can satisfy multiple uses of one machine.

[0192] Each target in this invention employs an individual temperature control system, enabling precise temperature control. This results in highly accurate and stable targets, ensuring temperature consistency and improving test accuracy.

[0193] Figure 12 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 12, the electronic device may include: a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can call logical instructions in the memory to execute a method for fusing infrared and visible light images. This method includes: obtaining affine matrices between visible and infrared images with corresponding relationships through a calibration board, and statistically analyzing multiple affine matrices at different distances to obtain an affine matrix set, wherein the correspondence is based on photographs taken of the same object at the same location; obtaining a set of correspondences between the affine matrices and relevant information of infrared images at different distances based on the affine matrix set; obtaining the affine matrices to be fused for the infrared and visible light images to be fused based on the correspondence set; and performing image fusion on the infrared and visible light images to be fused based on the affine matrices to be fused.

[0194] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the method for fusing infrared light images and visible light images provided in the above-described method embodiments, the method including: obtaining an affine matrix between a visible light image and an infrared light image that have a corresponding relationship through a calibration plate, and statistically analyzing multiple affine matrices at different distances to obtain an affine matrix set, the correspondence being that the same object is photographed at the same location; obtaining a set of correspondences between the affine matrix and relevant information of infrared light images at different distances according to the affine matrix set; obtaining an affine matrix to be fused for the infrared light image and the visible light image to be fused according to the set of correspondences; and performing image fusion on the infrared light image and the visible light image to be fused according to the affine matrix to be fused.

[0196] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for fusing infrared and visible light images provided in the above embodiments. The method includes: obtaining affine matrices between visible and infrared images that have a corresponding relationship through a calibration plate, and statistically analyzing multiple affine matrices at different distances to obtain an affine matrix set, wherein the correspondence is based on photographs taken of the same object at the same location; obtaining a set of correspondences between the affine matrices and related information of infrared images at different distances according to the set of affine matrices; obtaining the affine matrices to be fused for the infrared and visible light images to be fused according to the set of correspondences; and performing image fusion on the infrared and visible light images to be fused according to the affine matrices to be fused.

[0197] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0198] The above description is only a partial implementation of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for fusing infrared and visible light images, characterized in that, The method includes: obtaining an affine matrix between a visible light image and an infrared light image that have a corresponding relationship through a calibration plate, and statistically analyzing multiple affine matrices at different distances to obtain an affine matrix set, wherein the correspondence is based on taking pictures of the same object at the same position; the calibration plate body is provided with at least two sets of target holes, each set of target holes corresponding to one or more heating units; obtaining a set of correspondences between the affine matrix and the relevant information of infrared light images at different distances according to the affine matrix set; obtaining the affine matrix to be fused between the infrared light image to be fused and the visible light image to be fused according to the correspondence set; performing image fusion between the infrared light image to be fused and the visible light image to be fused according to the affine matrix to be fused; wherein obtaining the relevant information between the affine matrix and the infrared light images at different distances according to the affine matrix set includes: obtaining the relevant information of the infrared light image to be fused and the visible light image to be fused through a Hall element according to the focusing principle, wherein the relevant information includes: an x-direction component and a y-direction component, wherein the x-direction component is the length information of the infrared light image, and the y-direction component is the width information of the infrared light image; wherein, the step of obtaining the relevant information between the affine matrix and the infrared light image at different distances through a calibration plate ... The calibration board acquires the affine matrix between corresponding visible light and infrared images, including: acquiring visible light and infrared images at a set calibration distance; extracting and saving the center points of the calibration board in the visible light and infrared images at the set calibration distance; calculating the affine matrix at the set calibration distance by registering the center points of each circle in the visible light and infrared images at the set calibration distance; and calculating the affine matrix of the corresponding infrared image for each set calibration distance based on the affine matrix at the set calibration distance, specifically including: affine matrices for all calibration distances. The average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix are calculated. Using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance are corrected. Based on the corrected X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated, and the affine matrix at each set distance is generated.

2. The method as described in claim 1, characterized in that, The correction of the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance specifically includes: using the nearest neighbor distance method to correct the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

3. The method as described in claim 2, characterized in that, The step of calculating the X-direction offset component and Y-direction offset component of the affine matrix at each set distance based on the X-direction offset component and Y-direction offset component of the affine matrix at each set distance after correction includes: calculating the X-direction offset component and Y-direction offset component of the affine matrix at each set distance using interpolation based on the X-direction offset component and Y-direction offset component of the affine matrix at each set distance after correction.

4. The method as described in claim 1, characterized in that, The step of performing image fusion on the infrared light image and the visible light image to be fused according to the affine matrix to be fused includes: performing an affine transformation on the visible light image to be fused according to the affine matrix to be fused to obtain a transformed visible light image; extracting high-frequency image information from the transformed visible light image; and superimposing the high-frequency image information onto the infrared light image to be fused to obtain a fused image.

5. The method according to any one of claims 1-4, characterized in that, The step of obtaining a set of correspondences between affine matrices and images at different distances based on the affine matrix set includes: obtaining an average affine matrix based on the affine matrix set, including: obtaining the average value of the rotation component and the average value of the scaling component of the affine matrices in the affine matrix set; obtaining an average affine matrix based on the average value of the rotation component and the average value of the scaling component; obtaining the offset of the affine matrix to be obtained at different distances using a nearest neighbor offset distance algorithm based on the average affine matrix and the affine matrix to be obtained, wherein the offset includes: an offset t1 in the X direction and an offset t2 in the Y direction; and performing interpolation calculations on multiple offsets to obtain a set of correspondences between the offsets and the affine matrices.

6. A system for fusing infrared and visible light images, characterized in that, A method for fusing infrared and visible light images as described in claim 1, the system comprising: an affine matrix set acquisition module, configured to acquire affine matrices between visible and infrared images having a corresponding relationship through a calibration plate, and to collect a set of affine matrices at different distances, wherein the correspondence is based on photographs taken of the same object at the same location; a correspondence set acquisition module, configured to acquire a set of correspondences between affine matrices and relevant information of infrared images at different distances based on the affine matrix set; a affine matrix to be fused acquisition module, configured to acquire the affine matrices of the infrared and visible light images to be fused based on the correspondence set; and an image fusion module, configured to perform image fusion on the infrared and visible light images to be fused based on the affine matrices to be fused.

7. An apparatus for fusing infrared and visible light images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for fusing infrared light images and visible light images as described in any one of claims 1 to 5.

Images