Method and device for fusing infrared image with visible light image, and method for registering infrared image with visible light image

Abstract

A method and device for fusing an infrared image with a visible light image, and a method for registering an infrared image with a visible light image. In the method for fusing an infrared image with a visible light image, the centers of circles of calibration plates in a visible light image and an infrared image are extracted under each calibration distance and are then saved, affine matrices under various distances are calculated by means of registering the centers of circles of the visible light image with the centers of circles of the infrared image, and the average value of all the affine matrices is calculated; by using the centers of circles and average values of rotation and scaling components in X and Y directions in the affine matrices under the calibration distance, combined with a nearest neighbor distance method, X-direction offset components and Y-direction offset components under the distance are solved, and finally, the X-direction offset components and the Y-direction offset components under the distances are obtained by means of interpolation, and thus the generation of an affine matrix of the visible light image and the infrared image under any distance is realized; and finally, visible light high-frequency information is superimposed on the infrared image to realize fusion, thereby realizing the registration and fusion of double light images under any distance.

Description

Infrared and visible light image fusion method, device and registration method Technical Field

[0001] The present invention belongs to the technical field of image processing technology, and specifically relates to an infrared and visible light image fusion method, equipment and registration method. Background Art

[0002] Most existing infrared-visible light fusion methods are based on fixed-focus infrared lenses and fixed-focus visible light lenses for image acquisition. For example, after the CPU obtains the infrared image and visible light image, it executes the corresponding dual-light fusion algorithm to superimpose the target details in the visible light image onto the infrared image.

[0003] Because the camera has a fixed focus, both infrared and visible light images are only for target scenes at a specific distance. However, the infrared lens usually cannot maintain the same focal length and field of view as the visible light lens. In close-range target scenes, the phenomenon of inconsistent object imaging size will be particularly obvious. This leads to the current image fusion method in the industry only having a good dual-light fusion effect at a certain distance, and large image fusion errors will occur for close-range scenes.

[0004] Existing image fusion methods include:

[0005] 1. Based on the anisotropic diffusion fusion (ADF) method, the original image is decomposed into a base layer and a detail layer. After Karhunen-Loeve transformation, they are linearly superimposed to finally combine them to obtain a fused image. This algorithm is easy to implement and has low loss after fusion, but the final visual effect is not ideal.

[0006] 2. Based on convolutional neural networks (CNN) and residual neural networks (ResNet), convolutional neural networks and residual neural networks are built, and the parameters are optimized through extensive training to fuse images. The performance indicators of the fused images are better, but it requires building a dataset and consumes a lot of time, which makes training time-consuming.

[0007] 3. Based on the gradient transfer fusion (GTF) method, an objective function is first constructed and then solved iteratively to make the fused image closer to the infrared image and retain more appearance information. Its advantage is that the visual effect after fusion is good, but due to the need for iterative solution, the algorithm cannot meet the requirements of real-time processing.

[0008] 4. Based on the latent low-rank representation (LatLRR) method, the original image is first decomposed into a low-rank part and a salient part, and then two different strategies are used to fuse them and finally reconstruct the image. However, its visual effect is not very ideal.

[0009] Summary of the Invention

[0010] The purpose of the present invention is to overcome at least one defect in the prior art and provide an infrared-visible light image fusion method, device and registration method. The present invention realizes dual-light image registration and fusion at any distance, can achieve real-time performance, and achieve good visual effects.

[0011] The technical solution of the present invention is implemented as follows: The present invention discloses an infrared and visible light image fusion method, comprising the following steps:

[0012] Acquire visible light images and infrared images;

[0013] The dual-light image fusion is performed, specifically including: obtaining the current target distance, obtaining the corresponding affine matrix according to the current target distance, and fusing the visible light image and the infrared image using the affine matrix corresponding to the current target distance.

[0014] Furthermore, before the image fusion step, the following step is also included: obtaining and saving the affine matrix of the visible light image corresponding to the infrared image at a set distance.

[0015] Furthermore, obtaining the affine matrix of the visible light image corresponding to the infrared image at the set distance specifically includes:

[0016] Obtain visible light images and infrared images at each set calibration distance;

[0017] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0018] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at each calibration distance.

[0019] Furthermore, the affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated according to the affine matrix at the set calibration distance, specifically including:

[0020] The rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are averaged. The X and Y offset components of the affine matrix at each calibration distance are corrected using the center of the calibration plate in the visible light image and infrared image at each calibration distance and the average of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances.

[0021] According to the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are calculated to generate the affine matrix at each set distance.

[0022] Furthermore, an affine matrix at each set distance is generated, specifically including: generating an affine matrix at each set distance based on the X-direction offset component and the Y-direction offset component of the affine matrix at each set distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances.

[0023] Furthermore, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are corrected, specifically including:

[0024] The nearest neighbor distance method is used to correct the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

[0025] Furthermore, based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are calculated, specifically including: based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, using the interpolation method to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each set distance.

[0026] Furthermore, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated by using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, specifically including:

[0027] The nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance using the center of the calibration plate in the visible light image and infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances.

[0028] Furthermore, based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are calculated, specifically including: based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, using the interpolation method to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each set distance.

[0029] Furthermore, the current target distance is obtained, and its corresponding affine matrix is ​​obtained according to the current target distance, specifically including: obtaining its corresponding affine matrix according to the current target distance by looking up a table, or obtaining its corresponding X-direction offset component and Y-direction offset component of the affine matrix according to the current target distance by looking up a table, assigning the X-direction offset component and Y-direction offset component of the affine matrix at the current target distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances to the affine matrix, and thus the affine matrix corresponding to the current target distance can be obtained.

[0030] Furthermore, the visible light image and the infrared image are fused using the affine matrix corresponding to the current target distance to obtain a fused image, specifically including: performing an affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fusing the transformed visible light image with the acquired infrared image to obtain a fused image.

[0031] Furthermore, the transformed visible light image is fused with the acquired infrared image to obtain a fused image, specifically comprising: extracting visible light high-frequency information from the transformed visible light image, and superimposing the visible light high-frequency information on the infrared image to obtain a fused image.

[0032] Furthermore, the fused image is converted into an RGB fused image.

[0033] Furthermore, after obtaining the visible light image and the infrared image, distortion correction is first performed, and then the distortion-corrected visible light image and infrared image are used in the following steps.

[0034] Furthermore, after obtaining the current target distance, the following steps are also included: determining whether the current target distance is greater than a preset value; if so, scaling the visible light image or infrared image using fixed cropping and scaling parameters to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion; if not, scaling the visible light image or infrared image using a table lookup to obtain corresponding cropping and scaling parameters according to the current target distance to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for the image fusion step.

[0035] The present invention also discloses an infrared-visible light image fusion device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the infrared-visible light image fusion method described above are implemented.

[0036] The present invention also discloses an infrared-visible light image registration method, comprising the following steps:

[0037] Obtain visible light images and infrared images at each set calibration distance;

[0038] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0039] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at each calibration distance.

[0040] The present invention has at least the following beneficial effects:

[0041] The present invention proposes a multi-distance visible light and infrared dual-light registration and fusion method. The method extracts and saves the centers of calibration plates in visible light images and infrared images at each calibration distance. By aligning the centers of each circle in the visible light image with the centers of each circle in the infrared image, the affine matrix at each calibration distance is calculated, and the average of all affine matrices is obtained. The average of the rotation and scaling components in the X and Y directions in the affine matrices at all calibration distances is used, and the nearest neighbor distance method is used to calculate the X-direction offset component and the Y-direction offset component at each distance. Finally, the X-direction offset component and the Y-direction offset component at each distance are obtained by interpolation. The affine matrix of the visible light image and the infrared image at any distance is generated. Finally, the visible light high-frequency information is superimposed on the infrared image to achieve fusion. The dual-light image registration and fusion at any distance are realized, which can achieve real-time performance and achieve good visual effects.

[0042] The method of the present invention can realize an automated process in the calibration stage, and only needs to collect pictures to complete the automated calibration process and generate data; the fully automated process saves manpower and material resources.

[0043] The method of the present invention can achieve the fusion effect of infrared and visible light images in different fields of view, that is, adding the features of the infrared image and superimposing the edge information of the visible light to achieve a good visual effect.

[0044] The method of the present invention can realize real-time processing of the algorithm on the APP side, so that the frame rate of the fused image meets the visual requirements of the human eye and achieves the effect of real-time fusion.

[0045] The present invention also proposes a method for fusion of infrared and visible light based on a laser ranging module. This method uses the target distance obtained by the laser ranging module to dynamically adjust the cropping and scaling parameters of the visible light image, thereby resolving the problem of inconsistent dual-light fusion images in close-range scenarios. This method is simple, effective, and highly efficient, making it suitable for low-cost product development solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] FIG1 is a flow chart of an infrared-visible light image registration method provided by an embodiment of the present invention;

[0048] FIG2 is a flow chart of an infrared-visible light image fusion method provided by an embodiment of the present invention;

[0049] FIG3 is a flowchart of distortion correction provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of the extraction and sorting of the center of the IR calibration plate of the high dynamic range infrared image;

[0051] Figure 5 is a schematic diagram of the extraction and sorting of the center of the VIS calibration plate of the visible light image;

[0052] FIG6 is a flow chart of an infrared-visible light image fusion method provided by another embodiment of the present invention;

[0053] FIG7 is a schematic diagram of the appearance design of a camera and a ranging module provided by an embodiment of the present invention;

[0054] Figure 8 shows the dual-light fusion effect at 1.5 meters using fixed magnification parameters;

[0055] Figure 9 shows the dual-light fusion effect at 0.5 meters using fixed magnification parameters;

[0056] Figure 10 shows the effect of dual-light fusion using updated appropriate parameters at 0.5 meters;

[0057] Figure 11 is a high dynamic range infrared image IR;

[0058] Figure 12 is a fused image;

[0059] FIG13 is a schematic structural diagram of a calibration plate provided in an embodiment of the present invention;

[0060] FIG14 is a cross-sectional view of a calibration plate provided in an embodiment of the present invention;

[0061] FIG15 is an enlarged view of portion P of FIG14 ;

[0062] FIG16 is a schematic structural diagram of an infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0063] FIG17 is an exploded view of a heating target box of an infrared dual-light fusion test device provided by an embodiment of the present invention;

[0064] FIG18 is a schematic structural diagram of a heating unit of an infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0065] FIG19 is an exploded view of the frame of the infrared dual-light fusion testing device provided in an embodiment of the present invention.

[0066] In the accompanying drawings, 1 is a chassis, 2 is a heating unit, 21 is a heat source body, 211 is a first mounting hole, 212 is a second mounting hole, 22 is a heating element, 23 is a temperature detection element, 24 is a first heat insulation board, 3 is a heat source fixing plate, 4 is a second heat insulation board, 5 is a calibration board, 51 is a calibration board body, 52 is a first target hole, 53 is a second target hole, 6 is a sheet metal door assembly, 7 is an indicator light, 8 is a human-machine interface, 9 is a power switch, 10 is a start switch, 11 is a frame, 111 is a body frame, 112 is a fixing plate, 113 is a double-opening sheet metal door, 114 is a stainless steel cover, 115 is a frame bottom plate, and 116 is a caster. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] Example 1

[0069] 1 , an embodiment of the present invention provides an infrared-visible light image registration method, comprising the following steps:

[0070] Acquire paired visible light images and infrared images at each set calibration distance;

[0071] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0072] The affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix at each calibration distance.

[0073] Furthermore, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, specifically including:

[0074] The average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances is obtained, where the affine matrix includes the X-direction scaling component a11, the X-direction rotation component a12, the Y-direction rotation component a21, the Y-direction scaling component a22, and the X-direction offset component t1 and the Y-direction offset component t2;

[0075] Using the center of the calibration plate in the visible light image and infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, recalculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance (if not recalculated, the offset component cannot be increased or decreased regularly), and obtain the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance;

[0076] Calculate the X-direction offset component and the Y-direction offset component of the affine matrix at any distance based on the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance;

[0077] The affine matrix at any distance can be generated based on the X-direction offset component and Y-direction offset component of the affine matrix at each set distance, as well as the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibrated distances. That is, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are assigned to the affine average matrix, or the X-direction offset component and Y-direction offset component of the affine matrix at each set distance, as well as the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibrated distances, are assigned to the affine matrix to generate the affine matrix at any distance.

[0078] The average of the affine matrices at all calibration distances is calculated to obtain the affine average matrix.

[0079] Furthermore, the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances (that is, the mean value of the rotation and scaling matrices in the affine matrix at each calibration distance) are used to inversely calculate the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance.

[0080] Furthermore, the nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances.

[0081] The principle of the nearest neighbor distance method of the present invention is as follows: at a single distance, the coordinates of each circle center in the visible cursor calibration image are cyclically brought in, and a set of offset vectors in the X direction and the offset vector in the Y direction corresponding to each circle center coordinate can be calculated through the coordinates of the circle center in the corresponding infrared calibration image. For example, the first circle center after sorting in the visible cursor calibration image corresponds to the first circle center after sorting in the infrared calibration image, and the corresponding offset vector in the X direction and the offset vector in the Y direction are calculated, and recorded as an, bn. Then, by traversing the infrared calibration image center corresponding to all circle centers at this distance, the center of the visible cursor calibration image is converted to the center of the circle on the infrared calibration image, and the straight-line distance dis with the center extracted from the infrared calibration image is calculated; for example, 25 circle centers are extracted from the infrared calibration image and the visible cursor calibration image respectively, and the cycle is repeated 25 times, and the minimum straight-line distance is accumulated to obtain a set of an, bn, which are recorded as the offset vector in the X direction and the offset vector in the Y direction under the current distance.

[0082] The X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated by using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances using the nearest neighbor distance method. Specifically, at a single calibration distance, the center of the calibration plate in the visible light image and the infrared image at the calibration distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances are obtained. For each pair of corresponding visible light calibration image centers and infrared calibration image centers, the X-direction offset component and the Y-direction offset component and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances are first obtained to form a new affine matrix (a pair of corresponding visible light calibration image centers and infrared calibration image centers). The center of the circle corresponds to a new affine matrix), and each new affine matrix is ​​used to calculate the coordinates of the center of each visible light calibration image converted to the center of the infrared calibration image, which are marked as the second center of the circle on the infrared calibration image. The offset vector in the X direction and the offset vector in the Y direction as well as the straight-line distance dis between the center of the circle and the first center coordinate in the corresponding infrared calibration image (that is, the center of the circle in the extracted infrared calibration image) are calculated. The straight-line distances dis corresponding to all the center of the circle are summed to obtain the cumulative straight-line distance values ​​corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X direction and the offset vector in the Y direction, denoted as an and bn. The set of offset vectors in the X direction and the offset vector in the Y direction corresponding to the minimum value of the cumulative straight-line distance are used as the offset vectors in the X direction and the offset vectors in the Y direction at the current calibration distance.

[0083] Furthermore, based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated, specifically including: based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated using an interpolation method.

[0084] Based on the same inventive concept, the present invention also provides an infrared-visible light image registration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the infrared-visible light image registration method described above are implemented. The infrared-visible light image registration device also includes a visible light camera, an infrared camera, and a ranging module.

[0085] Example 2

[0086] 1 and 2 , an embodiment of the present invention provides an infrared-visible light image fusion method, comprising the following steps:

[0087] Obtain and save the affine matrix of the visible light image corresponding to the infrared image at a set distance (any distance or various distances);

[0088] Acquire paired visible light and infrared images;

[0089] The dual-light image fusion is performed, specifically including: obtaining the current target distance, obtaining the corresponding affine matrix according to the current target distance, and fusing the visible light image and the infrared image using the affine matrix corresponding to the current target distance to obtain a fused image.

[0090] Furthermore, to obtain the affine matrix of the visible light image corresponding to the infrared image at each distance, the following steps can be used:

[0091] Acquire paired visible light images and infrared images at each set calibration distance;

[0092] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0093] The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated based on the affine matrix at each calibration distance.

[0094] Furthermore, for a specific method of calculating the affine matrix of the visible light image corresponding to the infrared image at any distance based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, see Example 1.

[0095] Furthermore, the current target distance is obtained, and its corresponding affine matrix is ​​obtained according to the current target distance, specifically including: obtaining its corresponding affine matrix according to the current target distance by looking up a table, or obtaining its corresponding X-direction offset component and Y-direction offset component of the affine matrix according to the current target distance by looking up a table, and generating the affine matrix at the current target distance according to the X-direction offset component and the Y-direction offset component of the affine matrix at the current target distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibrated distances, that is, substituting the X-direction offset component and the Y-direction offset component of the affine matrix at the current target distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibrated distances into the affine matrix, so as to obtain the affine matrix at the current target distance.

[0096] Furthermore, the visible light image and the infrared image are fused using the affine matrix corresponding to the current target distance to obtain a fused image, specifically including: performing an affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fusing the infrared image with the transformed visible light image to obtain a fused image.

[0097] Furthermore, the infrared image is fused with the transformed visible light image to obtain a fused image, specifically including: extracting visible light high-frequency information from the transformed visible light image, and superimposing the visible light high-frequency information on the infrared image to obtain a fused image.

[0098] Furthermore, the fused image is converted into an RGB fused image.

[0099] Furthermore, after obtaining the visible light image and the infrared image, distortion correction is first performed, and then the distortion-corrected visible light image and infrared image are used in a calibration step and / or an image fusion step.

[0100] Furthermore, after obtaining the current target distance, the following steps are also included: determining whether the current target distance is greater than a preset value; if so, determining that the target is within the focal length overlap range of the infrared lens and the visible light lens; at this time, using a fixed cropping and scaling parameter to scale the visible light image to obtain a scaled visible light image, and using the scaled visible light image for the image fusion step; if not, determining that the target is outside the focal length overlap area shared by the infrared lens and the visible light lens; at this time, scaling the visible light image by obtaining the corresponding cropping and scaling parameter according to the current target distance table to obtain a scaled visible light image, and using the scaled visible light image for the image fusion step.

[0101] The above process is described in detail below through a specific example.

[0102] An infrared-visible light image fusion method comprises the following steps:

[0103] Calibration process: Obtain the affine matrix of the visible light image corresponding to the infrared image at each distance;

[0104] Image fusion process: The target distance is obtained through the Hall contrast table, and the affine matrix at the corresponding distance is obtained by looking up the table for real-time calculation and display of the fused image.

[0105] The calibration process includes:

[0106] Acquire 14-bit / 16-bit high dynamic range infrared images IR and 8-bit visible light images VIS.

[0107] Identify the center of the calibration plate on the 14-bit / 16-bit high dynamic range infrared image IR and the 8-bit visible light image VIS, store them as ir_centerCiclePoints and vis_centerCiclePoints respectively, and sort them, see Figures 4 and 5.

[0108] According to vis_centerCiclePoints and ir_centerCiclePoints, the affine matrix VISToIRMatrixs is obtained.

[0109] AffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints,VISToIRMatrixs);

[0110] The above process is used to perform calibration at the set calibration distance. For example, this embodiment has 9 calibration distances, ranging from 1 meter to 9 meters, a total of 9 groups. Of course, the number of calibration distances is not limited to 9 and can be adjusted as needed. All the extracted circle centers vis_centerCiclePointList and ir_centerCiclePointList are saved and the visible light image center and the infrared image center are matched one by one, thereby generating the affine matrix VISToIRMatrixs at each distance.

[0111] Take the average of the rotation and scaling components in the affine matrix under the 9 sets of data (i.e. a11, a12, a21, a22) and save it as VISToIRMatrix_average, which is expressed as:

[0112] In the affine matrix, a11 represents the scaling ratio in the X direction, a12 represents the rotation ratio in the X direction, a21 represents the rotation ratio in the Y direction, a22 includes the scaling ratio in the Y direction, t1 represents the translation distance in the X direction, and t2 represents the translation distance in the Y direction. In a two-dimensional image, the X-direction vector and the Y-direction vector have different differences in the affine transformation.

[0113] For example, in this embodiment, at a distance of 1 meter, a11 = 2.71771, a12 = 0.0005355351, t1 = -1027.16, a21 = -0.00148628, a22 = 2.77923, t2 = -1125.53.

[0114] The study found that the X-direction offset distance changes more at close distances, while the change is almost negligible at longer distances.

[0115] Using the saved vis_centerCiclePointList and ir_centerCiclePointList, and using the average affine matrix of VISToIRMatrix_average, the nearest neighbor offset distance method is used to calculate the X direction component t1 and the Y direction component t2 at each distance.

[0116] ReverseCalAffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints);

[0117] Interpolate the t1 and t2 values ​​calculated for each group, and calculate the t1 and t2 values ​​at each distance from 1 to 20 meters in units of 10 cm, where the x and y values ​​range from 1 to 9 meters, and the range is 1 to 20 meters with an interval of 0.1 meters. The output is a lookup table of the X-direction offset vector and the Y-direction offset vector;

[0118] Interp1HermitePchip(distance,x,range,t1);

[0119] Interp1HermitePchip(distance,y,range,t2);

[0120] Finally, the target distance is obtained according to the Hall contrast table, and the t1 and t2 values ​​at the corresponding distance are obtained according to the target distance lookup table and assigned to VISToIRMatrix_average for calculating the fused image at the current distance.

[0121] The image fusion process specifically includes:

[0122] A 14-bit / 16-bit high dynamic range infrared image IR and an 8-bit visible light image VIS are obtained. A high dynamic range infrared image IR according to an embodiment is shown in FIG11 .

[0123] Convert 14-bit / 16-bit high dynamic range infrared image IR and 8-bit visible light image VIS to YUV420 format IR_yuv and VIS_yuv.

[0124] bgrtoyuv420(IR_yuv,IR.data,n_ir_width,n_ir_height);

[0125] Get the current distance based on the Hall value contrast table, and get the affine matrix at that distance based on the calibration storage file and distance:

[0126] VISToIRMatrixs=SetAffineMat(distance);

[0127] Perform an affine transformation on the Y component of VIS_yuv according to the affine matrix obtained in the previous step, then extract the high-frequency information and superimpose it on the Y component of IR_yuv to obtain FusionImgYUV;

[0128] FusionImgYUV=ImgRegAndFusYUV(VISToIRMatrixs);

[0129] The IR_yuv of the overlay information is converted into an RGB image and displayed. The fused image is shown in FIG12 .

[0130] yuv420tobgr(Imgfuse.data,FusionImgYUV,n_ir_width,n_ir_height).

[0131] After obtaining the visible light image and infrared image, distortion correction is first performed, and then the distortion-corrected visible light image and infrared image are used in the calibration process and image fusion process.

[0132] An important application of image geometric transformation is to eliminate geometric distortion in digital images caused by cameras. Restoring a geometrically distorted image to its normal state using a computer image processing system is called geometric correction of image distortion. A mathematical model for geometric correction is established, and the model parameters are determined using known conditions. Finally, the model is used to perform geometric correction on the image. See Figure 3. The specific steps for distortion correction are as follows:

[0133] 1. Collect about 15 images of the calibration plate;

[0134] 2. Determine the number of internal points of the calibration plate according to the calibration plate used, and find the coordinates of the center of the calibration plate;

[0135] 3. Further extract sub-pixel corner information;

[0136] 4. Calculate and obtain the camera's intrinsic parameter matrix and distortion coefficient through the opencv function calibrateCamera;

[0137] 5. Correction and repair of distorted images. Use the opencv function undistort to obtain the distorted image.

[0138] Both the calibration process and the image fusion process use 14-bit / 16-bit high dynamic range infrared images IR and 8-bit visible light images VIS after distortion correction.

[0139] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the infrared-visible light image fusion method described above are implemented. The infrared-visible light image fusion device also includes a visible light camera, an infrared camera, and a ranging module.

[0140] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, comprising:

[0141] An affine matrix acquisition module is used to obtain and save the affine matrix of the visible light image corresponding to the infrared image at a set distance;

[0142] A visible light image acquisition module, used for acquiring visible light images;

[0143] Infrared image acquisition module, used for acquiring infrared images;

[0144] Distance acquisition module, used to obtain the current target distance;

[0145] The image fusion module is used to obtain the corresponding affine matrix according to the current target distance, perform affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fuse the infrared image with the transformed visible light image to obtain a fused image.

[0146] The fields of view of visible light and infrared are different. The present invention realizes the registration and fusion of dual-light images at any distance, can achieve real-time performance, and achieve good visual effects at the same time. The method of the present invention can realize an automated process in the calibration stage. It only needs to collect images to complete the automated calibration process to generate data. The fully automated process saves manpower and material resources. The method of the present invention can achieve the fusion effect of infrared and visible light images of different fields of view, that is, add the features of the infrared image, and superimpose the edge information of the visible light (that is, extract the Y channel in the visible light YUV, after registration, extract the edge information by subtracting the Y channel after mean filtering from the original Y, and then superimpose it on the Y channel of the infrared YUV to obtain the fused image YUV, and then convert it back to the fused image RGB for display), to achieve good visual effects. This method can effectively realize real-time processing on the Android side, and the visual effect experience is better.

[0147] The method of the present invention can realize real-time processing of the algorithm on the APP side, so that the frame rate of the fused image meets the visual requirements of the human eye and achieves the effect of real-time fusion.

[0148] Example 3

[0149] 6 , an embodiment of the present invention provides an infrared-visible light image fusion method, comprising the following steps:

[0150] Acquire paired visible light and infrared images;

[0151] Obtain the current target distance and determine whether the current target distance is greater than a preset value (such as 1.2 meters, which can be set as needed). If so, it is determined that the target is within the focal length overlap range of the infrared lens and the visible light lens. At this time, the visible light image or infrared image is scaled using a fixed cropping and scaling parameter to obtain a scaled visible light image or infrared image. If not, it is determined that the target is outside the focal length overlap area shared by the infrared lens and the visible light lens. At this time, the visible light image or infrared image is scaled using the corresponding cropping and scaling parameters obtained according to the current target distance to obtain a scaled visible light image or infrared image.

[0152] The scaled visible light image is fused with the unscaled infrared image or the scaled infrared image is fused with the unscaled visible light image.

[0153] Furthermore, the scaled visible light image is fused with the unscaled infrared image, specifically including: obtaining the corresponding affine matrix according to the current target distance, performing an affine transformation on the scaled visible light image according to the affine matrix corresponding to the current target distance, and fusing the infrared image with the transformed visible light image to obtain a fused image.

[0154] Furthermore, the scaled infrared image and the unscaled visible light image are subjected to dual-light image fusion, specifically including: obtaining the corresponding affine matrix according to the current target distance, performing affine transformation on the unscaled visible light image according to the affine matrix corresponding to the current target distance, and fusing the infrared image with the transformed visible light image to obtain a fused image.

[0155] Obtaining the corresponding affine matrix according to the current target distance specifically includes: obtaining the corresponding affine matrix according to the current target distance by looking up the table, or obtaining the X-direction offset component and the Y-direction offset component of the corresponding affine matrix according to the current target distance by looking up the table, assigning the X-direction offset component and the Y-direction offset component of the affine matrix at the current target distance to the stored affine average matrix (the affine average matrix is ​​obtained by taking the average value of the affine matrices at all calibration distances), and thus the affine matrix at the current target distance can be obtained.

[0156] Furthermore, to obtain the affine matrix of the visible light image corresponding to the infrared image at each distance, the following steps can be used:

[0157] Acquire paired visible light images and infrared images at each set calibration distance;

[0158] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0159] The affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix at each calibration distance.

[0160] Furthermore, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, specifically including:

[0161] Calculate the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, where the affine matrix contains the X-direction scaling component a11, the X-direction rotation component a12, the Y-direction rotation component a21, the Y-direction scaling component a22, and the X-direction offset component t1 and the Y-direction offset component t2;

[0162] Using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated to obtain the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance;

[0163] Calculate the X-direction offset component and the Y-direction offset component of the affine matrix at any distance based on the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance;

[0164] The affine matrix at any distance is generated by assigning the X-direction offset component and the Y-direction offset component of the affine matrix at any distance and the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibration distances to the affine matrix. That is, the X- and Y-direction rotation and scaling components of the affine matrix at any distance are the same as the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibration distances.

[0165] Furthermore, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated by using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, specifically including:

[0166] The nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance using the center of the calibration plate in the visible light image and infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances.

[0167] Furthermore, based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated, specifically including: based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated using an interpolation method.

[0168] Furthermore, when the current target distance is less than a preset value, the visible light image or infrared image is scaled by obtaining the corresponding cropping and scaling parameters according to the current target distance lookup table to obtain the scaled visible light image or infrared image, and the scaled visible light image or infrared image is used in the image fusion step.

[0169] The method of the present invention uses the target distance obtained by a laser ranging module to dynamically adjust the cropping and scaling parameters of visible light or infrared images, thereby resolving the problem of inconsistent dual-light fusion images in close-range scenarios. This method is simple, effective, and highly efficient, making it suitable for low-cost product development solutions.

[0170] This invention uses the VL53L1X short-range, low-power laser ranging module produced by ST to measure target distance in real time. The VL53L1X is ST's third-generation FlightSense time-of-flight sensor. Utilizing ST's latest ToF technology, it can measure distances up to 4 meters and frequencies up to 50Hz. The VL53L1X integrates a physical infrared filter and optical components, enabling distance measurement regardless of target color and reflectivity, while offering enhanced anti-interference capabilities.

[0171] Figure 7 shows the design of the camera and ranging module. Laser ranging module 3, installed between infrared camera 2 and visible light camera 1, facilitates accurate ranging. In medium-range measurement mode, the VL53L1X can measure up to 2.9 meters. In short-range measurement mode, the 1.3-meter measurement accuracy reaches ±25 mm, meeting the design accuracy requirements of this solution.

[0172] The ranging module updates the target distance in real time. If the target distance is greater than a preset value, such as 1.2 meters, the target is determined to be within the focal length overlap range of the infrared and visible light lenses. At this time, a fixed magnification parameter is used for dual-light fusion, as shown in Figure 8. If the target distance is less than a preset value, such as 1.2 meters, the target is determined to be outside the focal length overlap area shared by the two cameras. At this time, an index table of distance and magnification is queried to select the appropriate magnification factor for dual-light image fusion, as shown in Figure 10. Figure 9 shows the effect of dual-light fusion using a fixed magnification parameter, i.e., a traditional solution, at 0.5 meters. Comparing Figures 9 and 10, it can be seen that the method of the present invention achieves significantly better dual-light image fusion than the traditional solution.

[0173] In order to obtain appropriate cropping and scaling parameters within 1.2 meters, experiments need to be done in advance to obtain appropriate values.

[0174] The experimental method is to manually set the appropriate cropping and scaling at different distances as shown below, and record them until the relevant parameters for each distance position are obtained. In the actual experiment, the results we obtained are the close-range scaling parameter index table shown in Table 1:

[0175] Table 1

[0176] In Table 1, w represents the width of the cropped image, x represents the horizontal offset coordinate of the cropped image relative to the original image, h represents the height of the cropped image, and y represents the vertical offset coordinate of the cropped image relative to the original image. After obtaining the index table, it is written into the algorithm in the form of an array and can be directly queried when needed.

[0177] The present invention proposes an infrared and visible light image fusion method based on a laser ranging module.

[0178] The method solves the problem of mismatch between infrared and visible light lenses with different focal lengths in close-range fusion. The method is simple, has low computational complexity, good effect, and low power consumption. It can be applied to small embedded devices, has low platform requirements, and can run on any embedded platform.

[0179] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, comprising:

[0180] A visible light image acquisition module, used for acquiring visible light images;

[0181] Infrared image acquisition module, used for acquiring infrared images;

[0182] Distance acquisition module, used to obtain the current target distance;

[0183] An image scaling module is used to determine whether the current target distance is greater than a preset value. If so, the target is determined to be within the focal length overlap range of the infrared lens and the visible light lens. At this time, the visible light image is scaled using fixed cropping and scaling parameters to obtain a scaled visible light image. If not, the target is determined to be outside the focal length overlap range shared by the infrared lens and the visible light lens. At this time, the visible light image is scaled using the corresponding cropping and scaling parameters obtained based on the current target distance to obtain a scaled visible light image.

[0184] The image fusion module is used to perform bi-optical image fusion on the scaled visible light image and the infrared image.

[0185] Furthermore, the infrared-visible light image fusion device also includes an affine matrix acquisition module, which is used to obtain and save the affine matrix of the visible light image corresponding to the infrared image at a set distance.

[0186] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the infrared-visible light image fusion method described above are implemented. The infrared-visible light image fusion device also includes a visible light camera, an infrared camera, and a ranging module.

[0187] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is used to run program code or process data stored in the memory, such as the program code for the infrared-visible light image fusion method.

[0188] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or internal memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash memory card, etc. equipped on the electronic device. Of course, the memory may also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory is generally used to store operating methods and various application software installed in the electronic device, such as the program code of the infrared-visible light image fusion method. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0189] Based on the same inventive concept, the present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the infrared-visible light image fusion method described above are implemented.

[0190] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0191] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0192] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0194] The paired visible light image and infrared image in the present invention are the visible light image and infrared image obtained by taking pictures of the same object at adjacent or identical positions using a visible light camera and an infrared camera.

[0195] Example 4

[0196] Referring to Figures 13 to 15, an embodiment of the present invention discloses a calibration plate, including a calibration plate body 51, on which at least two groups of target holes are provided, each group of target holes includes multiple target holes, the target holes in the same group of target holes have the same aperture, and the target holes in different groups of target holes have different apertures.

[0197] Furthermore, each group of target wells corresponds to a region one by one, the area of ​​the region corresponding to each group of target wells is different, and all target wells in each group of target wells must be located in the corresponding region.

[0198] Furthermore, when the area of ​​the region corresponding to the first group of target wells is larger than the area of ​​the region corresponding to the second group of target wells, the target well diameter of the first group of target wells is larger than the target well diameter of the second group of target wells. The regions may overlap, for example, a larger region may completely cover a smaller region.

[0199] Furthermore, the area corresponding to each group of target wells is a circular area or a polygonal area (polygonal areas include triangular areas, quadrilateral areas, pentagonal areas, etc.). That is, the area corresponding to each group of target wells is one of a circular area, a triangular area, a quadrilateral area, a pentagonal area, etc. In this embodiment, the area corresponding to each group of target wells is a rectangular area.

[0200] Furthermore, the centers of the regions corresponding to the multiple groups of target holes are the same.

[0201] Furthermore, the areas corresponding to the multiple groups of target holes are the same as the center of the calibration plate 5 .

[0202] Furthermore, each group of target wells includes at least three target wells, and all target wells in each group of target wells are not all on a straight line.

[0203] Furthermore, all target wells in each group of target wells are distributed in an array.

[0204] Furthermore, after all target wells in each group of target wells are distributed in an array to form an array, the centers of the arrays are the same.

[0205] Furthermore, in each array, the distance between the centers of two adjacent target holes in each row is equal, and the distance between the centers of two adjacent target holes in each column is equal.

[0206] Furthermore, the target hole is a circular hole, and the back hole opening of the target hole is chamfered. After the back hole opening of the target hole is chamfered, it has a circular hole section and a conical hole section connected to each other, the circular hole section is away from the heat source module, and the conical hole section is close to the heat source module (the hole diameter of the conical hole section increases as it approaches the heat source module), and the minimum hole diameter of the conical hole section is equal to the hole diameter of the circular hole section.

[0207] Furthermore, the front surface of the calibration plate body is treated with matte white frosting, and the back surface of the calibration plate body is treated with bright black high-reflective treatment.

[0208] In one embodiment, two groups of target holes are provided on the calibration plate body 51, namely the first group of target holes and the second group of target holes. The first group of target holes includes a plurality of first target holes 52 (large circular holes), and the second group of target holes includes a plurality of second target holes 53 (small circular holes). The aperture of the first target holes 52 is larger than the aperture of the second target holes 53. All the first target holes 52 are located in the first area, and all the second target holes 53 are located in the second area.

[0209] Furthermore, the first area is a first rectangular area, the length of the first rectangular area is 800 mm, and the width is 600 mm; the second area is a second rectangular area, the length of the second rectangular area is 70 mm, and the width is 50 mm.

[0210] Furthermore, the centers of the first rectangular area and the second rectangular area are the same.

[0211] Furthermore, the first rectangular area, the second rectangular area and the calibration plate 5 have the same center.

[0212] Furthermore, the number of the first target holes 52 and the second target holes 53 is at least three, and all the first target holes 52 are not all on a straight line, and all the second target holes 53 are not all on a straight line.

[0213] The diameters of the first target hole 52 and the second target hole 53 are set as needed. In one embodiment, the diameter of the first target hole 52 is 100 mm, and the diameter of the second target hole 53 is 10 mm.

[0214] Furthermore, the plurality of first target holes 52 are distributed in an array to form a first array, and the plurality of second target holes 53 are distributed in an array to form a second array, and the center of the first array is the same as the center of the second array.

[0215] Furthermore, the distance between the centers of two adjacent first target holes 52 in each row of the first array is equal. The distance between the centers of two adjacent first target holes 52 in each column is equal.

[0216] Furthermore, the distance between the centers of two adjacent second target holes 53 in each row of the second array is equal. The distance between the centers of two adjacent second target holes 53 in each column is equal.

[0217] Furthermore, the first target hole 52 and the second target hole 53 are circular holes, and the back openings of the first target hole 52 and the second target hole 53 are both chamfered. The calibration plate 5 of this embodiment has a thickness of 3 mm and is made of aluminum alloy.

[0218] A specific embodiment of a calibration plate 5 is as follows: six first target holes 52 (the first target holes 52 are large circular holes with a diameter of 100 mm) and six second target holes 53 (the second target holes 53 are small circular holes with a diameter of 10 mm) are processed on the calibration plate body 51. The back of each hole is chamfered by 1.5x60°mm. After processing is completed, the front side of the calibration plate body 51, that is, the outward side, is subjected to matte white frosting treatment, and the back side of the calibration plate body 51, that is, the inward side, is subjected to bright black high-reflective treatment.

[0219] In this embodiment, the six first target holes 52 are arranged in a first array of two rows and three columns. The distance between the centers of two adjacent first target holes 52 in each row of the first array is 200 mm. The distance between the centers of two adjacent first target holes 52 in each column of the first array is 200 mm.

[0220] In this embodiment, the six second target holes 53 are arranged in a second array of two rows and three columns. The distance between the centers of two adjacent second target holes 53 in each row of the second array is 20 mm. The distance between the centers of two adjacent first target holes 52 in each column of the second array is 20 mm.

[0221] In this embodiment, the center of the first array, the center of the second array and the center of the calibration plate 5 are the same.

[0222] 13 to 19 , an embodiment of the present invention provides an infrared dual-light fusion testing device, including a heat source module and a calibration plate 5 , wherein the heat source module is used to provide a heat source for the calibration plate 5 .

[0223] The infrared-visible light image registration method described in the first embodiment can be calibrated using the calibration plate 5 of this embodiment.

[0224] Each group of target holes corresponds to a distance range one by one, and the distance between the infrared thermal imaging device and the calibration plate 5 is obtained. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with the set multiple distance ranges to determine which distance range the distance between the infrared thermal imaging device and the calibration plate 5 is in. When the distance between the infrared thermal imaging device and the calibration plate 5 is within a certain distance range, a group of target holes corresponding to the distance range is used for calibration (i.e., infrared and visible light image registration) or testing (during calibration, the visible light image at the calibration distance and the center of the group of target holes of the calibration plate in the infrared image are extracted and saved, and then calibration is performed).

[0225] For example, when two groups of target holes are set on the calibration plate 5, namely the first group of target holes (a group of large circular holes) and the second group of target holes (a group of small circular holes), paired visible light images and infrared images are collected by the infrared thermal imaging device, and the distance between the infrared thermal imaging device and the calibration plate 5 is compared with the set first distance (such as 2 meters). When the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to the first distance (such as 2 meters), the second target hole 53 (small circular hole) is used for calibration (that is, the visible light image at the calibration distance and the center of the second target hole of the calibration plate in the infrared image, i.e., the center of the small circle, are extracted and saved, and then calibration is performed). When the distance between the infrared thermal imaging device and the calibration plate 5 is greater than the first distance (such as 2 meters), the first target hole 52 (large circular hole) is used for calibration (that is, the visible light image at the calibration distance and the center of the first target hole of the calibration plate in the infrared image, i.e., the center of the large circle, are extracted and saved, and then calibration is performed). It is generally believed that when the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to 2 meters, the distance is considered to be close, and when it is greater than 2 meters, the distance is considered to be far.

[0226] Furthermore, the heat source module includes at least one heating unit, and each group of target holes corresponds to one or more heating units.

[0227] Furthermore, at least one group of target holes corresponds to a plurality of heating units.

[0228] Furthermore, at least one group of target holes corresponds to one heating unit.

[0229] Furthermore, when a certain group of target wells corresponds to multiple heating units, each target well in the group of target wells corresponds to one heating unit one by one.

[0230] Referring to Figure 13, two groups of target holes are set on the calibration plate 5 of this embodiment, namely the first group of target holes and the second group of target holes. The first group of target holes includes multiple first target holes 52 (large circular holes), and the second group of target holes includes multiple second target holes 53 (small circular holes).

[0231] Furthermore, each first target hole 52 of the calibration plate 5 corresponds to a heating unit (i.e., a first heating unit), and all second target holes 53 of the calibration plate 5 correspond to a heating unit (i.e., a second heating unit). The sizes of the first heating unit and the second heating unit may vary as needed. In this embodiment, the size of the first heating unit is larger than the size of the second heating unit.

[0232] This embodiment has a total of seven heating units, including six first heating units and one second heating unit. The six first heating units correspond one-to-one to the six first target holes 52 of the calibration plate 5 and are arranged relative to each other. Each first heating unit provides heat to the corresponding first target hole 52 on the calibration plate 5. One second heating unit corresponds to the six second target holes 53 of the calibration plate 5 and provides heat to the six second target holes 53 of the calibration plate 5.

[0233] Furthermore, the heating unit includes a heat source body 21, in which a heating element 22 and a temperature detection element 23 are installed. The heating element 22 is connected to the output end of the controller, and the temperature detection element 23 is connected to the input end of the controller. The controller is connected to the instruction input unit and / or the display unit.

[0234] Each heating unit is controlled individually.

[0235] Furthermore, the heating element 22 may be a heating rod, and the temperature detection element 23 may be a thermocouple.

[0236] The heat source body 21 of the first heating unit of this embodiment uses two heating rods and a thermocouple.

[0237] The heat source body 21 of the second heating unit of this embodiment adopts a heating rod and a thermocouple.

[0238] The heat source body 21 of this embodiment is a rectangular block. A first mounting hole 211 for mounting a heating rod and a second mounting hole 212 for mounting a thermocouple are provided on the heat source body 21. The lead ends of the heating rod and the lead ends of the thermocouple extend out of the heat source body 21.

[0239] The rectangular block of the heat source 21 is made of a material with good thermal conductivity, such as aluminum in this embodiment. The surface of the heat source 21, i.e., the aluminum plate, is treated with a bright black high-reflective treatment to improve imaging quality.

[0240] Furthermore, the heating unit is connected to the heat source fixing plate 3 via a first thermal insulation board 24, which is mounted on the back of the calibration plate 5. In this embodiment, the first thermal insulation board 24 is made of mica fiberboard, which provides thermal insulation. The heat source 21 is secured to the heat source fixing plate 3 using the first thermal insulation board 24 and does not contact the heat source fixing plate 3.

[0241] The heat source fixing plate 3 is provided with a mounting hole for mounting the heating unit. The heat source body 21 of this embodiment is mounted in the mounting hole of the heat source fixing plate 3.

[0242] A second heat insulation plate 4 is provided between the calibration plate 5 and the heat source fixing plate 3. The heat source fixing plate 3, the second heat insulation plate 4 and the calibration plate 5 are connected in sequence. The second heat insulation plate 4 is provided with a through hole for the heat source of the heating unit to pass through.

[0243] The second heat insulation board 4 of this embodiment is made of mica fiber board, which plays a heat insulation role.

[0244] Furthermore, each heating unit is connected to the first heat insulation plate 24 in a one-to-one correspondence, such as by being fixedly connected by bolts, and the first heat insulation plate 24 is connected to the heat source fixing plate 3, such as by being fixedly connected by bolts.

[0245] Furthermore, the second thermal insulation board 4 is provided with a plurality of first through holes for passage of the heat source of the first heating unit. The first through holes of the second thermal insulation board 4 correspond one-to-one with the first target holes 52 of the calibration board 5 and are arranged opposite each other. The apertures of the first through holes of the second thermal insulation board 4 are set as needed so that the heat source of the first heating unit can pass through the corresponding first through holes and reach the corresponding first target holes 52.

[0246] Furthermore, the second thermal insulation board 4 is provided with one or more second through holes for the heat source of the second heating unit to pass through. In this embodiment, the second thermal insulation board 4 is provided with one second through hole for the heat source of the second heating unit to pass through, and the second through hole is arranged opposite to all the second target holes 53 on the calibration board 5.

[0247] The number and size of the second through holes are set as needed, so that the heat source of the second heating unit can reach the second target hole 53 through the second through holes.

[0248] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected by bolts. The heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are respectively provided with bolt holes.

[0249] In this embodiment, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are all rectangular, and have equal lengths and widths.

[0250] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are sequentially connected to form a heating target assembly, which is fixed to a chassis 1 to form a heating target box. In this embodiment, the entire heating target assembly is fixed to the sheet metal chassis 1 by screws.

[0251] As shown in Figure 17, the entire chassis 1 (e.g., a sheet metal chassis) is machined from 2mm steel plate and sprayed with black paint. The heating units are heated using heating rods, with heat transferred to the heat source 21. The chassis 1 is equipped with a sheet metal door assembly 6 with a fan for heat dissipation. The calibration plate 5 serves as a side panel of the chassis 1, and the heat source mounting plate 3 is located within the chassis 1. The chassis 1 is equipped with a human-machine interface 8, a power switch 9, a start switch 10, and an indicator light 7 for displaying the heating status. The human-machine interface 8 and indicator light 7 are connected to a controller. The controller includes a control unit and a temperature control module. The control unit is connected to the temperature detection element 23, the temperature control module, the human-machine interface 8, and the indicator light 7. The temperature control module is connected to the heating element 22, and the temperature control module controls the temperature of each heat source 21. The temperature of each heating unit is controlled via the human-machine interface 8, and each target can be individually controlled as required. The chassis 1 is equipped with two handles for transporting the heating target section. Its working principle is that the heating rod heats and transfers heat to the heat source body 21. The built-in temperature sensor of the thermocouple senses the real-time temperature of the heat source body 21 and displays the actual sensed temperature through the human-machine interface 8. If the actual measurement requires the temperature to be adjusted up or down, the temperature can be set through the control interface of the human-machine interface 8. Each temperature can be controlled separately (adjustment range 25°~120°).

[0252] Furthermore, the heating target box is mounted on the frame 11. The heating target box of this embodiment is fixed on the frame 11 by screws.

[0253] Furthermore, casters 116 are provided at the lower end of the frame 11 .

[0254] The rack 11 includes a body frame 111, a fixing plate 112, a double-opening sheet metal door 113, a stainless steel cover 114, and a rack bottom plate 115. The body frame 111 is welded with stainless steel square tubes on all sides, and the rear and side fixing plates 112 are fixed to the rack 11 with stainless steel sheet metal by screws. The front of the body frame 111 adopts a stainless steel double-opening door, and the top stainless steel cover 114 is fixed to the rack 11 for fixing the heating target box. The bottom plate of the chassis 1 is made of stainless steel sheet metal and a universal wheel with a foot cup is installed at each corner for movement and fixation.

[0255] The calibration plate of the present invention has target holes of various diameters, which can adapt to tests of various distances (such as close distance and long distance, the distance refers to the distance between the infrared thermal imaging device and the calibration plate). It has strong versatility and can meet the needs of optical axis testing of existing products, and can meet the needs of multiple uses of one machine.

[0256] Each target body of the present invention adopts a separate temperature control system, which can accurately control the temperature. The target is highly accurate and more stable, which can ensure the consistency of temperature and improve the accuracy of the test.

[0257] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for fusion of infrared and visible light images, characterized in that: The steps include: Acquire visible light images and infrared images; The image fusion is performed, specifically including: obtaining the current target distance, obtaining the corresponding affine matrix according to the current target distance, and fusing the visible light image with the infrared image using the affine matrix corresponding to the current target distance to obtain a fused image.

2. The infrared-visible light image fusion method according to claim 1, characterized in that: Before the image fusion step, the following step is also included: obtaining and saving the affine matrix of the visible light image corresponding to the infrared image at a set distance.

3. The infrared-visible light image fusion method according to claim 2, characterized in that: The method for obtaining the affine matrix of the visible light image corresponding to the infrared image at a set distance specifically includes: Obtain visible light images and infrared images respectively at a set calibration distance; The centers of the calibration plates in the visible light image and the infrared image at the set calibration distance are respectively extracted and saved, and the affine matrix at the set calibration distance is calculated by aligning the centers of the visible light image and the infrared image at the set calibration distance; The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated according to the affine matrix at the set calibration distance.

4. The infrared-visible light image fusion method according to claim 3, characterized in that: The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated according to the affine matrix at the set calibration distance, specifically including: The rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are averaged, and the X and Y offset components of the affine matrix at each calibration distance are corrected using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances; According to the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are calculated to generate the affine matrix at each set distance.

5. The infrared-visible light image fusion method according to claim 4, characterized in that: Correct the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, including: The nearest neighbor distance method is used to correct the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

6. The infrared-visible light image fusion method according to claim 4, characterized in that: The X-direction offset component and the Y-direction offset component of the affine matrix at each set distance are calculated according to the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance, specifically including: according to the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and the Y-direction offset component of the affine matrix at each set distance are calculated by using an interpolation method.

7. The infrared-visible light image fusion method according to claim 1, characterized in that: The visible light image and the infrared image are fused using an affine matrix corresponding to the current target distance to obtain a fused image, specifically including: performing an affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fusing the transformed visible light image with the acquired infrared image to obtain a fused image.

8. The infrared-visible light image fusion method according to claim 7, characterized in that: The transformed visible light image is fused with the acquired infrared image to obtain a fused image, specifically including: extracting visible light high-frequency information from the transformed visible light image, and superimposing the visible light high-frequency information on the infrared image to obtain a fused image.

9. The infrared-visible light image fusion method according to claim 1, characterized in that: Distortion correction is performed after obtaining visible light images and infrared images.

10. The infrared and visible light image fusion method according to claim 1, characterized in that: After obtaining the current target distance, the following steps are also included: determining whether the current target distance is greater than a preset value; if so, scaling the visible light image or infrared image using fixed cropping and scaling parameters to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion; if not, scaling the visible light image or infrared image using a table lookup to obtain corresponding cropping and scaling parameters according to the current target distance to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion.

11. An infrared-visible light image fusion device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the infrared-visible light image fusion method according to any one of claims 1 to 10 are implemented.

12. An infrared visible light image registration method, characterized in that: The steps include: Obtain visible light images and infrared images respectively at each set calibration distance; The centers of the calibration plates in the visible light image and the infrared image at each calibration distance are extracted and saved respectively, and the affine matrix at each calibration distance is calculated by aligning the centers of the visible light image and the infrared image at each calibration distance; The affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated according to the affine matrix at each calibration distance.

Images