Infrared visible image fusion method, device and registration method

Abstract

The application discloses an infrared and visible light image fusion method, equipment and registration method. The infrared and visible light image fusion method extracts the center of the calibration board in the visible light image and the infrared image under each calibration distance and saves the center, registers each center of the visible light image with each center of the infrared image, calculates the affine matrix under each distance, calculates the average value of all affine matrices, uses the average value of the rotation and scaling components in the X and Y directions of the center and the affine matrix under all calibration distances, and uses the nearest distance method to calculate the X direction offset component and the Y direction offset component under each distance. Finally, the X direction offset component and the Y direction offset component under each distance are obtained in the interpolation mode, the affine matrix of the visible light image and the infrared image under any distance is generated, the visible light high frequency information is superimposed on the infrared image to realize fusion, and the double light image registration and fusion under any distance are realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an infrared and visible light image fusion method, device, and registration method. Background Technology

[0002] Most existing infrared-visible light fusion methods are based on image acquisition using fixed-focus infrared lenses and fixed-focus visible light lenses. For example, after the CPU obtains infrared and visible light images, it executes the corresponding dual-light fusion algorithm to overlay the target details in the visible light image onto the infrared image.

[0003] Because the camera is fixed-focus, both infrared and visible light images are only for target scenes at a specific distance. However, infrared lenses usually cannot keep the same focal length and field of view as visible light lenses. In close-range target scenes, the phenomenon of inconsistent object image size will be particularly obvious. This leads to the current image fusion methods in the industry only having good dual-light fusion effect at a certain distance, and large image fusion errors will occur for close-range scenes.

[0004] Existing image fusion methods include:

[0005] 1. The method based on anisotropic diffusion fusion (ADF) decomposes the original image into a base layer and a detail layer, performs Karhunen-Loeve transformation, and then performs linear superposition to finally obtain a fused image. The algorithm is easy to implement and the loss after fusion is small, but the final visual effect is not ideal.

[0006] 2. Methods based on convolutional neural networks (CNN) and residual neural networks (ResNet) are used to build convolutional neural networks and residual neural networks. Through extensive training, the parameters are optimized and then the images are fused. The performance of the fused images is better, but it requires the construction of a dataset and consumes a lot of time for training, which is time-consuming.

[0007] 3. The gradient transfer fusion (GTF) method first constructs an objective function and solves the objective function iteratively to make the fused image closer to the infrared image and retain more appearance information. Its advantage is that the fused image has a good visual effect, but because it requires iterative solution, the algorithm cannot meet the requirements of real-time processing.

[0008] 4. The method based on latent low-rank representation (LatLRR) first decomposes the original image to obtain the low-rank part and the salient part, then uses two different strategies to fuse them, and finally reconstructs the image, but its visual effect is not very ideal. Summary of the Invention

[0009] The purpose of this invention is to overcome at least one defect in the prior art and to provide an infrared and visible light image fusion method, device and registration method. This invention realizes dual-light image registration and fusion at any distance, can achieve real-time performance and achieve good visual effects.

[0010] The technical solution of this invention is implemented as follows: This invention discloses an infrared and visible light image fusion method, comprising the following steps:

[0011] Acquire visible light and infrared images;

[0012] The process of performing dual-light image fusion includes: obtaining the current target distance, obtaining the corresponding affine matrix based on the current target distance, and fusing the visible light image and the infrared image using the affine matrix corresponding to the current target distance.

[0013] Furthermore, before the image fusion step, the following steps are also included: obtaining and saving the affine matrix of the infrared image corresponding to the visible light image at a set distance.

[0014] Furthermore, the affine matrix of the visible light image corresponding to the infrared image at a set distance is obtained, specifically including:

[0015] Visible light images and infrared images were acquired at various set calibration distances.

[0016] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0017] The affine matrix of the infrared image corresponding to the visible light image at each set distance is calculated based on the affine matrix at each calibrated distance.

[0018] Furthermore, based on the affine matrix at the set calibration distance, the affine matrix of the visible light image corresponding to the infrared image at each set distance is calculated, specifically including:

[0019] The average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances are calculated. Using the center of the calibration plate in the visible light and infrared images at each calibration distance and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are corrected.

[0020] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated, and the affine matrix at each set distance is generated.

[0021] Furthermore, generating affine matrices at various set distances specifically includes: generating affine matrices at various set distances based on the X-direction offset components and Y-direction offset components of the affine matrices at various set distances, as well as the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrices at all calibrated distances.

[0022] Furthermore, the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance are corrected, specifically including:

[0023] The nearest neighbor distance method is used to correct the X-direction offset components and Y-direction offset components of the affine matrix at each calibration distance.

[0024] Furthermore, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated. Specifically, this includes: based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated using interpolation.

[0025] Furthermore, using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated, specifically including:

[0026] Using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance.

[0027] Furthermore, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated. Specifically, this includes: based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at each set distance are calculated using interpolation.

[0028] Further, the current target distance is obtained, and its corresponding affine matrix is ​​obtained based on the current target distance. Specifically, this includes: obtaining the corresponding affine matrix by looking up a table based on the current target distance, or obtaining the X-direction offset component and Y-direction offset component of the corresponding affine matrix by looking up a table based on the current target distance. The average value of the X-direction offset component and Y-direction offset component of the affine matrix at the current target distance, as well as the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances, is assigned to the affine matrix to obtain the affine matrix corresponding to the current target distance.

[0029] Furthermore, the visible light image and the infrared image are fused using the affine matrix corresponding to the current target distance to obtain a fused image. Specifically, this includes: performing an affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fusing the transformed visible light image with the acquired infrared image to obtain a fused image.

[0030] Furthermore, the transformed visible light image is fused with the acquired infrared image to obtain a fused image. Specifically, this includes: extracting visible light high-frequency information from the transformed visible light image and superimposing the visible light high-frequency information onto the infrared image to obtain the fused image.

[0031] Further, the fused image is converted into an RGB fused image.

[0032] Furthermore, after acquiring the visible light and infrared images, distortion correction is first performed, and then the distortion-corrected visible light and infrared images are used in the following steps.

[0033] Furthermore, after obtaining the current target distance, the process also includes the following steps: determining whether the current target distance is greater than a preset value; if so, scaling the visible light image or infrared image using fixed cropping and scaling parameters to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion; if not, obtaining the corresponding cropping and scaling parameters from a table based on the current target distance, scaling the visible light image or infrared image to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for the image fusion step.

[0034] The present invention also discloses an infrared and visible light image fusion device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared and visible light image fusion method described above.

[0035] This invention also discloses an infrared-visible light image registration method, comprising the following steps:

[0036] Visible light images and infrared images were acquired at various set calibration distances.

[0037] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0038] The affine matrix of the infrared image corresponding to the visible light image at each set distance is calculated based on the affine matrix at each calibrated distance.

[0039] The present invention has at least the following beneficial effects:

[0040] This invention proposes a multi-distance visible and infrared dual-beam calibration and fusion method. This method extracts and saves the centers of the calibration plates in the visible and infrared images at various calibration distances. By registering the centers of the visible and infrared images, the affine matrix at each calibration distance is calculated. The average value of all affine matrices is obtained. The average value of the rotation and scaling components in the X and Y directions of the affine matrices at all calibration distances is used, and the nearest neighbor distance method is applied to obtain the X and Y offset components at each distance. Finally, interpolation is used to obtain the X and Y offset components at each distance, realizing the generation of affine matrices for visible and infrared images at arbitrary distances. Finally, high-frequency visible light information is superimposed on the infrared image to achieve fusion. This method realizes dual-beam image registration and fusion at arbitrary distances, achieving real-time performance and good visual effects.

[0041] The method of this invention can automate the calibration process, requiring only image acquisition to complete the automated calibration process and generate data; the fully automated process saves manpower and resources.

[0042] The method of the present invention can achieve the fusion effect of infrared and visible light images from different fields of view, that is, by adding the features of infrared images and superimposing the edge information of visible light, a good visual effect is achieved.

[0043] The method of this invention enables real-time processing of the algorithm on the APP, so that the frame rate of the fused image meets the visual requirements of the human eye, achieving the effect of real-time fusion.

[0044] This invention also proposes an infrared-visible light fusion method based on a laser ranging module. This method dynamically adjusts the cropping and scaling parameters of the visible light image using the target distance obtained by the laser ranging module, thereby solving the problem of inconsistency in dual-light fusion images in close-range scenes. This method is simple, effective, and efficient, making it suitable for application in low-cost product development solutions. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart of an infrared-visible light image registration method provided in an embodiment of the present invention;

[0047] Figure 2 A flowchart of an infrared-visible light image fusion method provided in one embodiment of the present invention;

[0048] Figure 3 A flowchart for distortion correction provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the extraction and sorting of the center points of a high dynamic range infrared (IR) image calibration board.

[0050] Figure 5 A schematic diagram illustrating the extraction and sorting of the center points of the VIS calibration plate for visible light images;

[0051] Figure 6 A flowchart of an infrared-visible light image fusion method provided in another embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the external design of a camera and a ranging module according to an embodiment of the present invention;

[0053] Figure 8 The effect of dual-light fusion is shown when the magnification parameters are fixed at 1.5 meters.

[0054] Figure 9 The effect of dual-light fusion is shown when the magnification is 0.5 meters and fixed magnification parameters are used.

[0055] Figure 10 The image shows the effect of dual-light fusion using updated and appropriate parameters at a distance of 0.5 meters.

[0056] Figure 11 For high dynamic range infrared (IR) imaging;

[0057] Figure 12 To merge images;

[0058] Figure 13 This is a schematic diagram of the calibration plate provided in an embodiment of the present invention;

[0059] Figure 14 A cross-sectional view of the calibration plate provided in an embodiment of the present invention;

[0060] Figure 15 for Figure 14 Enlarged view of part P;

[0061] Figure 16 This is a schematic diagram of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0062] Figure 17 An exploded view of the heating target box of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0063] Figure 18 This is a schematic diagram of the heating unit of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0064] Figure 19 An exploded view of the frame of the infrared dual-light fusion testing device provided in an embodiment of the present invention.

[0065] In the attached diagram, 1 is the chassis, 2 is the heating unit, 21 is the heat source, 211 is the first mounting hole, 212 is the second mounting hole, 22 is the heating element, 23 is the temperature detection element, 24 is the first heat insulation plate, 3 is the heat source fixing plate, 4 is the second heat insulation plate, 5 is the calibration plate, 51 is the calibration plate body, 52 is the first target hole, 53 is the second target hole, 6 is the sheet metal door assembly, 7 is the indicator light, 8 is the human-machine interface, 9 is the power switch, 10 is the start switch, 11 is the frame, 111 is the machine frame, 112 is the fixing plate, 113 is the double sheet metal door, 114 is the stainless steel cover plate, 115 is the frame base plate, and 116 is the caster. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] See Figure 1This invention provides an infrared-visible light image registration method, comprising the following steps:

[0069] Paired visible light and infrared images were acquired at various set calibration distances.

[0070] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0071] The affine matrix of the infrared image corresponding to the visible light image at any distance is calculated based on the affine matrix at each calibration distance.

[0072] Furthermore, based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated, specifically including:

[0073] The average value of the rotation and scaling components in the X and Y directions of the affine matrix under all calibration distances is obtained. The affine matrix includes the X-direction scaling component a11, the X-direction rotation component a12, the Y-direction rotation component a21, the Y-direction scaling component a22, and the X-direction offset component t1 and the Y-direction offset component t2.

[0074] Using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are recalculated (if they are not recalculated, the regular increase or decrease of the offset components cannot be achieved), and the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are obtained.

[0075] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, calculate the X-direction offset components and Y-direction offset components of the affine matrix at any distance.

[0076] An affine matrix at any distance can be generated by averaging the X and Y offset components of the affine matrix at each set distance, along with the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances. This can be achieved by assigning the X and Y offset components of the affine matrix at each set distance to the affine average matrix, or by assigning the average of the X and Y offset components of the affine matrix at each set distance, along with the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances, to the affine matrix.

[0077] The affine matrix is ​​obtained by averaging the affine matrices for all calibrated distances.

[0078] Furthermore, by using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances (i.e., the average value of the rotation and scaling matrix in the affine matrix at each calibration distance), the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are calculated in reverse.

[0079] Furthermore, using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

[0080] The principle of the nearest neighbor distance method of this invention is as follows: Under a single distance, the coordinates of each circle center in the visible calibration image are cyclically input, and the offset vectors in the X direction and Y direction corresponding to the circle center coordinates in the corresponding infrared calibration image can be calculated. For example, the first circle center after sorting in the visible calibration image corresponds to the first circle center after sorting in the infrared calibration image. The corresponding offset vectors in the X direction and Y direction are calculated and denoted as an and bn. Then, by traversing all the circle centers corresponding to the infrared calibration image circles under this distance, the circle centers in the visible calibration image are transformed into the circle centers in the infrared calibration image, and the straight-line distance dis between the circle centers extracted from the infrared calibration image and the circle centers is calculated. For example, if 25 circle centers are extracted from both the infrared calibration image and the visible calibration image, the loop is repeated 25 times, and the minimum straight-line distance value is accumulated and a set of an and bn is denoted as the offset vectors in the X direction and Y direction under the current distance.

[0081] Using the centers of the calibration plate in the visible and infrared images at various calibration distances, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X and Y offset components of the affine matrix at each calibration distance. Specifically, for a single calibration distance, the centers of the calibration plate in the visible and infrared images at that calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, are obtained. For each pair of corresponding visible and infrared calibration image centers, the X and Y offset components, as well as the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, are first obtained to form a new affine matrix (a pair of corresponding visible and infrared calibration image centers). Each new affine matrix corresponds to a new circle center. Using each new affine matrix, the coordinates of the circle center in the visible calibration image transformed to the center in the infrared calibration image are calculated and designated as the second circle center in the infrared calibration image. The offset vectors in the X and Y directions, as well as the straight-line distance dis, between the circle center and the first circle center coordinates in the corresponding infrared calibration image (i.e., the extracted circle center in the infrared calibration image) are calculated. The straight-line distances dis corresponding to all circle centers are summed to obtain the cumulative straight-line distance value corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X and Y directions, denoted as an and bn. The set of offset vectors in the X and Y directions corresponding to the minimum value of the cumulative straight-line distance is taken as the offset vectors in the X and Y directions under the current calibration distance.

[0082] Furthermore, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated. Specifically, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated using interpolation.

[0083] Based on the same inventive concept, the present invention also provides an infrared-visible light image registration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared-visible light image registration method described above. The infrared-visible light image registration device also includes a visible light camera, an infrared camera, and a ranging module.

[0084] Example 2

[0085] See Figure 1 and Figure 2This invention provides an infrared-visible light image fusion method, comprising the following steps:

[0086] Obtain and save the affine matrix of the infrared image corresponding to the visible light image at a set distance (arbitrary distance or various distances);

[0087] Acquire paired visible light and infrared images;

[0088] The process of performing dual-light image fusion includes: obtaining the current target distance, obtaining the corresponding affine matrix based on the current target distance, and fusing the visible light image and the infrared image using the affine matrix corresponding to the current target distance to obtain a fused image.

[0089] Furthermore, the affine matrix of the infrared image corresponding to the visible light image at various distances can be obtained through the following steps:

[0090] Paired visible light and infrared images were acquired at various set calibration distances.

[0091] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0092] The affine matrix of the infrared image corresponding to the visible light image at each set distance is calculated based on the affine matrix at each calibrated distance.

[0093] Furthermore, for a specific method to calculate the affine matrix of the visible light image corresponding to the infrared image at any distance based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, please refer to Example 1.

[0094] Further, the current target distance is obtained, and its corresponding affine matrix is ​​obtained based on the current target distance. Specifically, this includes: obtaining the corresponding affine matrix by looking up a table based on the current target distance, or obtaining the X-direction offset component and Y-direction offset component of the corresponding affine matrix by looking up a table based on the current target distance. The affine matrix at the current target distance is generated by substituting the X-direction offset component and Y-direction offset component of the affine matrix at the current target distance, as well as the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances, into the affine matrix.

[0095] Furthermore, the visible light image and the infrared image are fused using the affine matrix corresponding to the current target distance to obtain a fused image. Specifically, this includes: performing an affine transformation on the visible light image based on the affine matrix corresponding to the current target distance, and then fusing the infrared image with the transformed visible light image to obtain a fused image.

[0096] Furthermore, the infrared image is fused with the transformed visible light image to obtain a fused image. Specifically, this includes: extracting visible light high-frequency information from the transformed visible light image and superimposing the visible light high-frequency information onto the infrared image to obtain the fused image.

[0097] Further, the fused image is converted into an RGB fused image.

[0098] Furthermore, after acquiring the visible light image and the infrared image, distortion correction is first performed, and then the distortion-corrected visible light image and infrared image are used in the calibration step and / or the image fusion step.

[0099] Furthermore, after obtaining the current target distance, the following steps are also included: determining whether the current target distance is greater than a preset value. If so, it is determined that the target is within the focal length overlap range of the infrared lens and the visible light lens. At this time, the visible light image is scaled using fixed cropping and scaling parameters to obtain a scaled visible light image, which is then used in the image fusion step. If not, it is determined that the target is outside the focal length overlap area shared by the infrared lens and the visible light lens. At this time, the corresponding cropping and scaling parameters are obtained by looking up the table based on the current target distance to scale the visible light image, which is then used in the image fusion step.

[0100] The process described above will be explained in detail below with specific examples.

[0101] An infrared-visible light image fusion method includes the following steps:

[0102] Calibration process: Obtain the affine matrix of the infrared image corresponding to the visible light image at each distance;

[0103] Image fusion process: Obtain the target distance through the Hall contrast table, look up the corresponding affine matrix at the distance in the table, perform real-time calculation, and display the fused image.

[0104] The calibration process specifically includes:

[0105] Acquire 14-bit / 16-bit high dynamic range infrared (IR) images and 8-bit visible light (VIS) images.

[0106] Identify the centers of the calibration plates on the 14-bit / 16-bit high dynamic range infrared (IR) image and the 8-bit visible light (VIS) image, and store them as ir_centerCiclePoints and vis_centerCiclePoints respectively, then sort them. See [link to documentation]. Figure 4 and Figure 5 .

[0107] Registration is performed based on vis_centerCiclePoints and ir_centerCiclePoints to obtain the affine matrix VISToIRMatrixs.

[0108] AffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints,VISToIRMatrixs);

[0109] The above process is used to perform calibration at a set calibration distance. In this embodiment, there are 9 calibration distances, ranging from 1 meter to 9 meters. Of course, the number of calibration distances is not limited to 9 and can be adjusted as needed. All extracted center points vis_centerCiclePointList and ir_centerCiclePointList are saved, and the center points of the visible light image and the center points of the infrared image are matched one by one to generate affine matrices VISToIRMatrixs for each distance.

[0110] The rotation and scaling components of the affine matrix under the 9 sets of data are averaged (i.e., a11, a12, a21, a22), and stored as VISToIRMatrix_average, which is represented as:

[0111] .

[0112] In the affine matrix, a11 represents the scaling ratio in the X direction, a12 represents the rotation ratio in the X direction, a21 represents the rotation ratio in the Y direction, a22 represents the scaling ratio in the Y direction, t1 represents the translation distance in the X direction, and t2 represents the translation distance in the Y direction. In a two-dimensional image, the X-direction vector and the Y-direction vector have different properties in the affine transformation.

[0113] For example, in this embodiment, at a distance of 1 meter, a11 = 2.71771, a12 = 0.0005355351, t1 = -1027.16, a21 = -0.00148628, a22 = 2.77923, t2 = -1125.53.

[0114] The study found that the X-direction offset changes more significantly at close range, while the change is almost negligible at greater distances.

[0115] Using the saved vis_centerCiclePointList and ir_centerCiclePointList, and the average affine matrix VISToIRMatrix_average, the nearest neighbor offset distance method is used to calculate the X-direction component t1 and Y-direction component t2 at each distance.

[0116] ReverseCalAffineTransTable(vis_centerCiclePoints, ir_centerCiclePoints);

[0117] Interpolate the t1 and t2 calculated for each group, and calculate the t1 and t2 values ​​for each distance from 1 to 20 meters in units of 10 centimeters. The x and y values ​​range from 1 to 9 meters, the range is 1 to 20 meters, and the interval is 0.1 meters. The output is a lookup table of the X-direction offset vector and the Y-direction offset vector.

[0118] Interp1HermitePchip(distance, x, range, t1);

[0119] Interp1HermitePchip(distance, y, range, t2);

[0120] Finally, the target distance is obtained from the Hall contrast table, and the t1 and t2 values ​​at the corresponding distance are obtained by looking up the table based on the target distance. These values ​​are then assigned to VISToIRMatrix_average to calculate the fused image at the current distance.

[0121] The image fusion process specifically includes:

[0122] Acquire 14-bit / 16-bit high dynamic range infrared (IR) images and 8-bit visible light (VIS) images. See one embodiment of the high dynamic range infrared (IR) image. Figure 11 As shown.

[0123] Convert 14-bit / 16-bit high dynamic range infrared (IR) images and 8-bit visible light (VIS) images to YUV420 format IR_yuv and VIS_yuv.

[0124] bgrtoyuv420(IR_yuv, IR.data, n_ir_width, n_ir_height);

[0125] Obtain the current distance from the Hall value contrast table, and then obtain the affine matrix at that distance based on the calibration storage file and the distance itself.

[0126] VISToIRMatrixs = SetAffineMat(distance);

[0127] The Y component in VIS_yuv is subjected to affine transformation based on the affine matrix obtained in the previous step, and then the high-frequency information is extracted and superimposed on the Y component in IR_yuv to obtain FusionImgYUV.

[0128] FusionImgYUV = ImgRegAndFusYUV(VISToIRMatrixs);

[0129] The IR_yuv image with overlaid information is converted to an RGB image and displayed. See the fused image for details. Figure 12 As shown.

[0130] yuv420tobgr(Imgfuse.data, FusionImgYUV, n_ir_width, n_ir_height).

[0131] After acquiring visible light and infrared images, distortion correction is first performed. Then, the distortion-corrected visible light and infrared images are used in the calibration and image fusion processes.

[0132] One important application of image geometric transformation is eliminating geometric distortion in digital images caused by cameras. Restoring geometrically distorted images to their normal state using a computer image processing system is called geometric correction of image distortion. This involves establishing a mathematical model for geometric correction, determining the model parameters using known conditions, and finally applying the model to the image for geometric correction. (See also...) Figure 3 The specific steps for distortion correction are as follows:

[0133] 1. Collect approximately fifteen images of the calibration board;

[0134] 2. Determine the number of interior points on the calibration plate based on the calibration plate used, and find the coordinates of the center of the calibration plate;

[0135] 3. Further extract sub-pixel corner information;

[0136] 4. Calculate and obtain the camera's intrinsic parameter matrix and distortion coefficients using the OpenCV function calibrateCamera;

[0137] 5. Correction and restoration of distorted images: Obtain the distorted image using the OpenCV function `undistort`.

[0138] Both the calibration process and the image fusion process use 14-bit / 16-bit high dynamic range infrared (IR) images and 8-bit visible light (VIS) images after distortion correction.

[0139] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared-visible light image fusion method described above. The infrared-visible light image fusion device further includes a visible light camera, an infrared camera, and a ranging module.

[0140] Based on the same inventive concept, the present invention also provides an infrared and visible light image fusion device, comprising:

[0141] The affine matrix acquisition module is used to acquire and save the affine matrix of the infrared image corresponding to the visible light image at a set distance;

[0142] Visible light image acquisition module, used to acquire visible light images;

[0143] Infrared image acquisition module, used to acquire infrared images;

[0144] The distance acquisition module is used to obtain the current target distance;

[0145] The image fusion module is used to obtain the corresponding affine matrix based on the current target distance, perform an affine transformation on the visible light image based on the affine matrix corresponding to the current target distance, and fuse the infrared image with the transformed visible light image to obtain a fused image.

[0146] The visible light and infrared fields of view differ. This invention achieves dual-light image registration and fusion at any distance, enabling real-time processing while delivering good visual effects. The method automates the calibration process, requiring only image acquisition to generate data, saving manpower and resources. It also achieves fusion of infrared and visible light images with different fields of view by adding infrared image features and superimposing visible light edge information (i.e., extracting the Y channel from the visible light YUV image, registering it, then subtracting the mean-filtered Y channel from the original Y channel to extract edge information, which is then superimposed on the infrared YUV Y channel to obtain a fused YUV image, which is then converted back to RGB for display), resulting in good visual effects. This method can effectively achieve real-time processing on Android devices with a good visual experience.

[0147] The method of this invention enables real-time processing of the algorithm on the APP, so that the frame rate of the fused image meets the visual requirements of the human eye, achieving the effect of real-time fusion.

[0148] Example 3

[0149] See Figure 6 This invention provides an infrared-visible light image fusion method, comprising the following steps:

[0150] Acquire paired visible light and infrared images;

[0151] Obtain the current target distance and determine if it is greater than a preset value (e.g., 1.2 meters, which can be set as needed). If it is, determine that the target is within the focal length overlap range of the infrared lens and the visible light lens. At this time, use fixed cropping and scaling parameters to scale the visible light image or infrared image to obtain a scaled visible light image or infrared image. If not, determine that the target is outside the focal length overlap area shared by the infrared lens and the visible light lens. At this time, obtain the corresponding cropping and scaling parameters based on the current target distance to scale the visible light image or infrared image to obtain a scaled visible light image or infrared image.

[0152] Perform two-light image fusion by combining a scaled visible light image with an unscaled infrared image, or perform two-light image fusion by combining a scaled infrared image with an unscaled visible light image.

[0153] Furthermore, the scaled visible light image and the unscaled infrared image are fused together, specifically including: obtaining the affine matrix corresponding to the current target distance, performing an affine transformation on the scaled visible light image based on the affine matrix corresponding to the current target distance, and fusing the infrared image with the transformed visible light image to obtain a fused image.

[0154] Furthermore, the scaled infrared image and the unscaled visible light image are fused together. Specifically, this includes: obtaining the affine matrix corresponding to the current target distance; performing an affine transformation on the unscaled visible light image based on the affine matrix corresponding to the current target distance; and fusing the infrared image with the transformed visible light image to obtain a fused image.

[0155] The affine matrix corresponding to the current target distance is obtained by: looking up the corresponding affine matrix in a table based on the current target distance, or looking up the X-direction offset component and Y-direction offset component of the corresponding affine matrix in a table based on the current target distance, and then assigning the X-direction offset component and Y-direction offset component of the affine matrix at the current target distance to the stored affine average matrix (the affine average matrix is ​​obtained by averaging the affine matrices at all calibrated distances), thus obtaining the affine matrix at the current target distance.

[0156] Furthermore, the affine matrix of the infrared image corresponding to the visible light image at various distances can be obtained through the following steps:

[0157] Paired visible light and infrared images were acquired at various set calibration distances.

[0158] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0159] The affine matrix of the infrared image corresponding to the visible light image at any distance is calculated based on the affine matrix at each calibration distance.

[0160] Furthermore, based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated, specifically including:

[0161] The average value of the rotation and scaling components in the X and Y directions of the affine matrix under all calibration distances is calculated. The affine matrix includes the scaling component a11 in the X direction, the rotation component a12 in the X direction, the rotation component a21 in the Y direction, the scaling component a22 in the Y direction, and the offset component t1 in the X direction and the offset component t2 in the Y direction.

[0162] Using the center of the calibration plate in the visible light and infrared images at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are calculated to obtain the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

[0163] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, calculate the X-direction offset components and Y-direction offset components of the affine matrix at any distance.

[0164] By assigning the X-direction offset component and Y-direction offset component of the affine matrix at any distance, as well as the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances, to the affine matrix, an affine matrix at any distance can be generated. That is, the rotation and scaling components in the X and Y directions of the affine matrix at any distance are the same as the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances.

[0165] Furthermore, using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated, specifically including:

[0166] Using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance.

[0167] Furthermore, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated. Specifically, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated using interpolation.

[0168] Furthermore, when the current target distance is less than a preset value, the visible light image or infrared image is scaled according to the corresponding cropping and scaling parameters obtained by looking up the table based on the current target distance, and the scaled visible light image or infrared image is used in the image fusion step.

[0169] The method of this invention dynamically adjusts the cropping and scaling parameters of visible light or infrared images by utilizing the target distance obtained by a laser ranging module, thereby solving the problem of inconsistency in dual-light fusion images in close-range scenes. This method is simple, effective, and efficient, making it suitable for application in low-cost product development solutions.

[0170] This invention utilizes the STMicroelectronics VL53L1X short-range, low-power laser ranging module to measure target distance in real time. The VL53L1X is ST's third-generation FlightSense time-of-flight sensor, employing ST's latest ToF technology, achieving a ranging range of up to 4 meters and a ranging frequency of up to 50Hz. The VL53L1X integrates a physical infrared filter and optical components, enabling distance measurement regardless of target color and reflectivity, and offering enhanced anti-interference capabilities.

[0171] The external design of the camera and ranging module is as follows: Figure 7 As shown, the laser ranging module 3 is installed in the middle of the infrared camera 2 and the visible light camera 1, which helps to ensure accurate ranging. In the medium-range measurement mode, the VL53L1X has a ranging length of 2.9 meters, and in the short-range measurement mode, the measurement accuracy of 1.3 meters can reach ±25mm, which is sufficient to meet the design accuracy requirements of this solution.

[0172] The ranging module updates the target distance in real time. If the target distance is greater than a preset value, such as 1.2 meters, it is determined that the target is within the focal length overlap range of the infrared and visible light lenses. In this case, fixed magnification parameters are used for dual-light fusion, and the effect is as follows. Figure 8As shown; if the target distance is less than a preset value, such as 1.2 meters, it is determined that the target is outside the focal overlap area shared by the two cameras. In this case, the index table of distance and magnification is consulted to select an appropriate magnification for dual-light image fusion, and the effect is as follows. Figure 10 As shown. Figure 9 The image shows the effect of dual-light fusion using fixed magnification parameters (traditional method) at a distance of 0.5 meters, compared to... Figure 9 and Figure 10 It can be seen that the method of the present invention achieves significantly better results in dual-light image fusion than the traditional method.

[0173] To obtain suitable clipping scaling parameters within 1.2 meters, it is necessary to conduct experiments beforehand to obtain appropriate values.

[0174] The experimental method involved manually setting appropriate clipping scaling at different distance points and recording the results until the relevant parameters for each distance were obtained. The results obtained in the actual experiment, namely the index table of near-distance scaling parameters, are shown in Table 1.

[0175] Table 1

[0176]

[0177] In Table 1, w represents the width of the cropped area image, x represents the horizontal offset coordinate of the cropped image relative to the original image, h represents the height of the cropped area image, and y represents the vertical offset coordinate of the cropped image relative to the original image. After obtaining this index table, it is written into the algorithm as an array, and can be directly queried when needed.

[0178] This invention proposes an infrared-visible light image fusion method based on a laser ranging module.

[0179] This method solves the problem of mismatch in close-range fusion between infrared and visible light lenses with different focal lengths. It is simple, computationally inefficient, effective, and consumes little power. It can be applied to small embedded devices with low platform requirements and can run on any embedded platform.

[0180] Based on the same inventive concept, the present invention also provides an infrared and visible light image fusion device, comprising:

[0181] Visible light image acquisition module, used to acquire visible light images;

[0182] Infrared image acquisition module, used to acquire infrared images;

[0183] The distance acquisition module is used to obtain the current target distance;

[0184] The image scaling module is used to determine whether the current target distance is greater than a preset value. If it is, it is determined that the target is within the focal length overlap range of the infrared lens and the visible light lens. At this time, the visible light image is scaled using fixed cropping and scaling parameters to obtain a scaled visible light image. If not, it is determined that the target is outside the focal length overlap area shared by the infrared lens and the visible light lens. At this time, the visible light image is scaled using the corresponding cropping and scaling parameters obtained according to the current target distance to obtain a scaled visible light image.

[0185] The image fusion module is used to perform dual-light image fusion between the scaled visible light image and the infrared image.

[0186] Furthermore, the infrared-visible image fusion device also includes an affine matrix acquisition module, which is used to acquire and save the affine matrix of the infrared image corresponding to the visible light image at a set distance.

[0187] Based on the same inventive concept, the present invention also provides an infrared-visible light image fusion device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared-visible light image fusion method described above. The infrared-visible light image fusion device further includes a visible light camera, an infrared camera, and a ranging module.

[0188] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is used to run program code stored in the memory or process data, such as program code for an infrared-visible light image fusion method.

[0189] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory is typically used to store operating methods and various application software installed on the electronic device, such as program code for infrared and visible light image fusion methods. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0190] Based on the same inventive concept, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the infrared-visible light image fusion method described above.

[0191] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0192] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] In this invention, the paired visible light image and infrared image are obtained by taking pictures of the same object from a nearby or same position using a visible light camera and an infrared camera.

[0196] Example 4

[0197] See Figures 13 to 15 This invention discloses a calibration plate, including a calibration plate body 51. The calibration plate body 51 is provided with at least two sets of target holes. Each set of target holes includes multiple target holes. The target holes in the same set of target holes have the same diameter, while the target holes in different sets of target holes have different diameters.

[0198] Furthermore, each group of target holes corresponds to a region, and the area of ​​the region corresponding to each group of target holes is different. All target holes in each group of target holes must be located within the corresponding region.

[0199] Furthermore, when the area corresponding to the first set of target holes is larger than the area corresponding to the second set of target holes, the aperture of the first set of target holes is larger than the aperture of the second set of target holes. Areas can overlap; for example, a large area can completely cover a small area.

[0200] Furthermore, the area corresponding to each group of target holes is either a circular area or a polygonal area (polygonal areas include triangular areas, quadrilateral areas, pentagonal areas, etc.). That is, the area corresponding to each group of target holes is one of a circular area, a triangular area, a quadrilateral area, a pentagonal area, etc. In this embodiment, the area corresponding to each group of target holes is a rectangular area.

[0201] Furthermore, the centers of the regions corresponding to multiple sets of target holes are the same.

[0202] Furthermore, the areas corresponding to the multiple sets of target holes are the same as the center of calibration plate 5.

[0203] Furthermore, each group of target holes contains at least three target holes, and not all target holes in each group are on a straight line.

[0204] Furthermore, all target holes in each group are arranged in an array.

[0205] Furthermore, after all the target holes in each group are arranged in an array to form an array, the centers of each array are the same.

[0206] Furthermore, the spacing between the centers of two adjacent target holes in each row of each array is equal, and the spacing between the centers of two adjacent target holes in each column is equal.

[0207] Furthermore, the target hole is a circular hole, and the back opening of the target hole is chamfered. After the back opening of the target hole is chamfered, it has a connected circular hole segment and a conical hole segment. The circular hole segment is away from the heat source module, and the conical hole segment is close to the heat source module (the diameter of the conical hole segment gradually increases as it approaches the heat source module). The minimum diameter of the conical hole segment is equal to the diameter of the circular hole segment.

[0208] Furthermore, the front of the calibration plate body is treated with a matte white frosted finish, and the back of the calibration plate body is treated with a glossy black high-reflection finish.

[0209] In one embodiment, the calibration plate body 51 is provided with two sets of target holes, namely a first set of target holes and a second set of target holes. The first set of target holes includes a plurality of first target holes 52 (large round holes), and the second set of target holes includes a plurality of second target holes 53 (small round holes). The diameter of the first target holes 52 is larger than the diameter of the second target holes 53. All the first target holes 52 are located in a first region, and all the second target holes 53 are located in a second region.

[0210] Furthermore, the first region is a first rectangular region with a length of 800mm and a width of 600mm; the second region is a second rectangular region with a length of 70mm and a width of 50mm.

[0211] Furthermore, the centers of the first rectangular region and the second rectangular region are the same.

[0212] Furthermore, the first rectangular region and the second rectangular region are centered on the calibration plate 5.

[0213] Furthermore, the number of the first target hole 52 and the second target hole 53 is at least three, and not all of the first target holes 52 are on a straight line, nor are all of the second target holes 53 on a straight line.

[0214] The diameters of the first target hole 52 and the second target hole 53 are set as needed. In one embodiment, the diameter of the first target hole 52 is 100mm and the diameter of the second target hole 53 is 10mm.

[0215] Furthermore, multiple first target holes 52 are arranged in an array to form a first array, and multiple second target holes 53 are arranged in an array to form a second array, with the center of the first array being the same as the center of the second array.

[0216] Furthermore, in the first array, the distance between the centers of two adjacent first target holes 52 in each row is equal. The distance between the centers of two adjacent first target holes 52 in each column is also equal.

[0217] Furthermore, in the second array, the distance between the centers of two adjacent second target holes 53 in each row is equal. The distance between the centers of two adjacent second target holes 53 in each column is also equal.

[0218] Furthermore, the first target hole 52 and the second target hole 53 are round holes, and the back openings of both the first target hole 52 and the second target hole 53 are chamfered. In this embodiment, the calibration plate 5 is 3mm thick and made of aluminum alloy.

[0219] A specific embodiment of a calibration plate 5 is as follows: the calibration plate body 51 is machined with six first target holes 52 (the first target holes 52 are large round holes with a diameter of 100mm) and six second target holes 53 (the second target holes 53 are small round holes with a diameter of 10mm). The back opening of each hole has a chamfer of 1.5 x 60° mm. After the machining is completed, the front side of the calibration plate body 51, i.e. the outward side, is treated with a matte white frosted finish, and the back side of the calibration plate body 51, i.e. the inward side, is treated with a glossy black high-reflection finish.

[0220] In this embodiment, the six first target holes 52 are arranged in two rows and three columns in a first array. The distance between the centers of two adjacent first target holes 52 in each row of the first array is 200 mm. The distance between the centers of two adjacent first target holes 52 in each column of the first array is also 200 mm.

[0221] In this embodiment, the six second target holes 53 are arranged in a second array of two rows and three columns. The distance between the centers of two adjacent second target holes 53 in each row of the second array is 20 mm. The distance between the centers of two adjacent first target holes 52 in each column of the second array is also 20 mm.

[0222] In this embodiment, the center of the first array and the center of the second array are the same as the center of the calibration board 5.

[0223] See Figures 13 to 19 This invention provides an infrared dual-light fusion testing device, including a heat source module and a calibration plate 5, wherein the heat source module is used to provide a heat source for the calibration plate 5.

[0224] The infrared-visible light image registration method described in Example 1 can be calibrated using the calibration plate 5 of this example.

[0225] Each set of target holes corresponds to a distance range, obtaining the distance between the infrared thermal imaging device and the calibration plate 5. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with multiple set distance ranges to determine which distance range the distance between the infrared thermal imaging device and the calibration plate 5 falls within. When the distance between the infrared thermal imaging device and the calibration plate 5 falls within a certain distance range, the set of target holes corresponding to that distance range is used for calibration (i.e., infrared-visible light image registration) or testing (during calibration, the center of the target hole circle of the calibration plate in the visible light image and infrared image at the calibration distance is extracted and saved, and then calibration is performed).

[0226] When two sets of target holes are set on the calibration plate 5, namely the first set of target holes (a set of large circular holes) and the second set of target holes (a set of small circular holes), the infrared thermal imaging device acquires paired visible light images and infrared images, and compares the distance between the infrared thermal imaging device and the calibration plate 5 with the set first distance (e.g., 2 meters). When the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to the first distance (e.g., 2 meters), the second target hole 53 (small circular hole) is used for calibration (i.e., extracting and saving the center of the second target hole of the calibration plate, i.e., the center of the small circle, in the visible light image and the infrared image at the calibration distance, and then performing calibration). When the distance between the infrared thermal imaging device and the calibration plate 5 is greater than the first distance (e.g., 2 meters), the first target hole 52 (large circular hole) is used for calibration (i.e., extracting and saving the center of the first target hole of the calibration plate, i.e., the center of the large circle, in the visible light image and the infrared image at the calibration distance, and then performing calibration). Generally, when the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to 2 meters, it is considered to be relatively close; when it is greater than 2 meters, it is considered to be relatively far.

[0227] Furthermore, the heat source module includes at least one heating unit, and each set of target holes corresponds to one or more heating units.

[0228] Furthermore, at least one set of target holes corresponds to multiple heating units.

[0229] Furthermore, at least one set of target holes corresponds to one heating unit.

[0230] Furthermore, when a set of target holes corresponds to multiple heating units, each target hole in that set of target holes corresponds to one heating unit.

[0231] See Figure 13 In this embodiment, the calibration plate 5 is provided with two sets of target holes, namely the first set of target holes and the second set of target holes. The first set of target holes includes multiple first target holes 52 (large round holes), and the second set of target holes includes multiple second target holes 53 (small round holes).

[0232] Furthermore, each first target hole 52 of the calibration plate 5 corresponds to one heating unit (i.e., the first heating unit), and all second target holes 53 of the calibration plate 5 correspond to one heating unit (i.e., the second heating unit). The sizes of the first heating unit and the second heating unit may differ as needed. In this embodiment, the size of the first heating unit is larger than the size of the second heating unit.

[0233] This embodiment has a total of seven heating units, including six first heating units and one second heating unit. The six first heating units correspond one-to-one with the six first target holes 52 on the calibration plate 5 and are arranged opposite each other. Each first heating unit provides a heat source to the corresponding first target hole 52 on the calibration plate 5. The second heating unit corresponds to the six second target holes 53 on the calibration plate 5 and provides a heat source to the six second target holes 53 on the calibration plate 5 through the second heating unit.

[0234] Furthermore, the heating unit includes a heat source 21, in which a heating element 22 and a temperature detection element 23 are installed. The heating element 22 is connected to the output terminal of the controller, and the temperature detection element 23 is connected to the input terminal of the controller. The controller is connected to an instruction input unit and / or a display unit.

[0235] Each heating unit is controlled independently.

[0236] Furthermore, the heating element 22 can be a heating rod, and the temperature sensing element 23 can be a thermocouple.

[0237] In this embodiment, the heat source 21 of the first heating unit uses two heating rods and one thermocouple.

[0238] In this embodiment, the heat source 21 of the second heating unit uses a heating rod and a thermocouple.

[0239] In this embodiment, the heat source 21 is a rectangular block. The heat source 21 is provided with a first mounting hole 211 for mounting a heating rod and a second mounting hole 212 for mounting a thermocouple. The lead-out ends of the heating rod and the thermocouple extend out of the heat source 21.

[0240] The rectangular block of the heat source 21 is made of a material with good thermal conductivity, such as aluminum in this embodiment. The surface of the heat source 21, i.e., the aluminum plate, is treated with a glossy black high-reflectivity finish to improve image quality.

[0241] Furthermore, the heating unit is connected to the heat source fixing plate 3 via a first heat insulation plate 24, and the heat source fixing plate 3 is installed on the back of the calibration plate 5. In this embodiment, the first heat insulation plate 24 is made of mica fiber board, which serves as heat insulation. The heat source body 21 is fixed to the heat source fixing plate 3 by the first heat insulation plate 24 and does not come into contact with the heat source fixing plate 3.

[0242] The heat source fixing plate 3 is provided with mounting holes for installing the heating unit. In this embodiment, the heat source body 21 is installed in the mounting holes of the heat source fixing plate 3.

[0243] A second heat insulation plate 4 is provided between the calibration plate 5 and the heat source fixing plate 3. The heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence. The second heat insulation plate 4 is provided with a through hole for the heat source of the heating unit to pass through.

[0244] In this embodiment, the second heat insulation board 4 is made of mica fiber board, which serves as a heat insulation agent.

[0245] Furthermore, each heating unit is connected to the first heat insulation plate 24 in a corresponding manner, such as by bolts. The first heat insulation plate 24 is also connected to the heat source fixing plate 3, such as by bolts.

[0246] Furthermore, the second heat insulation plate 4 is provided with a plurality of first through holes for the heat source of the first heating unit to pass through. The first through holes on the second heat insulation plate 4 correspond one-to-one with the first target holes 52 on the calibration plate 5 and are arranged opposite each other. The diameter of the first through holes on the second heat insulation plate 4 is set as needed so that the heat source of the first heating unit can reach the corresponding first target hole 52 through the corresponding first through hole.

[0247] Furthermore, the second heat insulation plate 4 is provided with one or more second through holes for the heat source of the second heating unit to pass through. In this embodiment, the second heat insulation plate 4 is provided with a second through hole for the heat source of the second heating unit to pass through, and this second through hole is arranged opposite to all the second target holes 53 on the calibration plate 5.

[0248] The number and size of the second through holes are set as needed so that the heat source of the second heating unit can reach the second target hole 53 through the second through holes.

[0249] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected by bolts. Bolt holes are provided on the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 respectively.

[0250] In this embodiment, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are all rectangular, and their lengths and widths are equal. The solution of this invention is not limited to the above embodiment.

[0251] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence to form a heating target assembly, which is fixed on a chassis 1 to form a heating target box. In this embodiment, the entire heating target assembly is fixed to the sheet metal chassis 1 with screws.

[0252] like Figure 17 As shown, the entire chassis 1 (e.g., a sheet metal chassis) is machined from 2mm steel plate and finished with black powder coating. The heating unit uses a heating rod for heating, with heat transferred to the heat source 21. A sheet metal door assembly 6 with a fan is installed on chassis 1; the fan is used for heat dissipation. The calibration plate 5 serves as a side panel of chassis 1, and the heat source fixing plate 3 is located inside chassis 1. A human-machine interface 8, a power switch 9, a start switch 10, and indicator lights 7 for displaying the heating status are installed on chassis 1. The human-machine interface 8 and indicator lights 7 are connected to a controller. The controller contains a control unit and a temperature control module. The control unit is connected to the temperature detection element 23, the temperature control module, the human-machine interface 8, and the indicator lights 7. The temperature control module is connected to the heating element 22, controlling the temperature of each heat source 21. The temperature of each heating unit is controlled through the human-machine interface 8, and each target can be controlled individually as required. Two handles are installed on chassis 1 for transporting the heating targets. Its working principle is that the heating rod heats the material and transfers the heat to the heat source 21. The thermocouple has a built-in temperature sensor to sense the real-time temperature of the heat source 21 and displays the actual sensed temperature through the human-machine interface 8. If the actual measurement requires the temperature to be adjusted to rise or fall, the temperature can be set through the human-machine interface 8. Each temperature can be controlled individually (adjustment range 25°~120°).

[0253] Furthermore, the heating target box is mounted on the frame 11. In this embodiment, the heating target box is fixed to the frame 11 with screws.

[0254] Furthermore, the lower end of the frame 11 is provided with casters 116.

[0255] The frame 11 includes a body frame 111, a fixing plate 112, a double sheet metal door 113, a stainless steel cover plate 114, and a frame base plate 115. The body frame 111 is made of stainless steel square tubing welded around its perimeter. The rear and side fixing plates 112 are made of stainless steel sheet metal and fixed to the frame 11 with screws. The front of the body frame 111 has a stainless steel double door, and the top stainless steel cover plate 114 is fixed to the frame 11 for fixing the heating target box. The base plate of the frame 11 is made of stainless steel sheet metal and is equipped with a caster wheel with feet at each corner for movement and fixation.

[0256] The calibration plate of this invention has target holes of various diameters, which can be adapted to tests at various distances (such as short distance and long distance, where distance refers to the distance between the infrared thermal imaging device and the calibration plate). It has strong versatility, can meet the optical axis testing needs of existing products, and can satisfy the requirement of one machine for multiple uses.

[0257] Each target in this invention employs an individual temperature control system, enabling precise temperature control, resulting in highly accurate and stable targets, ensuring temperature consistency, and improving test accuracy.

[0258] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An infrared-visible light image fusion method, characterized in that, Includes the following steps: Acquire visible light and infrared images; Image fusion specifically includes: obtaining the current target distance, obtaining the corresponding affine matrix based on the current target distance, and fusing the visible light image and the infrared image using the affine matrix corresponding to the current target distance to obtain a fused image; The method for obtaining the affine matrix of the visible light image corresponding to the infrared image at various distances specifically includes: Visible light images and infrared images were acquired at the set calibration distances. Extract and save the center of the calibration plate from the visible light image and infrared image at the calibration distance, and calculate the affine matrix at the calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at the calibration distance. The average value of the affine matrix at all calibration distances is calculated. Based on the center and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to correct the X and Y offset components of the affine matrix at each calibration distance. Based on the corrected X and Y offset components of the affine matrix at each calibration distance, the X and Y offset components of the affine matrix at each set distance are calculated by interpolation.

2. The infrared-visible light image fusion method as described in claim 1, characterized in that: Before the image fusion step, the following steps are also included: obtaining and saving the affine matrix of the infrared image corresponding to the visible light image at a set distance.

3. The infrared-visible light image fusion method as described in claim 1, characterized in that: The visible light image and the infrared image are fused using the affine matrix corresponding to the current target distance to obtain a fused image. Specifically, this includes: performing an affine transformation on the visible light image according to the affine matrix corresponding to the current target distance, and fusing the transformed visible light image with the acquired infrared image to obtain a fused image.

4. The infrared-visible light image fusion method as described in claim 3, characterized in that: The transformed visible light image is fused with the acquired infrared image to obtain a fused image. Specifically, this includes: extracting visible light high-frequency information from the transformed visible light image and superimposing the visible light high-frequency information onto the infrared image to obtain the fused image.

5. The infrared-visible light image fusion method as described in claim 1, characterized in that: Distortion correction is performed after acquiring visible light and infrared images.

6. The infrared-visible light image fusion method as described in claim 1, characterized in that: After obtaining the current target distance, the following steps are also included: determining whether the current target distance is greater than a preset value; if so, scaling the visible light image or infrared image using fixed cropping and scaling parameters to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion; if not, obtaining the corresponding cropping and scaling parameters from a table based on the current target distance, scaling the visible light image or infrared image to obtain a scaled visible light image or infrared image, and using the scaled visible light image or infrared image for image fusion.

7. An infrared-visible light image fusion device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the infrared-visible light image fusion method as described in any one of claims 1 to 6.

8. An infrared-visible light image registration method, characterized in that, Includes the following steps: Visible light images and infrared images were acquired at various set calibration distances. Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance. The affine matrix of the infrared image corresponding to the visible light image at each set distance is calculated based on the affine matrix at each calibration distance. Specifically, this includes: averaging the affine matrices at all calibration distances; using the nearest neighbor distance method to correct the X and Y offset components of the affine matrix at each calibration distance based on the center and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at each calibration distance; and using interpolation to calculate the X and Y offset components of the affine matrix at each set distance based on the corrected X and Y offset components of the affine matrix at each calibration distance.

Images