A method, fusion method, and registration device for infrared and visible light images.

By employing a two-stage precise registration strategy, and utilizing feature extraction and matching to correct translation parameter deviations, the shortcomings of existing infrared and visible light image registration methods in terms of accuracy and robustness are addressed, achieving efficient precise registration and fusion in infrared and visible light image systems.

CN117197238BActive Publication Date: 2026-11-10WUHAN GUIDE SENSMART TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311179651.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-13
Publication Date
2026-11-10
Estimated Expiration
2043-09-13

AI Technical Summary

Technical Problem

Existing infrared and visible light image registration methods are not accurate enough in some scenarios. Affine registration methods based on target calibration rely on the accuracy of optical design and Hall distance and are easily affected by errors. Methods that do not require data calibration have complex algorithms and are not robust, making them difficult to apply in practice.

Method used

A two-stage precise registration strategy is adopted. First, coarse registration is performed using the affine transformation parameters obtained from target calibration. Then, the translation parameter deviation in the affine transformation is corrected by feature extraction and matching. The translation parameter offset of the feature point set is used for correction to achieve precise registration of infrared and visible light images.

Benefits of technology

It achieves high-precision registration in infrared and visible light imaging systems, possesses strong robustness and practicality, and can automatically correct affine transformation deviations in real time, providing accurate dual-light fusion results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117197238B_ABST
    Figure CN117197238B_ABST
Patent Text Reader

Abstract

This invention discloses a registration method, fusion method, and registration device for infrared and visible light images. The registration method includes the following steps: acquiring a visible light image and an infrared image, using one image as a reference image and the other as the image to be registered; performing an affine transformation on the image to be registered using calibrated affine transformation parameters to obtain a first registered image; registering the reference image with the first registered image to obtain the translation parameter offset of the first registered image relative to the reference image; and correcting the translation parameters in the calibrated affine transformation parameters using the translation parameter offset to obtain the corrected affine transformation parameters. This invention designs a two-stage accurate registration strategy based on target-calibrated affine registration, which can automatically and in real-time correct registration deviations caused by affine transformations. It also possesses strong robustness, practicality, and universality, making it well-suited for application in infrared and visible light dual-light fusion systems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a registration method, fusion method, and registration device for infrared and visible light images. Background Technology

[0002] In existing infrared and visible light dual-light fusion systems, the mainstream methods mostly employ affine registration based on target calibration. This involves pre-calibrating target data at different distances for each device to generate corresponding affine registration parameters, achieving advantages such as low algorithm complexity, short processing time, strong robustness, and good practicality. However, the registration accuracy of this method is highly dependent on the optical design of the dual-light system and the accuracy of the Hall distance. Even slight errors in these two aspects can lead to significant deviations in the visible light features of the fused image.

[0003] Currently, scholars have also studied some methods that can directly register infrared and visible light images without data calibration, such as: region-based methods (correlation, mutual information, Fourier transform) and feature-based methods (points, lines, curves, surfaces). However, these methods can usually only be successfully registered in a very few specific scenarios and are extremely time-consuming, so they are difficult to apply to specific infrared and visible light dual-light fusion systems.

[0004] Therefore, the existing technical problems include at least the following: 1) affine registration methods based on target calibration will exhibit serious deviations in some scenarios; 2) methods that do not require data calibration have complex algorithms, weak robustness, and lack practical application value. Summary of the Invention

[0005] The purpose of this invention is to overcome at least one defect in the prior art and provide a registration method, fusion method and registration device for infrared and visible light images. Based on the affine registration method with target calibration, a two-stage accurate registration strategy is designed, which can automatically and in real time correct the registration deviation caused by affine transformation. It also has strong robustness, practicality and universality, and can be well applied in infrared and visible light dual-light fusion systems.

[0006] The technical solution of this invention is implemented as follows: This invention discloses a method for registering infrared and visible light images, comprising the following steps:

[0007] Acquire visible light and infrared images, and use one image as a reference image and the other as the image to be registered;

[0008] The first registered image is obtained by performing an affine transformation on the image to be registered using the affine transformation parameters obtained from the calibration.

[0009] The reference image is registered with the first registered image to obtain the translation parameter offset of the first registered image relative to the reference image. The translation parameter offset is then used to correct the translation parameter in the calibrated affine transformation parameters to obtain the corrected affine transformation parameters.

[0010] Furthermore, the infrared and visible light image registration method of the present invention also includes the following steps: performing an affine transformation on the image to be registered using the corrected affine transformation parameters to obtain a registered image.

[0011] Further, the reference image is registered with the first registered image to obtain the translation parameter offset of the first registered image relative to the reference image, specifically including:

[0012] Discretize the reference image and the first registration image respectively, and use one of the feature point sets as the reference point set and the other feature point set as the registration point set;

[0013] Register the reference point set with the point set to be registered, and calculate the translation parameter offset of the point set to be registered relative to the reference point set.

[0014] Furthermore, before discretizing the reference image and the first registered image, the following steps are also included:

[0015] Feature extraction includes: extracting the boundaries of the reference image and the first registration image respectively to obtain a first boundary binary image and a second boundary binary image;

[0016] Feature matching includes: matching the boundary features in the first boundary binary image and the second boundary binary image with each other, and deleting the boundary features in the first boundary binary image and the second boundary binary image that do not match each other.

[0017] Furthermore, feature matching specifically includes: identifying and deleting non-matching second boundary features in the second boundary binary image based on the first boundary features in the first boundary binary image, and identifying and deleting non-matching first boundary features in the first boundary binary image based on the second boundary features in the second boundary binary image.

[0018] Furthermore, based on the first boundary features in the first boundary binary image, mismatched second boundary features in the second boundary binary image are identified and deleted, specifically including:

[0019] Traverse the second boundary binary image and find the pixel positions that are greater than 0;

[0020] Find the same pixel position in the first boundary binary image, and search the neighborhood of that position with the set search radius to see if there is a pixel with a gray value greater than 0. If there is, keep the gray value of that pixel in the second boundary binary image unchanged; if not, set the gray value of that pixel in the second boundary binary image to 0.

[0021] Repeat the above steps until all pixels of the second boundary binary image have been traversed.

[0022] Furthermore, based on the second boundary features in the second boundary binary image, the first boundary binary image is identified and deleted if the first boundary features do not match. Specifically, this includes:

[0023] Traverse the first boundary binary image and find the pixel positions with gray values ​​greater than 0;

[0024] Find the same pixel position in the second boundary binary image. At this position, search the neighborhood for a pixel with a gray value greater than 0 using the set search radius. If there is a pixel, keep the gray value of that pixel in the first boundary binary image unchanged; otherwise, set the gray value of that pixel in the first boundary binary image to 0.

[0025] Repeat the above steps until all pixels of the first boundary binary image have been traversed.

[0026] Furthermore, an image preprocessing step is included before the feature matching step. The image preprocessing step includes:

[0027] The reference image and the first registration image are scaled respectively;

[0028] or / and,

[0029] Delete the edge portions around the reference image and the first registration image according to the set ratio;

[0030] or / and,

[0031] Convert the reference image and the first registered image into grayscale images;

[0032] or / and,

[0033] Image smoothing is performed on the reference image and the first registration image respectively.

[0034] Furthermore, the steps following the feature extraction step and before the feature matching step include the following steps: performing connected component analysis on the first boundary binary image and the second boundary binary image respectively, obtaining the area value of each connected region, and setting the gray value of pixels in connected regions with an area value less than a preset area threshold to 0.

[0035] Furthermore, after discretizing the reference image and the first registration image, a downsampling step is also included. The downsampling step includes downsampling the reference point set and the point set to be registered obtained after discretization.

[0036] Furthermore, the reference point set and the point set to be registered are registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated, specifically including:

[0037] (a) Initialization;

[0038] (b) Calculate the covariance matrix between the reference point set and the point set to be registered;

[0039] (c) Calculate the translation parameter offset of the point set to be registered relative to the reference point set based on the covariance matrix;

[0040] (d) Calculate the coordinates of the updated set of points to be registered and the set of reference points;

[0041] (e) Update the covariance matrix;

[0042] (f) Determine whether the stopping iteration condition is met based on the covariance matrix. If the stopping iteration condition is met, stop the iteration and output the currently calculated translation parameter offset; otherwise, return to step (c) to continue iterating until the stopping iteration condition is met.

[0043] Furthermore, based on the covariance matrix, the translation parameter offset of the set of points to be registered relative to the reference set is iteratively solved, specifically including: calculating the matching degree matrix based on the covariance matrix;

[0044] Calculate the mean of the reference point set and the mean of the point set to be registered based on the matching degree matrix;

[0045] The translation parameter offset of the point set to be registered relative to the reference point set is calculated based on the mean of the reference point set and the mean of the point set to be registered.

[0046] Furthermore, based on the covariance matrix, the translation parameter offset of the point set to be registered relative to the reference point set is iteratively solved, specifically including:

[0047] Calculate the matching degree matrix P, where each element p mn The calculation is as follows: in, w is a noise level parameter, σ 2 Let ir be the covariance matrix, and ir = {x_ir} n ,y_ir n Let} be the reference point set, N be the number of feature points in the reference point set, and vis = {x_vis} n ,y_vis nLet M be the set of points to be registered, and M be the number of feature points in the set of points to be registered.

[0048] Calculate the mean μ of the reference point set ir The formula is: Calculate the mean μ of the set of points to be registered vis The formula is: Where 1 represents the identity matrix of the corresponding dimension;

[0049] Calculate the translation parameter offset t = {e', f'} ​​of the point set to be registered relative to the reference point set, where t = μ. ir -μ vis .

[0050] Furthermore, the coordinates of the reference point set and the set to be registered are updated based on the mean of the reference point set and the mean of the set to be registered; the covariance matrix is ​​updated based on the updated coordinates of the reference point set and the set to be registered.

[0051] Furthermore, calculate the updated point set coordinates irNew and visNew: irNew = ir - μ ir T ,visNew=vis-μ vis T ir = {x_ir n ,y_ir n} is the reference point set, vis = {x_vis} n ,y_vis n} represents the set of points to be registered;

[0052] Update covariance matrix Where tr() is the trace of the matrix, d() is the diagonal element of the matrix, and 1 is the identity matrix of the corresponding dimension;

[0053] Furthermore, the limit is calculated based on the covariance matrix, and the stopping condition for iteration is: the absolute value of the percentage reduction of the initial limit by the calculated limit is less than the set limit threshold.

[0054] Furthermore, the formula for calculating the limit error E is: E = N * sum(logσ) 2 ), where sum() is the sum of all elements of the matrix, σ 2 Let N be the covariance matrix, and N be the number of feature points in the reference point set.

[0055] Furthermore, the corrected horizontal translation parameter is the sum of e and e', and the corrected vertical translation parameter is the sum of f and f'. e is the horizontal translation parameter obtained from calibration, e' is the horizontal translation parameter offset, f is the vertical translation parameter obtained from calibration, and f' is the vertical translation parameter offset.

[0056] The present invention also discloses an infrared and visible light image registration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared and visible light image registration method described above.

[0057] This invention also discloses a method for fusing infrared and visible light images, comprising the following steps:

[0058] Acquire visible light and infrared images, and use one image as a reference image and the other as the image to be registered;

[0059] The corrected affine transformation parameters are obtained using the registration method described above;

[0060] The affine transformation parameters are used to perform an affine transformation on the image to be registered. The transformed registered image is then fused with the reference image to obtain a fused image.

[0061] The present invention has at least the following beneficial effects: The present invention provides a two-stage accurate registration method for infrared and visible light images based on sparse features. For each frame of dual-light image, it can automatically and in real time correct the registration deviation caused by affine transformation, presenting an accurate dual-light fusion result. At the same time, it has strong robustness, practicality and universality, and can be well applied in infrared and visible light dual-light fusion systems. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating the infrared and visible light image registration method provided in an embodiment of the present invention;

[0064] Figure 2 A flowchart of the second-stage registration process provided in an embodiment of the present invention;

[0065] Figure 3 A feature matching flowchart provided for embodiments of the present invention;

[0066] Figure 4 A raw visible light image provided for one embodiment;

[0067] Figure 5 A raw infrared image provided for one embodiment;

[0068] Figure 6 An image obtained by fusing the original infrared image and the visible light image registered in the first stage, as provided in one embodiment;

[0069] Figure 7 An image obtained by fusing the original infrared image and the visible light image after second-stage registration, as provided in one embodiment;

[0070] Figure 8 This is a schematic diagram of the calibration plate provided in an embodiment of the present invention;

[0071] Figure 9 A cross-sectional view of the calibration plate provided in an embodiment of the present invention;

[0072] Figure 10 for Figure 9 Enlarged view of part P;

[0073] Figure 11 This is a schematic diagram of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0074] Figure 12 An exploded view of the heating target box of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0075] Figure 13 This is a schematic diagram of the heating unit of the infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0076] Figure 14 An exploded view of the frame of the infrared dual-light fusion testing device provided in an embodiment of the present invention.

[0077] In the attached diagram, 1 is the chassis, 2 is the heating unit, 21 is the heat source, 211 is the first mounting hole, 212 is the second mounting hole, 22 is the heating element, 23 is the temperature detection element, 24 is the first heat insulation plate, 3 is the heat source fixing plate, 4 is the second heat insulation plate, 5 is the calibration plate, 51 is the calibration plate body, 52 is the first target hole, 53 is the second target hole, 6 is the sheet metal door assembly, 7 is the indicator light, 8 is the human-machine interface, 9 is the power switch, 10 is the start switch, 11 is the frame, 111 is the machine frame, 112 is the fixing plate, 113 is the double sheet metal door, 114 is the stainless steel cover plate, 115 is the frame base plate, and 116 is the caster. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Example 1

[0080] See Figure 1 and Figure 2 This invention discloses a method for registering infrared and visible light images, comprising the following steps:

[0081] Acquire the original visible light image and the original infrared image, and use one of the images as the reference image and the other image as the image to be registered. For example, in this embodiment, the original infrared image is used as the reference image and the original visible light image is used as the image to be registered.

[0082] First-stage registration: Obtain the current target distance, and obtain its corresponding affine transformation parameters based on the current target distance. These affine transformation parameters are obtained through calibration. The affine transformation parameters obtained through calibration are used to perform an affine transformation on the image to be registered, and the first registered image is obtained.

[0083] Second-stage registration: Register the reference image with the first registration image to obtain the translation parameter offset of the first registration image relative to the reference image. Use the translation parameter offset to correct the translation parameters in the calibrated affine transformation parameters to obtain the corrected affine transformation parameters.

[0084] The image to be registered is then subjected to an affine transformation using the corrected affine transformation parameters to obtain the final registered image.

[0085] The first stage of registration mainly relies on affine registration based on target calibration to coarsely register the input raw visible light image and infrared image. At this time, the scaling and rotation parameters are accurate, but there will be offset errors in the translation parameters.

[0086] To address the problems existing in the first-stage registration (after the first-stage registration, the scaling and rotation parameters are accurate, but there will be offset errors in the translation parameters), this invention performs a second-stage precise registration based on the first-stage registration to accurately correct the translation parameters in the affine transformation, thereby achieving accurate registration between the infrared image and the visible light image. The calculation formula for the affine transformation is as follows:

[0087]

[0088] In the above formula, x and y represent the pixel positions of the input visible light image, respectively. These represent the pixel positions of the registered visible light image. a, b, c, and d are the rotation and scaling parameters used in the affine transformation, while e and f are the translation and rotation parameters. The rotation, scaling, and translation parameters for the first-stage affine transformation are generated by pre-calibrating target data at different distances for each device. During use, the corresponding distance can be found by looking up the Hall distance returned by the device.

[0089] In some embodiments, the calibrated affine transformation parameters specifically include:

[0090] Paired visible light and infrared images were acquired at various set calibration distances.

[0091] Extract and save the center of the calibration plate from the visible light image and infrared image at each calibration distance. Calculate the affine matrix at each calibration distance by registering the center of each circle in the visible light image and the center of each circle in the infrared image at each calibration distance.

[0092] The affine matrix of the infrared image corresponding to the visible light image at any distance is calculated based on the affine matrix at each calibration distance.

[0093] Furthermore, based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated, specifically including:

[0094] The average value of the rotation and scaling components in the X and Y directions of the affine matrix under all calibration distances is obtained. The affine matrix includes the X-direction scaling component a, the X-direction rotation component b, the Y-direction rotation component c, the Y-direction scaling component d, the X-direction offset component e, and the Y-direction offset component f.

[0095] Using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are recalculated (if they are not recalculated, the offset components cannot be regularly increased or decreased), and the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are obtained.

[0096] Based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, calculate the X-direction offset components and Y-direction offset components of the affine matrix at any distance.

[0097] An affine matrix at any distance can be generated by averaging the X and Y offset components of the affine matrix at each set distance, along with the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances. This can be achieved by assigning the X and Y offset components of the affine matrix at each set distance to the affine average matrix, or by assigning the average of the X and Y offset components of the affine matrix at each set distance, along with the rotation and scaling components in the X and Y directions of the affine matrix at all calibrated distances, to the affine matrix.

[0098] The affine matrix is ​​obtained by averaging the affine matrices for all calibrated distances.

[0099] Furthermore, by using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances (i.e., the average value of the rotation and scaling matrix in the affine matrix at each calibration distance), the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance are calculated in reverse.

[0100] Furthermore, using the center of the calibration plate in the visible light and infrared images at each calibration distance, and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance.

[0101] The principle of the nearest neighbor distance method of this invention is as follows: Under a single distance, the coordinates of each circle center in the visible calibration image are cyclically input, and the offset vectors in the X direction and Y direction corresponding to the circle center coordinates in the corresponding infrared calibration image can be calculated. For example, the first circle center after sorting in the visible calibration image corresponds to the first circle center after sorting in the infrared calibration image. The corresponding offset vectors in the X direction and Y direction are calculated and denoted as an, bn. Then, by traversing all the circle centers corresponding to the infrared calibration image circles under this distance, the circle centers in the visible calibration image are transformed into the circle centers in the infrared calibration image, and the straight-line distance dis between the circle centers extracted from the infrared calibration image and the circle centers is calculated. For example, if 25 circle centers are extracted from both the infrared calibration image and the visible calibration image, the loop is repeated 25 times, and the minimum straight-line distance value is accumulated and a set of an, bn is denoted as the offset vectors in the X direction and Y direction under the current distance.

[0102] Using the centers of the calibration plate in the visible and infrared images at various calibration distances, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, the nearest neighbor distance method is used to calculate the X and Y offset components of the affine matrix at each calibration distance. Specifically, this includes: at a single calibration distance, obtaining the centers of the calibration plate in the visible and infrared images at that calibration distance, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances; for each pair of corresponding visible and infrared calibration image centers, first obtaining the X and Y offset components, and the average values ​​of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, to form a new affine matrix (a pair of corresponding visible and infrared calibration image centers). Each new affine matrix corresponds to a new circle center. Using each new affine matrix, the coordinates of the circle center in the visible calibration image transformed to the center in the infrared calibration image are calculated and designated as the second circle center in the infrared calibration image. The offset vectors in the X and Y directions, as well as the straight-line distance dis, between the circle center and the first circle center coordinates in the corresponding infrared calibration image (i.e., the extracted circle center in the infrared calibration image) are calculated. The straight-line distances dis corresponding to all circle centers are summed to obtain the cumulative straight-line distance value corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X and Y directions, denoted as an and bn. The set of offset vectors in the X and Y directions corresponding to the minimum value of the cumulative straight-line distance is taken as the offset vectors in the X and Y directions under the current calibration distance.

[0103] Furthermore, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated. Specifically, based on the X-direction offset components and Y-direction offset components of the affine matrix at each corrected calibration distance, the X-direction offset components and Y-direction offset components of the affine matrix at any distance are calculated using interpolation.

[0104] One specific embodiment of the calibration process is as follows:

[0105] Acquire 14-bit / 16-bit high dynamic range infrared (IR) images and 8-bit visible light (VIS) images.

[0106] Identify the center of the calibration plate on the 14-bit / 16-bit high dynamic range infrared (IR) image and the 8-bit visible light (VIS) image, and store them as ir_centerCiclePoints and vis_centerCiclePoints respectively, and sort them.

[0107] Registration is performed based on vis_centerCiclePoints and ir_centerCiclePoints to obtain the affine matrix VISToIRMatrixs.

[0108] AffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints,VISToIRMatrixs);

[0109] The above process is used to perform calibration at a set calibration distance. In this embodiment, there are 9 calibration distances, ranging from 1 meter to 9 meters. Of course, the number of calibration distances is not limited to 9 and can be adjusted as needed. All extracted center points vis_centerCiclePointList and ir_centerCiclePointList are saved, and the center points of the visible light image and the center points of the infrared image are matched one by one to generate affine matrices VISToIRMatrixs for each distance.

[0110] The rotation and scaling components of the affine matrix under the 9 sets of data are averaged (i.e., a, b, c, d), and stored as VISToIRMatrix_average, which is represented as:

[0111]

[0112] In the affine matrix, a represents the scaling ratio in the X direction, b represents the rotation ratio in the X direction, c represents the rotation ratio in the Y direction, d represents the scaling ratio in the Y direction, e represents the translation distance in the X direction, and f represents the translation distance in the Y direction. In a two-dimensional image, the X-direction vector and the Y-direction vector have different properties during affine transformation.

[0113] For example, in this embodiment, at a distance of 1 meter, a = 2.71771, b = 0.0005355351, e = -1027.16, c = -0.00148628, d = 2.77923, f = -1125.53.

[0114] The study found that the X-direction offset changes more significantly at close range, while the change is almost negligible at greater distances.

[0115] Using the saved vis_centerCiclePointList and ir_centerCiclePointList, and the average affine matrix VISToIRMatrix_average, the nearest neighbor offset distance method is used to calculate the X-direction component e and the Y-direction component f at each distance.

[0116] ReverseCalAffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints);

[0117] Interpolate the calculated e and f values ​​for each group, and calculate the e and f values ​​for each distance from 1 to 20 meters in units of 10 centimeters. The x and y values ​​range from 1 to 9 meters, the range is 1 to 20 meters, and the interval is 0.1 meters. The output is a lookup table of the X-direction offset vector and the Y-direction offset vector.

[0118] Interp1HermitePchip(distance,x,range,e);

[0119] Interp1HermitePchip(distance,y,range,f);

[0120] Finally, the target distance is obtained from the Hall contrast table, and the e and f values ​​at the corresponding distance are obtained by looking up the table based on the target distance. These values ​​are then assigned to VISToIRMatrix_average to obtain the affine matrix at that distance.

[0121] Further, the reference image is registered with the first registered image to obtain the translation parameter offset of the first registered image relative to the reference image, specifically including:

[0122] Discretize the reference image and the first registration image respectively, and use one of the feature point sets as the reference point set and the other feature point set as the registration point set;

[0123] Register the reference point set with the point set to be registered, and calculate the translation parameter offset of the point set to be registered relative to the reference point set.

[0124] In some embodiments, the reference point set is obtained by discretizing a reference image, and the point set to be registered is obtained by discretizing a first registration image.

[0125] Furthermore, before discretizing the reference image and the first registered image, the following steps are also included:

[0126] Feature extraction includes: extracting the boundaries of the reference image and the first registration image respectively to obtain a first boundary binary image and a second boundary binary image;

[0127] Feature matching includes: matching the boundary features in the first boundary binary image and the second boundary binary image with each other, and deleting the boundary features in the first boundary binary image and the second boundary binary image that do not match each other.

[0128] Furthermore, an image preprocessing step is included before the feature matching step. The image preprocessing step includes scaling the reference image and the first registration image respectively.

[0129] After the first stage of registration, the image to be registered (such as the original visible light image) becomes the first registered image (i.e. the visible light image registered in the first stage). At this point, a series of preprocessing steps are performed on the infrared and visible light images to provide boundary feature point support for a more efficient and accurate second stage of registration.

[0130] One embodiment involves simultaneously scaling the visible light image and the original infrared image registered in the first stage. The purpose of this step is to downsample at the image level, reducing the amount of image data and thus lowering the algorithm's time complexity. Specifically, the scaling factor is related to the image resolution, and the scaling factor is primarily based on specific testing, aiming to scale as large as possible without significantly affecting registration accuracy. For example, a 1280*1024 resolution image is recommended to be scaled by a factor of 8.

[0131] Image registration typically requires geometric features located in the central region of the image. Therefore, after reducing the image size, a certain percentage of the edges are removed to further reduce the amount of data at each image level. For the same purpose, the image is converted to grayscale to reduce unnecessary channels involved in subsequent calculations.

[0132] In some embodiments, the image preprocessing step further includes: deleting the edge portions around the reference image and the first registration image according to a set ratio, and converting the reference image and the first registration image into grayscale images.

[0133] Since images themselves contain a great deal of texture information, but registration does not concern itself with texture information, focusing only on the main contour information. This invention subsequently requires boundary extraction of the image. If Canny boundary extraction is directly performed on the original image, this unnecessary texture information will generate many messy edges, increasing computational load and causing data errors that affect registration accuracy. Therefore, image smoothing is necessary before the boundary extraction step. This not only effectively smooths out textured areas but also enhances boundaries, allowing for the extraction of more boundary information in subsequent steps.

[0134] In some embodiments, the image preprocessing step further includes: smoothing the reference image and the first registration image respectively.

[0135] Furthermore, image smoothing is performed on the infrared image and the visible light image separately, specifically including: sparse smoothing of the infrared image and the visible light image separately. Preferably, L0 sparse smoothing is performed on the infrared image and the visible light image separately.

[0136] Applying L0 sparse smoothing to both infrared and visible grayscale images can smooth out small internal details while strengthening boundaries, providing high-dimensional and sparse boundary information for the next boundary extraction step. An example of L0 sparse smoothing for infrared grayscale images is provided below, with the specific steps as follows: First, establish an L0 sparse smoothing model:

[0137]

[0138] in, In the formula: irg is the input infrared grayscale image, irg L0 Let p represent a single pixel in the L0 sparse smoothed image. #{·} is a counting operator that counts the number of pixels satisfying the conditions within it. λ>0 is a non-negative regularization coefficient. These are the forward difference operators in the x and y directions, respectively. Next, the L0 sparse smooth model is transformed into a form solvable by decomposable variables using the variable partitioning method. By iteratively solving the subproblems, the optimal solution irg of equation (2) can be obtained. L0 .

[0139] Further, feature extraction specifically includes: performing Canny boundary extraction on the reference image and the first registration image respectively to obtain a first boundary binary image and a second boundary binary image.

[0140] By employing the L0 sparse smoothing model, minute details and textures in infrared and visible grayscale images are smoothed out, and strong boundaries are enhanced. This preserves high-dimensional sparse feature information for boundary extraction, enabling the extraction of more effective and significant high-dimensional boundary features. Boundary extraction mainly consists of vertical and horizontal gradient calculation, vertical and horizontal gradient smoothing, gradient amplitude calculation, adaptive threshold selection, and non-maximum suppression. Compared to the OpenCV version of the Canny function, this method uses double precision instead of integer data, and the threshold selection is changed from manual input to automatic selection based on the histogram statistics of the image gradient. This allows for the extraction of boundary features from low-contrast images that the OpenCV version of Canny cannot extract.

[0141] By employing adaptive high-precision Canny boundary extraction, binary sparse boundary images for infrared and visible light were obtained. In the extracted binary boundary images, small boundaries can be considered unimportant geometric information; deleting them reduces the amount of data and unnecessary noise. The boundary length is determined through connected component analysis. For the binary boundary image pair (i.e., the first and second boundary binary images), connected component analysis is performed on both (e.g., using OpenCV's `connectedComponentsWithStats` function) to obtain the area value of each connected region in the first and second boundary binary images (i.e., calculating the area length of the connected boundary). This invention reduces invalid registration information by setting a corresponding area threshold to delete small-area connected boundaries. Specifically, when traversing connected regions below the preset area threshold, the grayscale value of all pixels within the connected region is set to 0.

[0142] In some embodiments, 8-connected component analysis is performed on the infrared and visible light binary images respectively.

[0143] Infrared and visible light are heterogeneous images, and the boundary binary images extracted from them are quite different. The purpose of the feature matching step is to match similar boundary features and delete mismatched boundary features to reduce unnecessary noise data in the registration.

[0144] Furthermore, feature matching specifically includes: a first step and a second step. The first step includes: identifying and deleting non-matching visible light boundary features in the visible light boundary binary image based on the infrared boundary features in the infrared boundary binary image.

[0145] The second step includes: identifying and deleting non-matching infrared boundary features in the infrared boundary binary image based on the visible light boundary features in the visible light boundary binary image.

[0146] Furthermore, based on the first boundary features in the first boundary binary image, mismatched second boundary features in the second boundary binary image are identified and deleted, specifically including:

[0147] Traverse the second boundary binary image and find the pixel positions with gray values ​​greater than 0;

[0148] Find the same pixel position in the first boundary binary image, and search the neighborhood of that position with the set search radius to see if there is a pixel position with a gray value greater than 0. If there is, keep the gray value of that pixel in the second boundary binary image unchanged; if not, set the gray value of that pixel position in the second boundary binary image to 0.

[0149] Repeat the above steps until all pixels of the second boundary binary image have been traversed.

[0150] Furthermore, based on the second boundary features in the second boundary binary image, the first boundary binary image is identified and deleted if the first boundary features do not match. Specifically, this includes:

[0151] Traverse the first boundary binary image and find the pixel positions with gray values ​​greater than 0;

[0152] Find the same pixel position in the second boundary binary image. At this position, search the neighborhood for a pixel with a gray value greater than 0 using the set search radius. If there is a pixel, keep the gray value of that pixel in the first boundary binary image unchanged; otherwise, set the gray value of that pixel in the first boundary binary image to 0.

[0153] Repeat the above steps until all pixels of the first boundary binary image have been traversed.

[0154] The order in which the first and second steps are performed can be reversed.

[0155] See Figure 3 A specific implementation of feature matching is as follows: A search radius is set. First, pixels with a grayscale value greater than 0 are traversed in the visible light boundary binary image. The same position is found in the infrared boundary binary image. Using the set search radius, a search is performed to determine if there is a pixel with a grayscale value of 255 in that neighborhood. If found, the grayscale value of that pixel in the visible light boundary binary image remains unchanged; if not found, the grayscale value is set to 0. After traversing all visible light boundary binary images, the visible light image after matching deletion is obtained. Then, following the same logic, all positions in the infrared boundary binary image are searched. Using the visible light image after matching deletion as the standard for preserving the grayscale values ​​of pixels in the infrared boundary binary image, a matching-deleted infrared boundary binary image is generated.

[0156] Furthermore, the reference image and the first registration image are discretized, specifically including:

[0157] Establish coordinate systems in the reference image and the first registration image respectively: with the top left corner of the image as the origin, the horizontal direction as the positive x-axis, and the vertical direction as the positive y-axis. Perform a search and traversal in the obtained first and second boundary binary images after mutual matching and deletion, and generate a first feature point set and a second feature point set (i.e., infrared feature point set and visible light feature point set) according to the above coordinate system definition for the pixel positions where the grayscale value is not 0. Use one of the obtained feature point sets (boundary feature point set) as the reference point set and the other feature point set (boundary feature point set) as the registration point set, such as using the infrared feature point set as the reference point set and the visible light feature point set as the registration point set.

[0158] Further, after discretizing the reference image and the first registered image, the method further comprises a downsampling step, which includes: downsampling the reference point set and the to-be-registered point set obtained after the discretization processing.

[0159] The downsampling of the point set is intended to reduce the number of points in the point set as much as possible while minimizing the loss of registration accuracy, so as to reduce the time consumption of the subsequent registration algorithm. The principle of the point set downsampling algorithm is to select one point at intervals of a certain number of points in the horizontal direction and the vertical direction respectively, and delete all the other points in between. Before executing the algorithm, three point number thresholds pn_threshold1, pn_threshold2 and pn_threshold3 are set respectively in advance. Then, the number pn_ir of the reference point set (such as the number of infrared point sets) and the number pn_vis of the to-be-registered point set (such as the number of visible light point sets) are obtained respectively. For the reference point set (such as the number of infrared point sets), the point sampling interval is selected for downsampling according to the following rules: (a) if pn_ir ≤ threshold1, points are sampled at an interval of 1; (b) if threshold1 < pn_ir ≤ threshold2, points are sampled at an interval of 2; (c) if threshold3 ≤ pn_ir, points are sampled at an interval of 3. Similarly, for the to-be-registered point set (such as the number of visible light point sets), the point sampling interval is also selected for downsampling according to the following rules: (a) if pn_vis ≤ threshold1, points are sampled at an interval of 1; (b) if threshold1 < pn_vis ≤ threshold2, points are sampled at an interval of 2; (c) if threshold3 ≤ pn_vis, points are sampled at an interval of 3.

[0160] After the point set downsampling, a point set registration step is performed.

[0161] Since the coarse registration has been completed in the first-stage registration, the scaling and rotation parameters are accurate, and there is only offset error in the translation parameters. Therefore, in the second-stage registration, only the translation parameters are considered, and the registration between the infrared point set and the visible light point set is completed, so that the accurate correction of the translation parameters can be obtained. Therefore, the formula (1) of the affine transformation can be converted into the following form:

[0162]

[0163] Here, x' and y' respectively represent the horizontal and vertical coordinate values of points in the visible light point set after the first-stage registration, respectively represent the horizontal and vertical coordinate values of points in the visible light point set after the second-stage registration, and e' and f' are the translation correction parameters in the horizontal direction and the vertical direction obtained by the second-stage registration.

[0164] The registration scheme for the second stage of registration design is an affine transformation based on a Gaussian mixture model that only considers translation. The specific steps are as follows:

[0165] (a) Algorithm initialization, such as translation parameters e' = 0, f' = 0, noise level parameter w = 0.1, limit threshold tolE = 0.00001, initial limit E old =1;

[0166] (b) Calculate the covariance matrix between the reference point set and the point set to be registered: Where ir = {x_ir} n ,y_ir n} represents the original infrared point set, where the number of feature points is N. vis = {x_vis} n ,y_vis n} represents the set of visible light points after the first stage of registration, where the number of feature points is M.

[0167] (c) Calculate the translation parameter offset of the point set to be registered relative to the reference point set based on the covariance matrix;

[0168] (d) Calculate the coordinates of the updated set of points to be registered and the set of reference points;

[0169] (e) Update the covariance matrix;

[0170] (f) Determine whether the stopping iteration condition is met based on the covariance matrix of step (e). If the stopping iteration condition is met, stop the iteration and output the currently calculated translation parameter offset t = {e', f'}; otherwise, return to step (c) to continue iterating until the stopping iteration condition is met.

[0171] Furthermore, based on the covariance matrix, the translation parameter offset of the set of points to be registered relative to the reference set is iteratively solved, specifically including: calculating the matching degree matrix based on the covariance matrix;

[0172] Calculate the mean of the reference point set and the mean of the point set to be registered based on the matching degree matrix;

[0173] The translation parameter offset of the point set to be registered relative to the reference point set is calculated based on the mean of the reference point set and the mean of the point set to be registered.

[0174] Furthermore, based on the covariance matrix, the translation parameter offset of the point set to be registered relative to the reference point set is iteratively solved, specifically including:

[0175] Calculate the matching degree matrix P, where each element p mn The calculation is as follows: in, w is a noise level parameter, σ 2 Let ir be the covariance matrix, and ir = {x_ir} n ,y_ir n Let} be the reference point set, N be the number of feature points in the reference point set, and vis = {x_vis} n ,y_vis n Let} be the set of points to be registered, and M be the number of feature points in the set of points to be registered; p mn Let be the element in the m-th row and n-th column of matrix P.

[0176] Calculate the mean μ of the reference point set ir The formula is: Calculate the mean μ of the set of points to be registered vis The formula is: Where 1 represents the identity matrix of the corresponding dimension, 1 T P represents the transpose of the identity matrix 1. T ir is the transpose of P T For the transpose of ir, vis T This is the transpose of vis;

[0177] Calculate the translation parameter offset t = {e', f'} ​​of the point set to be registered relative to the reference point set, where t = μ. ir -μ vis .

[0178] Furthermore, the coordinates of the reference point set and the set to be registered are updated based on the mean of the reference point set and the mean of the set to be registered; the covariance matrix is ​​updated based on the updated coordinates of the reference point set and the set to be registered.

[0179] Furthermore, calculate the updated point set coordinates irNew and visNew: irNew = ir - μ ir T ,visNew=vis-μ vis T ir = {x_ir n ,y_ir n} is the reference point set, vis = {x_vis} n ,y_vis n Let} be the set of points to be registered, μ vir T For μ vis transpose, μ ir T For μ ir transpose;

[0180] Update covariance matrix Where tr() is the trace of the matrix, d() is the diagonal element of the matrix, 1 is the identity matrix of the corresponding dimension, and irNewT is the transpose of irNew.

[0181] Further, the limit difference E is calculated according to the covariance matrix, and the iteration stopping condition is: abs((E-E old ) / E)<tolE, where tolE is the limit difference threshold, E old is the initial limit difference.

[0182] Further, the calculation formula of the limit difference E is: E=N*sum(logσ 2 ), where sum() is the sum of all elements of the matrix, σ 2 is the covariance matrix, and N is the number of feature points of the reference point set.

[0183] Through the iterative solution of the second-stage registration, the offset t={e',f'} of the visible light image relative to the infrared image can be obtained, and then the translation parameter offset is used to correct the translation parameter of the affine transformation. The corrected translation parameter in the horizontal direction is the sum of e and e', the corrected translation parameter in the vertical direction is the sum of f and f', e is the translation parameter in the horizontal direction obtained by calibration, e' is the translation parameter offset in the horizontal direction, f is the translation parameter in the vertical direction obtained by calibration, and f' is the translation parameter offset in the vertical direction.

[0184] Input the offset t={e',f'} into the affine transformation parameters of the primary registration, then Equation (1) takes the following form

[0185]

[0186] Perform affine registration according to the expression of Equation (4), and output the final registered image (such as the final registered visible light image).

[0187] Based on the same inventive concept, the present invention further provides an infrared and visible light image registration apparatus, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein when the processor executes the computer program, the steps of the infrared and visible light image registration method as described above are implemented.

[0188] Second Embodiment

[0189] The embodiment of the present invention further discloses an infrared and visible light image fusion method, comprising the following steps:

[0190] Acquire an original visible light image and an original infrared image, and take one of the images as a reference image and the other image as an image to be registered;

[0191] Obtain corrected affine transformation parameters by using the registration method described in the first embodiment;

[0192] The affine transformation parameters are used to perform an affine transformation on the image to be registered. The transformed registered image is then fused with the reference image to obtain a fused image.

[0193] This invention performs a second stage of precise registration based on the first stage of registration to accurately correct the translation parameters in the affine transformation, thereby completing the precise registration of the infrared image and the visible light image.

[0194] Figure 4 The image shown is a raw visible light image provided in one embodiment; Figure 5 The image shown is an original infrared image provided in one embodiment; if the visible light image that has completed the first stage registration is fused with the original infrared image, the result is... Figure 6 The image shown, if the visible light image after the second stage of registration is fused with the original infrared image, yields... Figure 7 The image shown.

[0195] Based on the same inventive concept, the present invention also provides an infrared and visible light image fusion device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the infrared and visible light image fusion method described above.

[0196] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is used to run program code stored in the memory or process data, such as program code for a method of registering infrared and visible light images or a method of fusing infrared and visible light images.

[0197] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or RAM of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory may include both internal and external storage units of the electronic device. In this embodiment, the memory is typically used to store operating methods and various application software installed on the electronic device, such as program code for infrared and visible light image registration methods or infrared and visible light image fusion methods. Furthermore, the memory may also be used to temporarily store various types of data that have been output or will be output.

[0198] Based on the same inventive concept, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the infrared and visible light image registration method or the infrared and visible light image fusion method described above.

[0199] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0200] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] Example 3

[0204] See Figures 8 to 10 This invention discloses a calibration plate, including a calibration plate body 51. The calibration plate body 51 is provided with at least two sets of target holes. Each set of target holes includes multiple target holes. The target holes in the same set of target holes have the same diameter, while the target holes in different sets of target holes have different diameters.

[0205] Furthermore, each group of target holes corresponds to a region, and the area of ​​the region corresponding to each group of target holes is different. All target holes in each group of target holes must be located within the corresponding region.

[0206] Furthermore, when the area corresponding to the first set of target holes is larger than the area corresponding to the second set of target holes, the aperture of the first set of target holes is larger than the aperture of the second set of target holes. Areas can overlap; for example, a large area can completely cover a small area.

[0207] Furthermore, the area corresponding to each group of target holes is either a circular area or a polygonal area (polygonal areas include triangular areas, quadrilateral areas, pentagonal areas, etc.). That is, the area corresponding to each group of target holes is one of a circular area, a triangular area, a quadrilateral area, a pentagonal area, etc. In this embodiment, the area corresponding to each group of target holes is a rectangular area.

[0208] Furthermore, the centers of the regions corresponding to multiple sets of target holes are the same.

[0209] Furthermore, the areas corresponding to the multiple sets of target holes are the same as the center of calibration plate 5.

[0210] Furthermore, each group of target holes contains at least three target holes, and not all target holes in each group are on a straight line.

[0211] Furthermore, all target holes in each group are arranged in an array.

[0212] Furthermore, after all the target holes in each group are arranged in an array to form an array, the centers of each array are the same.

[0213] Furthermore, the spacing between the centers of two adjacent target holes in each row of each array is equal, and the spacing between the centers of two adjacent target holes in each column is equal.

[0214] Furthermore, the target hole is a circular hole, and the back opening of the target hole is chamfered. After the back opening of the target hole is chamfered, it has a connected circular hole segment and a conical hole segment. The circular hole segment is away from the heat source module, and the conical hole segment is close to the heat source module (the diameter of the conical hole segment gradually increases as it approaches the heat source module). The minimum diameter of the conical hole segment is equal to the diameter of the circular hole segment.

[0215] Furthermore, the front of the calibration plate body is treated with a matte white frosted finish, and the back of the calibration plate body is treated with a glossy black high-reflection finish.

[0216] In one embodiment, the calibration plate body 51 is provided with two sets of target holes, namely a first set of target holes and a second set of target holes. The first set of target holes includes a plurality of first target holes 52 (large round holes), and the second set of target holes includes a plurality of second target holes 53 (small round holes). The diameter of the first target holes 52 is larger than the diameter of the second target holes 53. All the first target holes 52 are located in a first region, and all the second target holes 53 are located in a second region.

[0217] Furthermore, the first region is a first rectangular region with a length of 800mm and a width of 600mm; the second region is a second rectangular region with a length of 70mm and a width of 50mm.

[0218] Furthermore, the centers of the first rectangular region and the second rectangular region are the same.

[0219] Furthermore, the first rectangular region and the second rectangular region are centered on the calibration plate 5.

[0220] Furthermore, the number of the first target hole 52 and the second target hole 53 is at least three, and not all of the first target holes 52 are on a straight line, nor are all of the second target holes 53 on a straight line.

[0221] The diameters of the first target hole 52 and the second target hole 53 are set as needed. In one embodiment, the diameter of the first target hole 52 is 100mm and the diameter of the second target hole 53 is 10mm.

[0222] Furthermore, multiple first target holes 52 are arranged in an array to form a first array, and multiple second target holes 53 are arranged in an array to form a second array, with the center of the first array being the same as the center of the second array.

[0223] Furthermore, in the first array, the distance between the centers of two adjacent first target holes 52 in each row is equal. The distance between the centers of two adjacent first target holes 52 in each column is also equal.

[0224] Furthermore, in the second array, the distance between the centers of two adjacent second target holes 53 in each row is equal. The distance between the centers of two adjacent second target holes 53 in each column is also equal.

[0225] Furthermore, the first target hole 52 and the second target hole 53 are round holes, and the back openings of both the first target hole 52 and the second target hole 53 are chamfered. In this embodiment, the calibration plate 5 is 3mm thick and made of aluminum alloy.

[0226] A specific embodiment of a calibration plate 5 is as follows: the calibration plate body 51 is machined with six first target holes 52 (the first target holes 52 are large round holes with a diameter of 100mm) and six second target holes 53 (the second target holes 53 are small round holes with a diameter of 10mm). The back opening of each hole has a chamfer of 1.5x60°mm. After the machining is completed, the front side of the calibration plate body 51, i.e. the outward side, is treated with a matte white frosted finish, and the back side of the calibration plate body 51, i.e. the inward side, is treated with a glossy black high-reflection finish.

[0227] In this embodiment, the six first target holes 52 are arranged in two rows and three columns in a first array. The distance between the centers of two adjacent first target holes 52 in each row of the first array is 200 mm. The distance between the centers of two adjacent first target holes 52 in each column of the first array is also 200 mm.

[0228] In this embodiment, the six second target holes 53 are arranged in a second array of two rows and three columns. The distance between the centers of two adjacent second target holes 53 in each row of the second array is 20 mm. The distance between the centers of two adjacent first target holes 52 in each column of the second array is also 20 mm.

[0229] In this embodiment, the center of the first array and the center of the second array are the same as the center of the calibration board 5.

[0230] Example 4

[0231] See Figures 8 to 14This invention provides an infrared dual-light fusion testing device, including a heat source module and a calibration plate 5, wherein the heat source module is used to provide a heat source for the calibration plate 5.

[0232] The calibration process in Example 1 can be performed using calibration board 5.

[0233] Each set of target holes corresponds to a distance range, obtaining the distance between the infrared thermal imaging device and the calibration plate 5. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with multiple set distance ranges to determine which distance range the distance between the infrared thermal imaging device and the calibration plate 5 falls within. When the distance between the infrared thermal imaging device and the calibration plate 5 falls within a certain distance range, the set of target holes corresponding to that distance range is used for calibration (during calibration, the center of the target hole circle of the calibration plate in the visible light image and infrared image at that calibration distance is extracted and saved, and then calibration is performed).

[0234] When two sets of target holes are set on the calibration plate 5, namely the first set of target holes (a set of large circular holes) and the second set of target holes (a set of small circular holes), the infrared thermal imaging device acquires paired visible light images and infrared images. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with a set first distance (e.g., 2 meters). When the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to the first distance (e.g., 2 meters), the second target hole 53 (small circular hole) is used for calibration (i.e., the center of the second target hole of the calibration plate, i.e., the center of the small circle, is extracted from the visible light image and the infrared image at the calibration distance, and saved, and then calibration is performed). When the distance between the infrared thermal imaging device and the calibration plate 5 is greater than the first distance (e.g., 2 meters), the first target hole 52 (large circular hole) is used for calibration (i.e., the center of the first target hole of the calibration plate, i.e., the center of the large circle, is extracted from the visible light image and the infrared image at the calibration distance, and saved, and then calibration is performed). Generally, when the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to 2 meters, it is considered to be relatively close; when it is greater than 2 meters, it is considered to be relatively far.

[0235] Furthermore, the heat source module includes at least one heating unit, and each set of target holes corresponds to one or more heating units.

[0236] Furthermore, at least one set of target holes corresponds to multiple heating units.

[0237] Furthermore, at least one set of target holes corresponds to one heating unit.

[0238] Furthermore, when a set of target holes corresponds to multiple heating units, each target hole in that set of target holes corresponds to one heating unit.

[0239] In this embodiment, the calibration plate 5 is provided with two sets of target holes, namely the first set of target holes and the second set of target holes. The first set of target holes includes a plurality of first target holes 52 (large round holes), and the second set of target holes includes a plurality of second target holes 53 (small round holes).

[0240] Furthermore, each first target hole 52 of the calibration plate 5 corresponds to one heating unit (i.e., the first heating unit), and all second target holes 53 of the calibration plate 5 correspond to one heating unit (i.e., the second heating unit). The sizes of the first heating unit and the second heating unit may differ as needed. In this embodiment, the size of the first heating unit is larger than the size of the second heating unit.

[0241] This embodiment has a total of seven heating units, including six first heating units and one second heating unit. The six first heating units correspond one-to-one with the six first target holes 52 on the calibration plate 5 and are arranged opposite each other. Each first heating unit provides a heat source to the corresponding first target hole 52 on the calibration plate 5. The second heating unit corresponds to the six second target holes 53 on the calibration plate 5 and provides a heat source to the six second target holes 53 on the calibration plate 5 through the second heating unit.

[0242] Furthermore, the heating unit includes a heat source 21, in which a heating element 22 and a temperature detection element 23 are installed. The heating element 22 is connected to the output terminal of the controller, and the temperature detection element 23 is connected to the input terminal of the controller. The controller is connected to an instruction input unit and / or a display unit.

[0243] Each heating unit is controlled independently.

[0244] Furthermore, the heating element 22 can be a heating rod, and the temperature sensing element 23 can be a thermocouple.

[0245] In this embodiment, the heat source 21 of the first heating unit uses two heating rods and one thermocouple.

[0246] In this embodiment, the heat source 21 of the second heating unit uses a heating rod and a thermocouple.

[0247] In this embodiment, the heat source 21 is a rectangular block. The heat source 21 is provided with a first mounting hole 211 for mounting a heating rod and a second mounting hole 212 for mounting a thermocouple. The lead-out ends of the heating rod and the thermocouple extend out of the heat source 21.

[0248] The rectangular block of the heat source 21 is made of a material with good thermal conductivity, such as aluminum in this embodiment. The surface of the heat source 21, i.e., the aluminum plate, is treated with a glossy black high-reflectivity finish to improve image quality.

[0249] Furthermore, the heating unit is connected to the heat source fixing plate 3 via a first heat insulation plate 24, and the heat source fixing plate 3 is installed on the back of the calibration plate 5. In this embodiment, the first heat insulation plate 24 is made of mica fiber board, which serves as heat insulation. The heat source body 21 is fixed to the heat source fixing plate 3 by the first heat insulation plate 24 and does not come into contact with the heat source fixing plate 3.

[0250] The heat source fixing plate 3 is provided with mounting holes for installing the heating unit. In this embodiment, the heat source body 21 is installed in the mounting holes of the heat source fixing plate 3.

[0251] A second heat insulation plate 4 is provided between the calibration plate 5 and the heat source fixing plate 3. The heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence. The second heat insulation plate 4 is provided with a through hole for the heat source of the heating unit to pass through.

[0252] In this embodiment, the second heat insulation board 4 is made of mica fiber board, which serves as a heat insulation agent.

[0253] Furthermore, each heating unit is connected to the first heat insulation plate 24 in a one-to-one correspondence, such as by bolt fixing. The first heat insulation plate 24 is connected to the heat source fixing plate 3, such as by bolt fixing.

[0254] Furthermore, the second heat insulation plate 4 is provided with a plurality of first through holes for the heat source of the first heating unit to pass through. The first through holes on the second heat insulation plate 4 correspond one-to-one with the first target holes 52 on the calibration plate 5 and are arranged opposite each other. The diameter of the first through holes on the second heat insulation plate 4 is set as needed so that the heat source of the first heating unit can reach the corresponding first target hole 52 through the corresponding first through hole.

[0255] Furthermore, the second heat insulation plate 4 is provided with one or more second through holes for the heat source of the second heating unit to pass through. In this embodiment, the second heat insulation plate 4 is provided with a second through hole for the heat source of the second heating unit to pass through, and this second through hole is arranged opposite to all the second target holes 53 on the calibration plate 5.

[0256] The number and size of the second through holes are set as needed so that the heat source of the second heating unit can reach the second target hole 53 through the second through holes.

[0257] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected by bolts. Bolt holes are provided on the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 respectively.

[0258] In this embodiment, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are all rectangular, and their lengths and widths are equal. The solution of this invention is not limited to the above embodiment.

[0259] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected in sequence to form a heating target assembly, which is fixed on a chassis 1 to form a heating target box. In this embodiment, the entire heating target assembly is fixed to the sheet metal chassis 1 with screws.

[0260] The entire chassis 1 (e.g., a sheet metal chassis) is machined from 2mm steel plate and finished with black powder coating. The heating unit uses a heating rod for heating, with heat transferred to the heat source 21. A sheet metal door assembly 6 with a fan is installed on chassis 1; the fan is used for heat dissipation. The calibration plate 5 serves as a side panel of chassis 1, and the heat source fixing plate 3 is located inside chassis 1. A human-machine interface 8, a power switch 9, a start switch 10, and indicator lights 7 for displaying the heating status are installed on chassis 1. The human-machine interface 8 and indicator lights 7 are connected to a controller. The controller contains a control unit and a temperature control module. The control unit is connected to the temperature detection element 23, the temperature control module, the human-machine interface 8, and the indicator lights 7. The temperature control module is connected to the heating element 22, controlling the temperature of each heat source 21. The temperature of each heating unit is controlled through the human-machine interface 8, and each target can be individually controlled as required. Two handles are installed on chassis 1 for transporting the heating targets. Its working principle is that the heating rod heats the material and transfers the heat to the heat source 21. The thermocouple has a built-in temperature sensor to sense the real-time temperature of the heat source 21 and displays the actual sensed temperature through the human-machine interface 8. If the actual measurement requires the temperature to be adjusted to rise or fall, the temperature can be set through the human-machine interface 8. Each temperature can be controlled individually (adjustment range 25°~120°).

[0261] Furthermore, the heating target box is mounted on the frame 11. In this embodiment, the heating target box is fixed to the frame 11 with screws.

[0262] Furthermore, the lower end of the frame 11 is provided with casters 116.

[0263] The frame 11 includes a body frame 111, a fixing plate 112, a double sheet metal door 113, a stainless steel cover plate 114, and a frame base plate 115. The body frame 111 is made of stainless steel square tubing welded around its perimeter. The rear and side fixing plates 112 are made of stainless steel sheet metal and fixed to the frame 11 with screws. The front of the body frame 111 has a stainless steel double door, and the top stainless steel cover plate 114 is fixed to the frame 11 for fixing the heating target box. The base plate of the frame 11 is made of stainless steel sheet metal and is equipped with a caster wheel with feet at each corner for movement and fixation.

[0264] The calibration plate of this invention has target holes of various diameters, which can be adapted to tests at various distances (such as short distance and long distance, where distance refers to the distance between the infrared thermal imaging device and the calibration plate). It has strong versatility, can meet the optical axis testing needs of existing products, and can satisfy multiple uses of one machine.

[0265] Each target in this invention employs an individual temperature control system, enabling precise temperature control, resulting in highly accurate and stable targets, ensuring temperature consistency, and improving test accuracy.

[0266] This invention provides a two-stage accurate registration method for infrared and visible light images based on sparse features, involving the extraction and matching of sparse features, a two-stage accurate registration strategy, and a second registration correction method, to improve the accuracy and robustness of infrared and visible light image registration.

[0267] The method of this invention can automatically and in real time correct the registration deviation caused by affine transformation for each frame of dual-light image, presenting an accurate dual-light fusion result. At the same time, it has strong robustness, practicality and universality, and can be well applied in infrared and visible light dual-light fusion systems.

[0268] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for registering infrared and visible light images, characterized in that, Includes the following steps: Acquire visible light and infrared images, and use one image as a reference image and the other as the image to be registered; The first registered image is obtained by performing an affine transformation on the image to be registered using the affine transformation parameters obtained from the calibration. The reference image is registered with the first registered image to obtain the translation parameter offset of the first registered image relative to the reference image, specifically including: Discretize the reference image and the first registration image respectively, and use one of the feature point sets as the reference point set and the other feature point set as the registration point set; Register the reference point set with the point set to be registered, and calculate the translation parameter offset of the point set to be registered relative to the reference point set, including: (a) Initialization; (b) Calculate the covariance matrix between the reference point set and the point set to be registered; (c) Calculate the translation parameter offset of the point set to be registered relative to the reference point set based on the covariance matrix, including: Calculate the matching degree matrix P based on the covariance matrix, and the elements of the matching degree matrix P. ,in, , For noise level parameters, Let covariance matrix be the variance matrix. For the reference point set, The number of feature points in the reference point set. For the set of points to be registered, The number of feature points in the set of points to be registered; Calculate the mean of the reference point set based on the matching degree matrix. Mean of the set of points to be registered , , , in, It is the identity matrix of the corresponding dimension; The translation parameter offset of the point set to be registered relative to the reference point set is calculated based on the mean of the reference point set and the mean of the point set to be registered. The translation parameter offset is the difference between the mean of the reference point set and the mean of the point set to be registered. (d) Calculate the coordinates of the updated set of points to be registered and the set of reference points; (e) Update the covariance matrix; (f) Determine whether the stopping iteration condition is met based on the covariance matrix. If the stopping iteration condition is met, stop the iteration and output the currently calculated translation parameter offset; otherwise, return to step (c) and continue iterating until the stopping iteration condition is met. The translation parameters in the calibrated affine transformation parameters are corrected using the translation parameter offset to obtain the corrected affine transformation parameters.

2. The infrared and visible light image registration method as described in claim 1, characterized in that: It also includes the following steps: performing an affine transformation on the image to be registered using the corrected affine transformation parameters to obtain the registered image.

3. The method for registering infrared and visible light images as described in claim 1, characterized in that: Before discretizing the reference image and the first registration image, the following steps are also included: Feature extraction includes: extracting the boundaries of the reference image and the first registration image respectively to obtain a first boundary binary image and a second boundary binary image; Feature matching includes: matching the boundary features in the first boundary binary image and the second boundary binary image with each other, and deleting the boundary features in the first boundary binary image and the second boundary binary image that do not match each other.

4. The infrared and visible light image registration method as described in claim 3, characterized in that: The steps following the feature extraction step and before the feature matching step include the following steps: performing connected component analysis on the first boundary binary image and the second boundary binary image respectively, obtaining the area value of each connected region, and setting the gray value of pixels in connected regions with an area value less than a preset area threshold to 0.

5. The method for registering infrared and visible light images as described in claim 1, characterized in that: The limit is calculated based on the covariance matrix. The stopping condition for iteration is: the absolute value of the percentage reduction of the initial limit by the calculated limit is less than the set limit threshold.

6. The infrared and visible light image registration method as described in claim 5, characterized in that: The formula for calculating the limit error E is: ,in The sum of all elements of the matrix. Let N be the covariance matrix, and N be the number of feature points in the reference point set.

7. The method for registering infrared and visible light images as described in claim 1, characterized in that: Updating the covariance matrix specifically includes: updating the coordinates of the reference point set and the set to be registered based on the mean of the reference point set and the mean of the set to be registered; and updating the covariance matrix based on the updated coordinates of the reference point set and the set to be registered.

8. The method for registering infrared and visible light images as described in claim 7, characterized in that: Updated point set coordinates for: ; For the reference point set, For the set of points to be registered, for transpose, for Transpose of; Updated covariance matrix ,in, The trace of the matrix, These are the diagonal elements of the matrix. For the corresponding dimension, for The transpose of .

9. A registration device for infrared and visible light images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the infrared and visible light image registration method as described in any one of claims 1 to 8.

10. A method for fusing infrared and visible light images, characterized in that, Includes the following steps: Acquire visible light and infrared images, and use one image as a reference image and the other as the image to be registered; The corrected affine transformation parameters are obtained by using the registration method described in any one of claims 1 to 8; The affine transformation parameters are used to perform an affine transformation on the image to be registered. The transformed registered image is then fused with the reference image to obtain a fused image.

Citation Information

Patent Citations

  • Method and system for extracting field image edges of plant leaves

    CN101901477A

  • Dual-light imaging system image registration method and related device

    CN111311658A