Infrared and visible image registration method, fusion method and registration device

Abstract

Disclosed in the present invention are an infrared and visible image registration method, fusion method and registration device. The registration method comprises the following steps: acquiring a visible image and an infrared image, using one of the images as a reference image, and using the other image as an image to be registered; using calibrated affine transformation parameters to perform affine transformation on the image to be registered, so as to obtain a first registered image; and registering the reference image and the first registered image, so as to obtain the translation parameter offset of the first registered image with respect to the reference image, and using the translation parameter offset to correct a translation parameter among the calibrated affine transformation parameters, so as to obtain corrected affine transformation parameters. On the basis of target calibration-based affine registration, the present invention designs a two-stage precise registration policy, can automatically correct registration deviations of affine transformation in real time, has characteristics such as high robustness, practicability, and universality, and can be well applied to infrared and visible dual-optical fusion systems.

Description

A registration method, fusion method and registration device for infrared and visible light images Technical Field

[0001] The present invention belongs to the technical field of image processing technology, and particularly relates to an infrared and visible light image registration method, a fusion method and a registration device. Background Art

[0002] In existing infrared and visible light dual-light fusion systems, the mainstream method uses affine registration based on target calibration. This method pre-calibrates target data at different distances for each device to generate corresponding affine registration parameters. This approach achieves low algorithm complexity, reduced time consumption, strong robustness, and excellent practicality. However, the registration accuracy of this method is extremely dependent on the optical design of the dual-light system and the accuracy of the Hall effect distance. Even the slightest error in these two factors can lead to significant deviations in the visible light features in the fused image.

[0003] Currently, researchers have studied methods that can directly register infrared and visible light images without data calibration, such as region-based methods (correlation, mutual information, Fourier transform) and feature-based methods (points, lines, curves, and surfaces). However, these methods can only successfully register in a few specific scenarios and are extremely time-consuming, making them difficult to apply to specific infrared and visible light fusion systems.

[0004] Therefore, existing technical problems include at least: 1) the affine registration method based on target calibration will have serious deviations in some scenes; 2) the method that does not require data calibration has highly complex algorithms and weak robustness, and has no practical application value.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to overcome at least one defect in the prior art and provide an infrared and visible light image registration method, a fusion method and a registration device. A two-stage precise registration strategy is designed based on the affine registration method of target calibration. The two-stage precise registration strategy can automatically correct the registration deviation caused by the affine transformation in real time. It also has strong robustness, practicality, universality and other characteristics, and can be well applied in infrared and visible light dual-light fusion systems.

[0007] The technical solution of the present invention is implemented as follows: The present invention discloses a method for registering infrared and visible light images, comprising the following steps:

[0008] Obtain a visible light image and an infrared image, and use one of the images as a reference image and the other as an image to be registered;

[0009] Performing affine transformation on the image to be registered using the affine transformation parameters obtained by calibration to obtain a first registered image;

[0010] The reference image is registered with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image, and the translation parameter in the calibrated affine transformation parameters is corrected using the translation parameter offset to obtain corrected affine transformation parameters.

[0011] Furthermore, the infrared and visible light image registration method of the present invention further includes the following steps: performing affine transformation on the image to be registered using the corrected affine transformation parameters to obtain a registered image.

[0012] Furthermore, registering the reference image with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image specifically includes:

[0013] Discretize the reference image and the first registered image respectively, use one of the obtained feature point sets as the reference point set, and the other feature point set as the point set to be registered;

[0014] The reference point set is registered with the point set to be registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated.

[0015] Furthermore, before discretizing the reference image and the first registered image, the following steps are also included:

[0016] Feature extraction includes: performing boundary extraction on the reference image and the first registered image respectively to obtain a first boundary binary image and a second boundary binary image;

[0017] The feature matching includes: matching boundary features in the first boundary binary image and the second boundary binary image, and deleting boundary features in the first boundary binary image and the second boundary binary image that do not match each other.

[0018] Furthermore, feature matching specifically includes: identifying and deleting the second boundary features that do not match each other in the second boundary binary image based on the first boundary features in the first boundary binary image, and identifying and deleting the first boundary features that do not match each other in the first boundary binary image based on the second boundary features in the second boundary binary image.

[0019] Furthermore, identifying and deleting the second boundary features that do not match each other in the second boundary binary image based on the first boundary features in the first boundary binary image specifically includes:

[0020] Traverse the second boundary binary image and find the pixel position greater than 0;

[0021] Find the same pixel position in the first boundary binary image, and search the neighborhood at that position with a set search radius to see if there is a pixel with a grayscale value greater than 0. If so, keep the grayscale value of the pixel in the second boundary binary image unchanged; if not, set the grayscale value of the pixel in the second boundary binary image to 0;

[0022] Repeat the above steps until all pixels of the second boundary binary image are traversed.

[0023] Furthermore, identifying and deleting the first boundary features that do not match each other in the first boundary binary image based on the second boundary features in the second boundary binary image specifically includes:

[0024] Traverse the first boundary binary image and find the pixel position with gray value greater than 0;

[0025] Find the same pixel position in the second boundary binary image, and search the neighborhood at that position with the set search radius to see if there is a pixel with a grayscale value greater than 0. If so, keep the grayscale value of the pixel in the first boundary binary image unchanged; if not, set the grayscale value of the pixel in the first boundary binary image to 0;

[0026] Repeat the above steps until all pixels of the first boundary binary image are traversed.

[0027] Furthermore, an image preprocessing step is also included before the feature matching step, and the image preprocessing step includes:

[0028] Performing scaling processing on the reference image and the first registered image respectively;

[0029] or / and,

[0030] Delete the edge parts around the reference image and the first registered image according to the set ratio;

[0031] or / and,

[0032] Convert the reference image and the first registered image into grayscale images;

[0033] or / and,

[0034] Image smoothing is performed on the reference image and the first registered image respectively.

[0035] Furthermore, after the feature extraction step and before the feature matching step, the following steps are also included: performing connected domain analysis on the first boundary binary image and the second boundary binary image respectively to obtain the area value of each connected area, and setting the pixel grayscale values ​​of the connected area whose area value is less than the preset area threshold to 0.

[0036] Furthermore, after the reference image and the first registered image are discretized, a downsampling step is further included. The downsampling step includes: downsampling the reference point set and the to-be-registered point set obtained after the discretization process.

[0037] Furthermore, the reference point set is registered with the point set to be registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated, specifically including:

[0038] (a) Initialization;

[0039] (b) Calculate the covariance matrix between the reference point set and the point set to be registered;

[0040] (c) calculating the translation parameter offset of the point set to be registered relative to the reference point set based on the covariance matrix;

[0041] (d) calculating the coordinates of the updated set of points to be registered and the set of reference points;

[0042] (e) Update the covariance matrix;

[0043] (f) Determine whether the stop iteration condition is met based on the covariance matrix. If the stop iteration condition is met, stop the iteration and output the currently calculated translation parameter offset; otherwise, return to step (c) and continue iterating until the stop iteration condition is met.

[0044] Furthermore, the translation parameter offset of the to-be-registered point set relative to the reference point set is iteratively solved according to the covariance matrix, specifically including: calculating the matching matrix according to the covariance matrix;

[0045] Calculate the mean of the reference point set and the mean of the point set to be registered according to the matching matrix;

[0046] The translation parameter offset of the to-be-registered point set relative to the reference point set is calculated based on the mean of the reference point set and the mean of the to-be-registered point set.

[0047] Furthermore, the translation parameter offset of the point set to be registered relative to the reference point set is iteratively solved according to the covariance matrix, specifically including:

[0048] Calculate the matching matrix P, where each element p mn The calculation is as follows: in, w is the noise level parameter, σ 2 is the covariance matrix, ir={x_ir n ,y_ir n} is the reference point set, N is the number of feature points in the reference point set, vis={x_vis n ,y_vis n} is the point set to be registered, M is the number of feature points in the point set to be registered;

[0049] Calculate the mean μ of the reference point set ir , the formula is: Calculate the mean μ of the set of points to be registered vis , the formula is: Among them, 1 is the unit matrix of the corresponding dimension;

[0050] Calculate the translation parameter offset t = {e', f'}, t = μ of the point set to be registered relative to the reference point set ir -μ vis .

[0051] Furthermore, the coordinates of the to-be-registered point set and the reference point set are updated according to the mean of the reference point set and the mean of the to-be-registered point set; and the covariance matrix is ​​updated according to the updated coordinates of the to-be-registered point set and the reference point set.

[0052] Further, calculate the updated point set coordinates irNew, visNew: irNew = ir-μ ir T ,visNew=vis-μ vis T , ir={x_ir n ,y_ir n} is the reference point set, vis={x_vis n ,y_vis n} is the point set to be registered;

[0053] Update the covariance matrix Where tr() is the trace of the matrix, d() is the diagonal element of the matrix, and 1 is the identity matrix of the corresponding dimension;

[0054] Furthermore, the tolerance is calculated according to the covariance matrix, and the iteration stopping condition is: the absolute value of the percentage by which the set initial tolerance is reduced compared to the calculated tolerance is less than a set tolerance threshold.

[0055] Furthermore, the calculation formula of the limit error E is: E=N*sum(logσ 2 ), where sum() is the sum of all elements of the matrix, σ 2 is the covariance matrix, and N is the number of feature points in the reference point set.

[0056] Furthermore, the corrected horizontal translation parameter is the sum of e and e', and the corrected vertical translation parameter is the sum of f and f', where e is the calibrated horizontal translation parameter, e' is the horizontal translation parameter offset, f is the calibrated vertical translation parameter, and f' is the vertical translation parameter offset.

[0057] The present invention also discloses a device for registering infrared and visible light images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for registering infrared and visible light images are implemented.

[0058] The present invention also discloses a method for fusing infrared and visible light images, comprising the following steps:

[0059] Obtain a visible light image and an infrared image, and use one of the images as a reference image and the other as an image to be registered;

[0060] The modified affine transformation parameters are obtained by using the registration method described above;

[0061] The modified affine transformation parameters are used to perform affine transformation on the image to be registered, and the transformed registered image is fused with the reference image to obtain a fused image.

[0062] The present invention has at least the following beneficial effects: the present invention provides a two-stage precise registration method for infrared and visible light images based on sparse features. For each frame of dual-light image, the registration deviation caused by affine transformation can be automatically corrected in real time, presenting accurate dual-light fusion results. At the same time, it has strong robustness, practicality, universality and other characteristics, and can be well applied in infrared and visible light dual-light fusion systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] FIG1 is a flow chart of a method for registering infrared and visible light images according to an embodiment of the present invention;

[0065] FIG2 is a flowchart of the second stage registration provided by an embodiment of the present invention;

[0066] FIG3 is a flow chart of feature matching provided by an embodiment of the present invention;

[0067] FIG4 is an original visible light image provided by an embodiment;

[0068] FIG5 is an original infrared image provided by an embodiment;

[0069] FIG6 is an image obtained by fusing an original infrared image with a visible light image after first-stage registration, provided by an embodiment;

[0070] FIG7 is an image obtained by fusing an original infrared image with a visible light image after second-stage registration, provided by an embodiment;

[0071] FIG8 is a schematic structural diagram of a calibration plate provided in an embodiment of the present invention;

[0072] FIG9 is a cross-sectional view of a calibration plate provided in an embodiment of the present invention;

[0073] FIG10 is an enlarged view of portion P of FIG9 ;

[0074] FIG11 is a schematic structural diagram of an infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0075] FIG12 is an exploded view of a heating target box of an infrared dual-light fusion test device provided by an embodiment of the present invention;

[0076] FIG13 is a schematic structural diagram of a heating unit of an infrared dual-light fusion testing device provided in an embodiment of the present invention;

[0077] FIG14 is an exploded view of a frame of the infrared dual-light fusion testing device provided in an embodiment of the present invention.

[0078] In the accompanying drawings, 1 is a chassis, 2 is a heating unit, 21 is a heat source body, 211 is a first mounting hole, 212 is a second mounting hole, 22 is a heating element, 23 is a temperature detection element, 24 is a first heat insulation board, 3 is a heat source fixing plate, 4 is a second heat insulation board, 5 is a calibration board, 51 is a calibration board body, 52 is a first target hole, 53 is a second target hole, 6 is a sheet metal door assembly, 7 is an indicator light, 8 is a human-machine interface, 9 is a power switch, 10 is a start switch, 11 is a frame, 111 is a body frame, 112 is a fixing plate, 113 is a double-opening sheet metal door, 114 is a stainless steel cover, 115 is a frame bottom plate, and 116 is a caster. DETAILED DESCRIPTION

[0079] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0080] Example 1

[0081] 1 and 2 , an embodiment of the present invention discloses a method for registering infrared and visible light images, comprising the following steps:

[0082] Acquire an original visible light image and an original infrared image, and use one of the images as a reference image and the other as an image to be registered. For example, in this embodiment, the original infrared image is used as the reference image and the original visible light image is used as the image to be registered.

[0083] First stage registration: obtain the current target distance, and obtain the corresponding affine transformation parameters according to the current target distance. The affine transformation parameters are obtained through calibration. The affine transformation parameters obtained by calibration are used to perform affine transformation on the image to be registered to obtain the first registered image;

[0084] Second stage registration: registering the reference image with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image, and using the translation parameter offset to correct the translation parameters in the affine transformation parameters obtained by calibration to obtain corrected affine transformation parameters;

[0085] The corrected affine transformation parameters are used to perform affine transformation on the image to be registered to obtain the final registered image.

[0086] The first stage of registration mainly relies on affine registration based on target calibration to roughly align the input original visible light image with the infrared image. At this time, the scaling and rotation parameters are accurate, but there will be offset errors in the translation parameters.

[0087] To address the problem that exists after the first stage of registration (after the first stage of registration, the scaling and rotation parameters are accurate, but there will be offset errors in the translation parameters), the present invention performs a second stage of precise registration based on the first stage of registration to accurately correct the translation parameters in the affine transformation, so as to complete the precise registration of the infrared image and the visible light image. The calculation formula of the affine transformation is as follows:

[0088] In the above formula, x and y represent the pixel positions of the input visible light image, They represent the pixel positions of the registered visible light image. a, b, c, and d are the rotation and scaling parameters used in the affine transformation, and e and f are the translation and rotation parameters used in the affine transformation. The rotation, scaling, and translation parameters for the first stage of the affine transformation are generated by pre-calibrating each device with target data at different distances. During use, the corresponding distance is found based on the Hall effect distance returned by the device.

[0089] In some embodiments, the affine transformation parameters obtained by calibration specifically include:

[0090] Acquire paired visible light images and infrared images at each set calibration distance;

[0091] Extract and save the centers of the calibration plates in the visible light image and infrared image at each calibration distance respectively. Calculate the affine matrix at each calibration distance by aligning the centers of the visible light image and infrared image at each calibration distance.

[0092] The affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix at each calibration distance.

[0093] Furthermore, the affine matrix of the visible light image corresponding to the infrared image at any distance is calculated based on the affine matrix of the visible light image corresponding to the infrared image at each calibration distance, specifically including:

[0094] The average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances is obtained, where the affine matrix contains the X-direction scaling component a, the X-direction rotation component b, the Y-direction rotation component c, the Y-direction scaling component d, and the X-direction offset component e and the Y-direction offset component f.

[0095] Using the center of the calibration plate in the visible light image and infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances, recalculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance (if not recalculated, the offset component cannot be increased or decreased regularly), and obtain the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance;

[0096] Calculate the X-direction offset component and the Y-direction offset component of the affine matrix at any distance based on the corrected X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance;

[0097] The affine matrix at any distance can be generated based on the X-direction offset component and Y-direction offset component of the affine matrix at each set distance, as well as the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibrated distances. That is, the X-direction offset component and Y-direction offset component of the affine matrix at each set distance are assigned to the affine average matrix, or the X-direction offset component and Y-direction offset component of the affine matrix at each set distance, as well as the average value of the X- and Y-direction rotation and scaling components of the affine matrix at all calibrated distances, are assigned to the affine matrix to generate the affine matrix at any distance.

[0098] The average of the affine matrices at all calibration distances is calculated to obtain the affine average matrix.

[0099] Furthermore, the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances (that is, the mean value of the rotation and scaling matrices in the affine matrix at each calibration distance) are used to inversely calculate the X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance.

[0100] Furthermore, the nearest neighbor distance method is used to calculate the X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions of the affine matrix at all calibration distances.

[0101] The principle of the nearest neighbor distance method of the present invention is as follows: at a single distance, the coordinates of each circle center in the visible cursor calibration image are cyclically brought in, and a set of offset vectors in the X direction and the offset vector in the Y direction corresponding to each circle center coordinate can be calculated through the coordinates of the circle center in the corresponding infrared calibration image. For example, the first circle center after sorting in the visible cursor calibration image corresponds to the first circle center after sorting in the infrared calibration image, and the corresponding offset vector in the X direction and the offset vector in the Y direction are calculated, and recorded as an, bn. Then, by traversing the infrared calibration image center corresponding to all circle centers at this distance, the center of the visible cursor calibration image is converted to the center of the circle on the infrared calibration image, and the straight-line distance dis with the center extracted from the infrared calibration image is calculated; for example, 25 circle centers are extracted from the infrared calibration image and the visible cursor calibration image respectively, and the cycle is repeated 25 times, and the minimum straight-line distance is accumulated to obtain a set of an, bn, which are recorded as the offset vector in the X direction and the offset vector in the Y direction under the current distance.

[0102] The X-direction offset component and the Y-direction offset component of the affine matrix at each calibration distance are calculated by using the center of the calibration plate in the visible light image and the infrared image at each calibration distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances using the nearest neighbor distance method. Specifically, at a single calibration distance, the center of the calibration plate in the visible light image and the infrared image at the calibration distance and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances are obtained. For each pair of corresponding visible light calibration image centers and infrared calibration image centers, the X-direction offset component and the Y-direction offset component and the average value of the rotation and scaling components in the X and Y directions in the affine matrix at all calibration distances are first obtained to form a new affine matrix (a pair of corresponding visible light calibration image centers and infrared calibration image centers). The center of the circle corresponds to a new affine matrix), and each new affine matrix is ​​used to calculate the coordinates of the center of each visible light calibration image converted to the center of the infrared calibration image, which are marked as the second center of the circle on the infrared calibration image. The offset vector in the X direction and the offset vector in the Y direction as well as the straight-line distance dis between the center of the circle and the first center coordinate in the corresponding infrared calibration image (that is, the center of the circle in the extracted infrared calibration image) are calculated. The straight-line distances dis corresponding to all the center of the circle are summed to obtain the cumulative straight-line distance values ​​corresponding to each new affine matrix. A new affine matrix corresponds to a set of offset vectors in the X direction and the offset vector in the Y direction, denoted as an and bn. The set of offset vectors in the X direction and the offset vector in the Y direction corresponding to the minimum value of the cumulative straight-line distance are used as the offset vectors in the X direction and the offset vectors in the Y direction at the current calibration distance.

[0103] Furthermore, based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated, specifically including: based on the corrected X-direction offset component and Y-direction offset component of the affine matrix at each calibration distance, the X-direction offset component and Y-direction offset component of the affine matrix at any distance are calculated using an interpolation method.

[0104] A specific embodiment of the calibration process is:

[0105] Acquire 14-bit / 16-bit high dynamic range infrared images IR and 8-bit visible light images VIS.

[0106] Identify the center of the calibration plate on the 14-bit / 16-bit high dynamic range infrared image IR and the 8-bit visible light image VIS, store them as ir_centerCiclePoints and vis_centerCiclePoints respectively, and sort them.

[0107] According to vis_centerCiclePoints and ir_centerCiclePoints, the affine matrix VISToIRMatrixs is obtained.

[0108] AffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints,VISToIRMatrixs);

[0109] The above process is used to perform calibration at the set calibration distance. For example, this embodiment has 9 calibration distances, ranging from 1 meter to 9 meters, a total of 9 groups. Of course, the number of calibration distances is not limited to 9 and can be adjusted as needed. All the extracted circle centers vis_centerCiclePointList and ir_centerCiclePointList are saved and the visible light image center and the infrared image center are matched one by one, thereby generating the affine matrix VISToIRMatrixs at each distance.

[0110] Take the average of the rotation and scaling components of the affine matrix under the 9 sets of data (i.e. a, b, c, d) and save it as VISToIRMatrix_average, which is expressed as:

[0111] In the affine matrix, a represents the scaling ratio in the X direction, b represents the rotation ratio in the X direction, c represents the rotation ratio in the Y direction, d represents the scaling ratio in the Y direction, e represents the translation distance in the X direction, and f represents the translation distance in the Y direction. In a two-dimensional image, there are different differences between the X direction vector and the Y direction vector in the affine transformation.

[0112] For example, in this embodiment, at a distance of 1 meter, a=2.71771, b=0.0005355351, e=-1027.16, c=-0.00148628, d=2.77923, f=-1125.53.

[0113] The study found that the X-direction offset distance changes more at close distances, while the change is almost negligible at longer distances.

[0114] Using the saved vis_centerCiclePointList and ir_centerCiclePointList, and using the average affine matrix of VISToIRMatrix_average, the nearest neighbor offset distance method is used to calculate the X-direction component e and the Y-direction component f at each distance.

[0115] ReverseCalAffineTransTable(vis_centerCiclePoints,ir_centerCiclePoints);

[0116] Interpolate the e and f values ​​obtained from each set of calculations, and calculate the e and f values ​​at each distance from 1 to 20 meters in units of 10 centimeters. The x and y values ​​range from 1 to 9 meters, and the range is 1 to 20 meters with an interval of 0.1 meters. The output is a lookup table of the X-direction offset vector and the Y-direction offset vector.

[0117] Interp1HermitePchip(distance,x,range,e);

[0118] Interp1HermitePchip(distance,y,range,f);

[0119] Finally, the target distance is obtained according to the Hall contrast table. The e and f values ​​at the corresponding distance are obtained by looking up the table according to the target distance, and assigned to VISToIRMatrix_average to obtain the affine matrix at the distance.

[0120] Furthermore, registering the reference image with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image specifically includes:

[0121] Discretize the reference image and the first registered image respectively, use one of the obtained feature point sets as the reference point set, and the other feature point set as the point set to be registered;

[0122] The reference point set is registered with the point set to be registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated.

[0123] In some embodiments, the reference point set is obtained by discretizing the reference image, and the to-be-registered point set is obtained by discretizing the first registered image.

[0124] Furthermore, before discretizing the reference image and the first registered image, the following steps are also included:

[0125] Feature extraction includes: performing boundary extraction on the reference image and the first registered image respectively to obtain a first boundary binary image and a second boundary binary image;

[0126] The feature matching includes: matching boundary features in the first boundary binary image and the second boundary binary image, and deleting boundary features in the first boundary binary image and the second boundary binary image that do not match each other.

[0127] Furthermore, an image preprocessing step is included before the feature matching step, and the image preprocessing step includes: performing scaling processing on the reference image and the first registration image respectively.

[0128] After the image to be registered (such as the original visible light image) undergoes the first stage of registration, the first registered image of the first stage of registration (i.e., the visible light image of the first stage of registration) is obtained. At this time, a series of preprocessing of the infrared and visible light images begins to provide boundary feature point support for more efficient and accurate second stage registration.

[0129] One implementation involves simultaneously scaling the visible light image and the original infrared image from the first stage of registration. This step aims to downsample the image data at the image level, reducing the algorithm's time complexity. Specifically, the scaling factor depends on the image resolution and is determined based on specific testing. The goal is to achieve the highest possible scaling factor without significantly impacting registration accuracy. For example, a scaling factor of 8 is recommended for a 1280*1024 resolution image.

[0130] The geometric features required for image registration are usually located in the center of the image. Therefore, a certain percentage of the image edges are removed from the reduced image to further reduce the amount of image layer data. For the same purpose, the image is converted to grayscale to reduce the number of unnecessary channels involved in subsequent calculations.

[0131] In some embodiments, the image preprocessing step further includes: deleting edge portions around the reference image and the first registered image according to a set ratio, and converting the reference image and the first registered image into grayscale images.

[0132] Since the image itself contains a lot of texture information, but the registration does not care about texture information, it only focuses on the main contour information. The present invention needs to perform boundary extraction on the image later. If the canny boundary extraction is performed directly on the original image, these unnecessary texture information will also produce a lot of messy edges, which will not only increase the amount of calculation, but also cause data errors and affect the registration accuracy. Therefore, the image needs to be smoothed before the boundary extraction step. It can not only smooth the texture area well, but also enhance the boundary, so that the subsequent boundary extraction can obtain more boundary information.

[0133] In some embodiments, the image preprocessing step further includes: performing image smoothing on the reference image and the first registered image respectively.

[0134] Furthermore, image smoothing is performed on the infrared image and the visible light image respectively, specifically including: performing sparse smoothing on the infrared image and the visible light image respectively. Preferably, L0 sparse smoothing is performed on the infrared image and the visible light image respectively.

[0135] Performing L0 sparse smoothing on infrared and visible grayscale images can smooth out small details inside the image while strengthening the boundary, providing high-dimensional and sparse boundary information for the next step of boundary extraction. The following example illustrates L0 sparse smoothing on infrared grayscale images. The specific steps are as follows: First, establish the L0 sparse smoothing model:

[0136] in, Where: irg is the input infrared grayscale image, irg L0 is the image after L0 sparse smoothing, and p represents a single pixel in the image. #{·} is a counting operator that counts the number of pixels that meet the conditions. λ>0 is a non-negative regularization coefficient, are the forward difference operators in the x-direction and y-direction respectively. Then, the variable partitioning method is used to transform the L0 sparse smooth model into a decomposable variable solution form for equation (2), and the sub-problems are iteratively solved to obtain the optimal solution irg of equation (2) L0 .

[0137] Furthermore, feature extraction specifically includes: performing canny boundary extraction on the reference image and the first registered image respectively to obtain a first boundary binary image and a second boundary binary image.

[0138] Through processing with the L0 sparse smoothing model, minute details and textures in infrared and visible grayscale images are smoothed and strong boundaries are enhanced, preserving high-dimensional sparse feature information for boundary extraction, enabling the extraction of more effective and significant high-dimensional boundary features. Boundary extraction primarily involves vertical and horizontal gradient calculation, vertical and horizontal gradient smoothing, gradient amplitude calculation, adaptive threshold selection, and non-maximum suppression. Compared to the OpenCV version of the Canny function, this method uses double precision instead of integer data and changes the threshold selection from manual input to automatic proportional selection based on the image's gradient histogram statistics. This allows the extraction of boundary features in low-contrast images that cannot be extracted by the OpenCV version of Canny.

[0139] Through adaptive high-precision canny boundary extraction, a binary sparse boundary image of infrared and visible light is obtained. In the extracted boundary binary image, small boundaries can be considered as unimportant geometric information. Deleting them can reduce the amount of data while reducing unnecessary noise data. The length of the boundary is based on the connected domain analysis. For the binary boundary image pair (i.e., the first boundary binary image and the second boundary binary image), the first boundary binary image and the second boundary binary image are respectively subjected to connected domain analysis (such as the connected domain analysis function connectedComponentsWithStats of OpenCV can be used for connected domain analysis), and the area value of each connected region in the first boundary binary image and the second boundary binary image is obtained (i.e., the area length of the connected boundary is calculated). The present invention deletes the connected boundaries of small areas by setting the corresponding area threshold to reduce invalid registration information. Specifically: traverse the connected area below the preset area threshold and set the grayscale value of all pixels in the connected area to 0.

[0140] In some embodiments, 8-connected domain analysis is performed on the infrared and visible light binary images respectively.

[0141] Infrared and visible light are heterogeneous images, and the boundary binary images extracted from the two are quite different. The purpose of the feature matching step is to match similar boundary features and delete unmatched boundary features to reduce noise data that is not needed for registration.

[0142] Furthermore, feature matching specifically includes: a first step and a second step, wherein the first step includes: identifying and deleting the visible light boundary features that do not match each other in the visible light boundary binary image based on the infrared boundary features in the infrared boundary binary image,

[0143] The second step includes: identifying and deleting infrared boundary features that do not match each other in the infrared boundary binary image based on the visible light boundary features in the visible light boundary binary image.

[0144] Furthermore, identifying and deleting the second boundary features that do not match each other in the second boundary binary image based on the first boundary features in the first boundary binary image specifically includes:

[0145] Traverse the second boundary binary image and find the pixel position whose gray value is greater than 0;

[0146] Find the same pixel position in the first boundary binary image, and search (i.e., search) whether there is a pixel position with a grayscale value greater than 0 in the neighborhood of the position with a set search radius. If so, keep the grayscale value of the pixel in the second boundary binary image unchanged; if not, set the grayscale value of the pixel in the second boundary binary image to 0;

[0147] Repeat the above steps until all pixels of the second boundary binary image are traversed.

[0148] Furthermore, identifying and deleting the first boundary features that do not match each other in the first boundary binary image based on the second boundary features in the second boundary binary image specifically includes:

[0149] Traverse the first boundary binary image and find the pixel position with gray value greater than 0;

[0150] Find the same pixel position in the second boundary binary image, and search the neighborhood at that position with the set search radius to see if there is a pixel with a grayscale value greater than 0. If so, keep the grayscale value of the pixel in the first boundary binary image unchanged; if not, set the grayscale value of the pixel in the first boundary binary image to 0;

[0151] Repeat the above steps until all pixels of the first boundary binary image are traversed.

[0152] The order of execution of the first step and the second step can be swapped.

[0153] Referring to Figure 3, a specific embodiment of feature matching is as follows: a search radius is set. First, pixels greater than 0 are traversed in the visible light boundary binary image. The same location is found in the infrared boundary binary image. The neighborhood is then searched using the set search radius to see if there are any pixels with a grayscale value of 255. If a pixel is found, the grayscale value of the pixel in the visible light boundary binary image remains unchanged; if not, the grayscale value is set to 0. In this way, after traversing all visible light boundary binary images, a matching post-deletion visible light image is obtained. Next, following the same logic, all locations in the infrared boundary binary image are searched, using the matching post-deletion visible light image as the grayscale value retention standard for the infrared boundary binary image pixels to generate a matching post-deletion infrared boundary binary image.

[0154] Furthermore, the reference image and the first registered image are discretized, specifically including:

[0155] A coordinate system is established in the reference image and the first registration image respectively: the upper left corner of the image is the coordinate origin, the horizontal direction is the positive direction of the x-axis, and the vertical direction is the positive direction of the y-axis. A search is performed in the first boundary binary image and the second boundary binary image obtained after mutual matching and deletion, and the pixel positions with grayscale values ​​not equal to 0 are respectively used to generate the first feature point set and the second feature point set (i.e., the infrared feature point set and the visible light feature point set) according to the above coordinate system definition. One of the feature point sets (the boundary feature point set) is used as the reference point set, and the other feature point set (the boundary feature point set) is used as the point set to be registered, such as the infrared feature point set as the reference point set and the visible light feature point set as the point set to be registered.

[0156] Further, after the discretization of the reference image and the first registration image, a downsampling step is further included. The downsampling step includes: downsampling the reference point set and the point set to be registered obtained after the discretization process.

[0157] Point set downsampling is to reduce the number of points in the point set as much as possible without losing registration accuracy as much as possible, so as to reduce the time consumption of subsequent registration algorithms. The principle of the point set downsampling algorithm is to take a point at a certain interval in the horizontal and vertical directions respectively, and all other points in the middle are deleted. Before the algorithm is executed, three point number thresholds pn_threshold1, pn_threshold2, and pn_threshold3 are set respectively. Then, the reference point set (such as the number of infrared points) pn_ir and the point set to be registered (such as the number of visible light points) pn_vis are obtained respectively. The reference point set (such as the number of infrared points) selects the sampling interval according to the following rules for downsampling: (a) If pn_ir ≤ threshold1, sample points at an interval of 1; (b) If threshold1 < pn_ir ≤ threshold2, sample points at an interval of 2; (c) If threshold3 ≤ pn_ir, sample points at an interval of 3. Similarly, the point set to be registered (such as the number of visible light points) also selects the sampling interval according to the following rules for downsampling: (a) If pn_vis ≤ threshold1, sample points at an interval of 1; (b) If threshold1 < pn_vis ≤ threshold2, sample points at an interval of 2; (c) If threshold3 ≤ pn_vis, sample points at an interval of 3.

[0158] After the point set downsampling, the point set registration step is executed.

[0159] Since the rough registration has been completed in the first stage of registration, the scaling and rotation parameters are accurate, and there is only an offset error in the translation parameter. Therefore, only the translation parameter is considered in the second stage of registration. By completing the registration between the infrared and visible light point sets, an accurate correction of the translation parameter can be obtained. Therefore, the formula (1) of the affine transformation can be converted into the following form:

[0160] Here, x' and y' respectively represent the horizontal and vertical coordinate values of the points in the visible light point set after the first stage of registration. respectively represent the horizontal and vertical coordinate values of the points in the visible light point set after the second stage of registration, and e' and f' are the translation correction parameters obtained in the horizontal and vertical directions in the second stage of registration.

[0161] The registration scheme designed for the second stage of registration is an affine transformation based on a mixture Gaussian model that only considers translation. The specific steps are as follows:

[0162] (a) Algorithm initialization, such as translation parameters e'=0, f'=0, noise level parameter w=0.1, tolerance threshold tolE=0.00001, initial tolerance E old =1;

[0163] (b) Calculate the covariance matrix between the reference point set and the point set to be registered: Where ir={x_ir n ,y_ir n} is the original infrared point set, where the number of feature points is N. vis={x_vis n ,y_vis n} is the set of visible light points after the first stage of registration, where the number of feature points is M.

[0164] (c) calculating the translation parameter offset of the point set to be registered relative to the reference point set based on the covariance matrix;

[0165] (d) calculating the coordinates of the updated set of points to be registered and the set of reference points;

[0166] (e) Update the covariance matrix;

[0167] (f) Determine whether the stop iteration condition is met based on the covariance matrix of step (e). If the stop iteration condition is met, stop the iteration and output the currently calculated translation parameter offset t = {e', f'}; otherwise, return to step (c) and continue iterating until the stop iteration condition is met.

[0168] Furthermore, the translation parameter offset of the to-be-registered point set relative to the reference point set is iteratively solved according to the covariance matrix, specifically including: calculating the matching matrix according to the covariance matrix;

[0169] Calculate the mean of the reference point set and the mean of the point set to be registered according to the matching matrix;

[0170] The translation parameter offset of the to-be-registered point set relative to the reference point set is calculated based on the mean of the reference point set and the mean of the to-be-registered point set.

[0171] Furthermore, the translation parameter offset of the point set to be registered relative to the reference point set is iteratively solved according to the covariance matrix, specifically including:

[0172] Calculate the matching matrix P, where each element p mn The calculation is as follows: in, w is the noise level parameter, σ 2 is the covariance matrix, ir={x_ir n ,y_irn} is the reference point set, N is the number of feature points in the reference point set, vis={x_vis n ,y_vis n} is the point set to be registered, M is the number of feature points in the point set to be registered; mn is the element in the mth row and nth column of the matrix P.

[0173] Calculate the mean μ of the reference point set ir , the formula is: Calculate the mean μ of the set of points to be registered vis , the formula is: Among them, 1 is the unit matrix of the corresponding dimension, 1 T represents the transpose of the identity matrix 1, P T is the transpose of P, ir T is the transpose of ir, vis T is the transpose of vis;

[0174] Calculate the translation parameter offset t = {e', f'}, t = μ of the point set to be registered relative to the reference point set ir -μ vis .

[0175] Furthermore, the coordinates of the to-be-registered point set and the reference point set are updated according to the mean of the reference point set and the mean of the to-be-registered point set; and the covariance matrix is ​​updated according to the updated coordinates of the to-be-registered point set and the reference point set.

[0176] Further, calculate the updated point set coordinates irNew, visNew: irNew = ir-μ ir T ,visNew=vis-μ vis T , ir={x_ir n ,y_ir n} is the reference point set, vis={x_vis n ,y_vis n} is the point set to be registered, μ vis T μ vis The transpose of μ ir T μ ir The transpose of

[0177] Update the covariance matrix Where tr() is the trace of the matrix, d() is the diagonal element of the matrix, 1 is the identity matrix of the corresponding dimension, and irNew T The transpose of irNew.

[0178] Further, the limit error E is calculated based on the covariance matrix, and the iteration stop condition is: abs((E - E old ) / E) < tolE, where tolE is the limit error threshold, and E old is the initial limit error.

[0179] Further, the calculation formula for the limit error E is: E = N * sum(logσ 2 ), where sum() is the sum of all elements of the matrix, σ 2 is the covariance matrix, and N is the number of feature points in the reference point set.

[0180] Through the iterative solution of the second-stage registration, the offset t = {e', f'} of the visible light image relative to the infrared image can be obtained. Then, the affine transformation translation parameters are corrected using the translation parameter offset. After correction, the translation parameter in the horizontal direction is the sum of e and e', and the translation parameter in the vertical direction is the sum of f and f'. e is the translation parameter in the horizontal direction obtained by calibration, e' is the offset of the translation parameter in the horizontal direction, f is the translation parameter in the vertical direction obtained by calibration, and f' is the offset of the translation parameter in the vertical direction.

[0181] Input the offset t = {e', f'} into the affine transformation parameters of the first-stage registration, then Equation (1) becomes the following form

[0182] Perform affine registration according to the expression of Equation (4), and output the final registered image (such as the finally registered visible light image).

[0183] Based on the same inventive concept, the present invention also provides a registration device for infrared and visible light images, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned registration method for infrared and visible light images are implemented.

[0184] Embodiment 2

[0185] The embodiment of the present invention also discloses a fusion method for infrared and visible light images, including the following steps:

[0186] Obtain the original visible light image and the original infrared image, and use one of the images as the reference image and the other image as the image to be registered;

[0187] Adopt the registration method as described in Embodiment 1 to obtain the corrected affine transformation parameters;

[0188] Use the corrected affine transformation parameters to perform an affine transformation on the image to be registered, and fuse the transformed registered image with the reference image to obtain a fused image.

[0189] The present invention performs a second stage of precise registration based on the first stage of registration to precisely correct the translation parameters in the affine transformation and complete the precise registration of the infrared image and the visible light image.

[0190] Figure 4 shows an original visible light image provided by an embodiment; Figure 5 shows an original infrared image provided by an embodiment; if the visible light image that has completed the first stage of registration is fused with the original infrared image, the image shown in Figure 6 is obtained, and if the visible light image that has completed the second stage of registration is fused with the original infrared image, the image shown in Figure 7 is obtained.

[0191] Based on the same inventive concept, the present invention also provides a device for fusing infrared and visible light images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for fusing infrared and visible light images are implemented.

[0192] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is used to execute program code or process data stored in the memory, such as program code for a method for registering infrared and visible light images or a method for fusing infrared and visible light images.

[0193] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or internal memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash memory card, etc. equipped on the electronic device. Of course, the memory may also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory is generally used to store operating methods and various application software installed in the electronic device, such as program code for infrared and visible light image registration methods or infrared and visible light image fusion methods. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0194] Based on the same inventive concept, the present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the infrared and visible light image registration method or the infrared and visible light image fusion method as described above.

[0195] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0196] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0197] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0199] Example 3

[0200] 8 to 10 , an embodiment of the present invention discloses a calibration plate, including a calibration plate body 51 , on which at least two groups of target holes are provided, each group of target holes comprising a plurality of target holes, the target holes in the same group of target holes having the same aperture, and the target holes in different groups of target holes having different apertures.

[0201] Furthermore, each group of target wells corresponds to a region one by one, the area of ​​the region corresponding to each group of target wells is different, and all target wells in each group of target wells must be located in the corresponding region.

[0202] Furthermore, when the area of ​​the region corresponding to the first group of target wells is larger than the area of ​​the region corresponding to the second group of target wells, the target well diameter of the first group of target wells is larger than the target well diameter of the second group of target wells. The regions may overlap, for example, a larger region may completely cover a smaller region.

[0203] Furthermore, the area corresponding to each group of target wells is a circular area or a polygonal area (polygonal areas include triangular areas, quadrilateral areas, pentagonal areas, etc.). That is, the area corresponding to each group of target wells is one of a circular area, a triangular area, a quadrilateral area, a pentagonal area, etc. In this embodiment, the area corresponding to each group of target wells is a rectangular area.

[0204] Furthermore, the centers of the regions corresponding to the multiple groups of target holes are the same.

[0205] Furthermore, the areas corresponding to the multiple groups of target holes are the same as the center of the calibration plate 5 .

[0206] Furthermore, each group of target wells includes at least three target wells, and all target wells in each group of target wells are not all on a straight line.

[0207] Furthermore, all target wells in each group of target wells are distributed in an array.

[0208] Furthermore, after all target wells in each group of target wells are distributed in an array to form an array, the centers of the arrays are the same.

[0209] Furthermore, in each array, the distance between the centers of two adjacent target holes in each row is equal, and the distance between the centers of two adjacent target holes in each column is equal.

[0210] Furthermore, the target hole is a circular hole, and the back hole opening of the target hole is chamfered. After the back hole opening of the target hole is chamfered, it has a circular hole section and a conical hole section connected to each other, the circular hole section is away from the heat source module, and the conical hole section is close to the heat source module (the hole diameter of the conical hole section increases as it approaches the heat source module), and the minimum hole diameter of the conical hole section is equal to the hole diameter of the circular hole section.

[0211] Furthermore, the front surface of the calibration plate body is treated with matte white frosting, and the back surface of the calibration plate body is treated with bright black high-reflective treatment.

[0212] In one embodiment, two groups of target holes are provided on the calibration plate body 51, namely the first group of target holes and the second group of target holes. The first group of target holes includes a plurality of first target holes 52 (large circular holes), and the second group of target holes includes a plurality of second target holes 53 (small circular holes). The aperture of the first target holes 52 is larger than the aperture of the second target holes 53. All the first target holes 52 are located in the first area, and all the second target holes 53 are located in the second area.

[0213] Furthermore, the first area is a first rectangular area, the length of the first rectangular area is 800 mm, and the width is 600 mm; the second area is a second rectangular area, the length of the second rectangular area is 70 mm, and the width is 50 mm.

[0214] Furthermore, the centers of the first rectangular area and the second rectangular area are the same.

[0215] Furthermore, the first rectangular area, the second rectangular area and the calibration plate 5 have the same center.

[0216] Furthermore, the number of the first target holes 52 and the second target holes 53 is at least three, and all the first target holes 52 are not all on a straight line, and all the second target holes 53 are not all on a straight line.

[0217] The diameters of the first target hole 52 and the second target hole 53 are set as needed. In one embodiment, the diameter of the first target hole 52 is 100 mm, and the diameter of the second target hole 53 is 10 mm.

[0218] Furthermore, the plurality of first target holes 52 are distributed in an array to form a first array, and the plurality of second target holes 53 are distributed in an array to form a second array, and the center of the first array is the same as the center of the second array.

[0219] Furthermore, the distance between the centers of two adjacent first target holes 52 in each row of the first array is equal. The distance between the centers of two adjacent first target holes 52 in each column is equal.

[0220] Furthermore, the distance between the centers of two adjacent second target holes 53 in each row of the second array is equal. The distance between the centers of two adjacent second target holes 53 in each column is equal.

[0221] Furthermore, the first target hole 52 and the second target hole 53 are circular holes, and the back openings of the first target hole 52 and the second target hole 53 are both chamfered. The calibration plate 5 of this embodiment has a thickness of 3 mm and is made of aluminum alloy.

[0222] A specific embodiment of a calibration plate 5 is as follows: six first target holes 52 (the first target holes 52 are large circular holes with a diameter of 100 mm) and six second target holes 53 (the second target holes 53 are small circular holes with a diameter of 10 mm) are processed on the calibration plate body 51. The back of each hole is chamfered by 1.5x60°mm. After processing is completed, the front side of the calibration plate body 51, that is, the outward side, is subjected to matte white frosting treatment, and the back side of the calibration plate body 51, that is, the inward side, is subjected to bright black high-reflective treatment.

[0223] In this embodiment, the six first target holes 52 are arranged in a first array of two rows and three columns. The distance between the centers of two adjacent first target holes 52 in each row of the first array is 200 mm. The distance between the centers of two adjacent first target holes 52 in each column of the first array is 200 mm.

[0224] In this embodiment, the six second target holes 53 are arranged in a second array of two rows and three columns. The spacing between the centers of two adjacent second target holes 53 in each row of the second array is 20 mm. The spacing between the centers of two adjacent first target holes 52 in each column of the second array is 20 mm.

[0225] In this embodiment, the center of the first array, the center of the second array and the center of the calibration plate 5 are the same.

[0226] Example 4

[0227] 8 to 14 , an embodiment of the present invention provides an infrared dual-light fusion testing device, including a heat source module and a calibration plate 5 , wherein the heat source module is used to provide a heat source for the calibration plate 5 .

[0228] The calibration process in the first embodiment can be calibrated using the calibration plate 5 .

[0229] Each group of target holes corresponds to a distance range one by one, and the distance between the infrared thermal imaging device and the calibration plate 5 is obtained. The distance between the infrared thermal imaging device and the calibration plate 5 is compared with the set multiple distance ranges to determine which distance range the distance between the infrared thermal imaging device and the calibration plate 5 is in. When the distance between the infrared thermal imaging device and the calibration plate 5 is within a certain distance range, a group of target holes corresponding to the distance range is used for calibration (during calibration, the visible light image at the calibration distance and the center of the group of target holes of the calibration plate in the infrared image are extracted and saved, and then calibration is performed).

[0230] For example, when two groups of target holes are set on the calibration plate 5, namely the first group of target holes (a group of large circular holes) and the second group of target holes (a group of small circular holes), paired visible light images and infrared images are collected by the infrared thermal imaging device, and the distance between the infrared thermal imaging device and the calibration plate 5 is compared with the set first distance (such as 2 meters). When the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to the first distance (such as 2 meters), the second target hole 53 (small circular hole) is used for calibration (that is, the visible light image at the calibration distance and the center of the second target hole of the calibration plate in the infrared image, i.e., the center of the small circle, are extracted and saved, and then calibration is performed). When the distance between the infrared thermal imaging device and the calibration plate 5 is greater than the first distance (such as 2 meters), the first target hole 52 (large circular hole) is used for calibration (that is, the visible light image at the calibration distance and the center of the first target hole of the calibration plate in the infrared image, i.e., the center of the large circle, are extracted and saved, and then calibration is performed). It is generally believed that when the distance between the infrared thermal imaging device and the calibration plate 5 is less than or equal to 2 meters, the distance is considered to be close, and when it is greater than 2 meters, the distance is considered to be far.

[0231] Furthermore, the heat source module includes at least one heating unit, and each group of target holes corresponds to one or more heating units.

[0232] Furthermore, at least one group of target holes corresponds to a plurality of heating units.

[0233] Furthermore, at least one group of target holes corresponds to one heating unit.

[0234] Furthermore, when a certain group of target wells corresponds to multiple heating units, each target well in the group of target wells corresponds to one heating unit one by one.

[0235] Two groups of target holes are set on the calibration plate 5 of this embodiment, namely the first group of target holes and the second group of target holes. The first group of target holes includes multiple first target holes 52 (large circular holes), and the second group of target holes includes multiple second target holes 53 (small circular holes).

[0236] Furthermore, each first target hole 52 of the calibration plate 5 corresponds to a heating unit (i.e., a first heating unit), and all second target holes 53 of the calibration plate 5 correspond to a heating unit (i.e., a second heating unit). The sizes of the first heating unit and the second heating unit may vary as needed. In this embodiment, the size of the first heating unit is larger than the size of the second heating unit.

[0237] This embodiment has a total of seven heating units, including six first heating units and one second heating unit. The six first heating units correspond one-to-one to the six first target holes 52 of the calibration plate 5 and are arranged relative to each other. Each first heating unit provides heat to the corresponding first target hole 52 on the calibration plate 5. One second heating unit corresponds to the six second target holes 53 of the calibration plate 5 and provides heat to the six second target holes 53 of the calibration plate 5.

[0238] Furthermore, the heating unit includes a heat source body 21, in which a heating element 22 and a temperature detection element 23 are installed. The heating element 22 is connected to the output end of the controller, and the temperature detection element 23 is connected to the input end of the controller. The controller is connected to the instruction input unit and / or the display unit.

[0239] Each heating unit is controlled individually.

[0240] Furthermore, the heating element 22 may be a heating rod, and the temperature detection element 23 may be a thermocouple.

[0241] The heat source body 21 of the first heating unit of this embodiment uses two heating rods and a thermocouple.

[0242] The heat source body 21 of the second heating unit of this embodiment adopts a heating rod and a thermocouple.

[0243] The heat source body 21 of this embodiment is a rectangular block. A first mounting hole 211 for mounting a heating rod and a second mounting hole 212 for mounting a thermocouple are provided on the heat source body 21. The lead ends of the heating rod and the lead ends of the thermocouple extend out of the heat source body 21.

[0244] The rectangular block of the heat source 21 is made of a material with good thermal conductivity, such as aluminum in this embodiment. The surface of the heat source 21, i.e., the aluminum plate, is treated with a bright black high-reflective treatment to improve imaging quality.

[0245] Furthermore, the heating unit is connected to the heat source fixing plate 3 via a first thermal insulation board 24, which is mounted on the back of the calibration plate 5. In this embodiment, the first thermal insulation board 24 is made of mica fiberboard, which provides thermal insulation. The heat source 21 is secured to the heat source fixing plate 3 using the first thermal insulation board 24 and does not contact the heat source fixing plate 3.

[0246] The heat source fixing plate 3 is provided with a mounting hole for mounting the heating unit. The heat source body 21 of this embodiment is mounted in the mounting hole of the heat source fixing plate 3.

[0247] A second heat insulation plate 4 is provided between the calibration plate 5 and the heat source fixing plate 3. The heat source fixing plate 3, the second heat insulation plate 4 and the calibration plate 5 are connected in sequence. The second heat insulation plate 4 is provided with a through hole for the heat source of the heating unit to pass through.

[0248] The second heat insulation board 4 of this embodiment is made of mica fiber board, which plays a heat insulation role.

[0249] Furthermore, each heating unit is connected to the first heat insulation plate 24 in a one-to-one correspondence, such as being fixedly connected by bolts, and the first heat insulation plate 24 is connected to the heat source fixing plate 3, such as being fixedly connected by bolts.

[0250] Furthermore, the second thermal insulation board 4 is provided with a plurality of first through holes for passage of the heat source of the first heating unit. The first through holes of the second thermal insulation board 4 correspond one-to-one with the first target holes 52 of the calibration board 5 and are arranged opposite each other. The apertures of the first through holes of the second thermal insulation board 4 are set as needed so that the heat source of the first heating unit can pass through the corresponding first through holes and reach the corresponding first target holes 52.

[0251] Furthermore, the second thermal insulation board 4 is provided with one or more second through holes for the heat source of the second heating unit to pass through. In this embodiment, the second thermal insulation board 4 is provided with one second through hole for the heat source of the second heating unit to pass through, and the second through hole is arranged opposite to all the second target holes 53 on the calibration board 5.

[0252] The number and size of the second through holes are set as needed, so that the heat source of the second heating unit can reach the second target hole 53 through the second through holes.

[0253] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are connected by bolts. The heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are respectively provided with bolt holes.

[0254] In this embodiment, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are all rectangular, and have equal lengths and widths.

[0255] Furthermore, the heat source fixing plate 3, the second heat insulation plate 4, and the calibration plate 5 are sequentially connected to form a heating target assembly, which is fixed to a chassis 1 to form a heating target box. In this embodiment, the entire heating target assembly is fixed to the sheet metal chassis 1 by screws.

[0256] The entire chassis 1 (such as a sheet metal chassis) is processed using 2mm steel plates and then sprayed with black paint. The heating unit is heated by a heating rod, and the heat is transferred to the heat source 21. A sheet metal door assembly 6 with a fan is installed on the chassis 1. The fan is used for heat dissipation. The calibration plate 5 serves as a side panel of the chassis 1, and the heat source fixing plate 3 is located inside the chassis 1. The chassis 1 is equipped with a human-machine interface 8, a power switch 9, a start switch 10, and an indicator light 7 for displaying the heating status. The human-machine interface 8 and the indicator light 7 are connected to the controller. The controller is provided with a control unit and a temperature control module. The control unit is connected to the temperature detection element 23, the temperature control module, the human-machine interface 8, and the indicator light 7. The temperature control module is connected to the heating element 22, and the temperature of each heat source 21 is controlled by controlling the temperature control module. The temperature of each heating unit is controlled by the human-machine interface 8, and each target can be controlled separately as required. Two handles are installed on the chassis 1 for carrying the heating target part. Its working principle is that the heating rod heats and transfers heat to the heat source body 21. The built-in temperature sensor of the thermocouple senses the real-time temperature of the heat source body 21 and displays the actual sensed temperature through the human-machine interface 8. If the actual measurement requires the temperature to be adjusted up or down, the temperature can be set through the control interface of the human-machine interface 8. Each temperature can be controlled separately (adjustment range 25°~120°).

[0257] Furthermore, the heating target box is mounted on the frame 11. The heating target box of this embodiment is fixed on the frame 11 by screws.

[0258] Furthermore, casters 116 are provided at the lower end of the frame 11 .

[0259] The rack 11 includes a body frame 111, a fixing plate 112, a double-opening sheet metal door 113, a stainless steel cover 114, and a rack bottom plate 115. The body frame 111 is welded with stainless steel square tubes on all sides, and the rear and side fixing plates 112 are fixed to the rack 11 with stainless steel sheet metal by screws. The front of the body frame 111 adopts a stainless steel double-opening door, and the top stainless steel cover 114 is fixed to the rack 11 for fixing the heating target box. The bottom plate of the chassis 1 is made of stainless steel sheet metal and a universal wheel with a foot cup is installed at each corner for movement and fixation.

[0260] The calibration plate of the present invention has target holes of various diameters, which can adapt to tests of various distances (such as close distance and long distance, the distance refers to the distance between the infrared thermal imaging device and the calibration plate). It has strong versatility and can meet the needs of optical axis testing of existing products, and can meet the needs of multiple uses of one machine.

[0261] Each target body of the present invention adopts a separate temperature control system, which can accurately control the temperature. The target is highly accurate and more stable, which can ensure the consistency of temperature and improve the accuracy of the test.

[0262] The present invention provides a two-stage precise registration method for infrared and visible light images based on sparse features, which involves the extraction and matching of sparse features, a two-stage precise registration strategy, and a second registration correction method to improve the accuracy and robustness of infrared and visible light image registration.

[0263] The method of the present invention can automatically correct the registration deviation caused by affine transformation in real time for each frame of bi-optical image, presenting accurate bi-optical fusion results. At the same time, it has strong robustness, practicality, universality and other characteristics, and can be well applied in infrared and visible light bi-optical fusion systems.

[0264] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for registering infrared and visible light images, characterized in that: The steps include: Obtain a visible light image and an infrared image, and use one of the images as a reference image and the other as an image to be registered; Performing affine transformation on the image to be registered using the affine transformation parameters obtained by calibration to obtain a first registered image; The reference image is registered with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image, and the translation parameter offset is used to correct the translation parameter in the calibrated affine transformation parameters to obtain corrected affine transformation parameters.

2. The infrared and visible light image registration method according to claim 1, characterized in that: The method also includes the following steps: performing affine transformation on the image to be registered using the corrected affine transformation parameters to obtain a registered image.

3. The infrared and visible light image registration method according to claim 1, characterized in that: The reference image is registered with the first registered image to obtain a translation parameter offset of the first registered image relative to the reference image, specifically including: Discretize the reference image and the first registered image respectively, and use one of the obtained feature point sets as the reference point set and the other feature point set as the point set to be registered; The reference point set is registered with the point set to be registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated.

4. The infrared and visible light image registration method according to claim 3, characterized in that: Before discretizing the reference image and the first registered image, the following steps are also included: Feature extraction includes: extracting boundaries of the reference image and the first registered image respectively to obtain a first boundary binary image and a second boundary binary image; The feature matching includes: matching the boundary features in the first boundary binary image and the second boundary binary image, and deleting the boundary features in the first boundary binary image and the second boundary binary image that are not matched with each other.

5. The infrared and visible light image registration method according to claim 4, characterized in that: After the feature extraction step and before the feature matching step, the following steps are also included: performing connected domain analysis on the first boundary binary image and the second boundary binary image respectively to obtain the area value of each connected area, and setting the pixel grayscale values ​​of the connected area whose area value is less than the preset area threshold to 0.

6. The infrared and visible light image registration method according to claim 3, characterized in that: The reference point set is registered with the point set to be registered, and the translation parameter offset of the point set to be registered relative to the reference point set is calculated, specifically including: (a) Initialization; (b) Calculate the covariance matrix between the reference point set and the point set to be registered; (c) calculating the translation parameter offset of the point set to be registered relative to the reference point set according to the covariance matrix; (d) calculating the coordinates of the updated set of points to be registered and the set of reference points; (e) updating the covariance matrix; (f) Determine whether the stop iteration condition is met according to the covariance matrix. If the stop iteration condition is met, stop the iteration and output the currently calculated translation parameter offset; otherwise, return to step (c) to continue iterating until the stop iteration condition is met.

7. The infrared and visible light image registration method according to claim 6, characterized in that: Iteratively solving the translation parameter offset of the to-be-registered point set relative to the reference point set according to the covariance matrix, specifically including: calculating the matching degree matrix according to the covariance matrix; Calculate the mean of the reference point set and the mean of the point set to be registered according to the matching degree matrix; The translation parameter offset of the point set to be registered relative to the reference point set is calculated according to the mean value of the reference point set and the mean value of the point set to be registered.

8. The infrared and visible light image registration method according to claim 6, characterized in that: The tolerance is calculated based on the covariance matrix, and the iteration stopping condition is: the absolute value of the percentage by which the set initial tolerance is reduced compared to the calculated tolerance is less than the set tolerance threshold.

9. An infrared and visible light image registration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the infrared and visible light image registration method according to any one of claims 1 to 8 are implemented.

10. A method for fusing infrared and visible light images, characterized in that: The steps include: Obtain a visible light image and an infrared image, and use one of the images as a reference image and the other as an image to be registered; Using the registration method according to any one of claims 1 to 8 to obtain the corrected affine transformation parameters; The modified affine transformation parameters are used to perform affine transformation on the image to be registered, and the transformed registered image is fused with the reference image to obtain a fused image.

Images