Image registration method based on multi-distance calibration and linear fusion of matrix parameters

Abstract

The application provides an image registration method based on multi-distance calibration and linear fusion of matrix parameters, and relates to the technical field of image processing, and comprises the following steps: a plurality of discrete standard working distances are selected in advance to form a standard working distance set, and infrared calibration images and visible light calibration images are synchronously collected; a corresponding perspective transformation matrix is respectively solved according to each standard working distance; the perspective transformation matrix is disassembled into independent matrix parameter components, and then a geometric mapping relationship between the corresponding standard working distance and the independent matrix parameter component is constructed; a linear parameter fusion interpolation model is constructed based on two adjacent standard working distances and the geometric mapping relationship; an actual working distance is obtained, and a corresponding target perspective transformation matrix is solved through the linear parameter fusion interpolation model; and the target infrared image or the target visible light image is corrected through perspective transformation according to the target perspective transformation matrix, so that the target infrared image and the target visible light image are registered and aligned.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image registration method based on multi-distance calibration and linear fusion of matrix parameters. Background Technology

[0002] Infrared and visible light imaging fusion sensing technology is widely used in many fields such as power equipment inspection, industrial defect detection, vehicle-mounted assisted driving, and drone security monitoring. Due to inherent differences between infrared and visible light cameras in terms of installation position, lens distortion, and imaging field of view, the two acquired images have perspective shift and projection deviation. Therefore, image registration is required to achieve pixel-level alignment, and perspective transformation models are currently the mainstream correction method for dual-band image registration.

[0003] Currently, infrared and visible light image registration mainly employs two implementation methods. One method involves extracting feature points such as SIFT and ORB frame by frame, relying on cross-modal feature matching combined with the RANSAC algorithm to calculate the perspective transformation matrix in real time to complete the registration. This method is greatly affected by the differences between infrared and visible light modes, resulting in inconsistent distribution of image texture features, which easily leads to problems such as mismatch and feature loss. Its registration robustness is insufficient, and the computational load of feature extraction and matrix solving throughout the process is large, making it difficult to adapt to the real-time operation requirements of embedded low-computing-power platforms. The other method involves calibrating a fixed perspective matrix only at a single standard working distance, and reusing this matrix for registration throughout the device's operation. This method is only suitable for the calibration distance condition; when the actual working distance changes, the registration accuracy drops significantly, making it unsuitable for continuously variable distance scenarios. Manually calibrating multiple sets of distance matrices is not only cumbersome and costly, but also fails to smoothly adapt to any intermediate working distance.

[0004] Existing technologies cannot simultaneously achieve adaptability to variable-distance working conditions, registration accuracy, and hardware real-time performance. Online feature solving has high computational overhead and poor robustness, while fixed matrix calibration has a narrow range of working conditions, making it difficult to meet the requirements of high-precision and stable registration of infrared and visible light images under variable working distances in practical engineering.

[0005] Therefore, we now present an image registration method based on multi-distance calibration and linear fusion of matrix parameters. Summary of the Invention

[0006] To address the shortcomings of existing infrared and visible light image registration techniques, such as high computational cost of online feature solving, susceptibility to mismatching of cross-modal features, poor robustness, inability of a single fixed-distance perspective matrix to adapt to continuous variable-distance conditions, cumbersome multi-distance manual calibration process, and insufficient registration accuracy at intermediate working distances, this invention aims to provide an image registration method based on multi-distance calibration and linear fusion of matrix parameters.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an image registration method based on multi-distance calibration and linear fusion of matrix parameters, the method comprising:

[0008] Multiple discrete standard working distances are pre-selected, and the perspective transformation matrix of each standard working distance is solved respectively; the perspective transformation matrix is ​​decomposed into independent matrix parameter components, and then the geometric mapping relationship between the corresponding standard working distance and the independent matrix parameter components is constructed.

[0009] Based on the adjacent standard working distances and the geometric mapping relationship, a linear parameter fusion interpolation model is constructed;

[0010] The actual working distance of the device is obtained, and the target perspective transformation matrix corresponding to the actual working distance is solved by the linear parameter fusion interpolation model.

[0011] The infrared image or visible light image of the target is corrected by perspective transformation according to the target perspective transformation matrix, so as to achieve registration and alignment between the infrared image and the visible light image of the target.

[0012] Furthermore, the process of solving for the perspective transformation matrix of each of the aforementioned standard working distances includes:

[0013] Calibration targets are placed sequentially at multiple discrete standard working distances, and corresponding infrared calibration images and visible light calibration images are acquired simultaneously to obtain dual-modal calibration image data.

[0014] The target corner points of the calibration image data are extracted as matching feature point pairs. The corresponding perspective transformation matrix is ​​fitted based on the least squares algorithm, and the RANSAC algorithm is used to remove erroneous feature point pairs.

[0015] Furthermore, the process of constructing a linear parameter fusion interpolation model based on the adjacent standard working distances and the aforementioned geometric mapping relationship includes:

[0016] Traverse the set of standard working distances and divide adjacent standard working distance intervals;

[0017] For each group of standard working distance intervals, based on the geometric mapping relationship of the standard working distances of the two endpoints in the standard working distance interval, obtain the eight independent matrix parameter components corresponding to the standard working distances of the two endpoints, and treat the eight independent matrix parameter components as one-dimensional variables respectively.

[0018] Based on the one-dimensional variable and the working distance, and the matrix parameters corresponding one-to-one with the eight independent matrix parameter components in the one-dimensional variable, and based on the linear interpolation algorithm, a linear variation model of the eight independent matrix parameter components in the one-dimensional variable with the matrix parameters corresponding one-to-one with the working distance is established, which is denoted as the linear parameter fusion interpolation model.

[0019] Furthermore, the process of solving for the target perspective transformation matrix corresponding to the actual working distance through the linear parameter fusion interpolation model includes:

[0020] The geometric mapping relationship corresponding to the standard working distance at both ends of the standard working distance interval is retrieved, and then the corresponding 8 independent matrix parameter components are obtained;

[0021] According to the linear parameter fusion interpolation model, weighted interpolation calculations are performed based on the eight independent matrix parameter components corresponding to the actual working distance to obtain the independent matrix parameter components of the actual working distance, forming the corresponding actual complete parameter vector; the independent matrix parameter components in the actual complete parameter vector are substituted into the perspective transformation matrix, and the perspective transformation matrix is ​​reconstructed according to the scale normalization property of the third-order homography matrix to generate the target perspective transformation matrix of the actual working distance.

[0022] Furthermore, the process of performing perspective transformation correction on the target infrared image or target visible light image based on the target perspective transformation matrix includes:

[0023] A global mapping rule is established based on the target perspective transformation matrix; wherein, the global mapping rule includes a global mapping relationship from visible light pixel coordinates to infrared pixel coordinates and a global mapping relationship from infrared pixel coordinates to visible light pixel coordinates;

[0024] Traverse all the original pixel coordinates in the image to be registered, and calculate the target pixel coordinates corresponding to the original pixel coordinates according to the global mapping rules.

[0025] Furthermore, the process of grayscale resampling and filling of the target pixel coordinates based on the bilinear interpolation algorithm includes:

[0026] Locate the four adjacent integer pixels of the target pixel coordinates, and obtain the original grayscale values ​​of the image corresponding to the four neighboring pixels respectively;

[0027] First, perform linear interpolation on the gray levels of two adjacent pixels along the horizontal axis of the image to obtain the intermediate gray level values ​​of the two horizontal interpolations; then, perform a second linear interpolation on the two intermediate gray level values ​​along the vertical axis of the image to finally obtain the interpolated gray level value corresponding to the floating-point target coordinates.

[0028] The interpolated grayscale value obtained by interpolation is assigned to the target pixel coordinates corresponding to the output image to be registered; for edge pixels of the image to be registered that do not have complete four neighboring points, the edge pixels are filled based on the boundary grayscale copy, thereby realizing the registration and alignment of the target infrared image and the target visible light image.

[0029] Furthermore, the process of extracting the corner point position deviation for corner points where the calibration target is not directly facing the device, and constructing an attitude compensation correction model includes:

[0030] Based on the real-time collected spatial pitch data, yaw attitude data, and the rotation data of the calibration target plane itself, a corresponding overall target attitude rotation matrix is ​​constructed; based on the corresponding standard working distance, the lateral offset of the calibration target to the device, and the longitudinal offset, a translation transformation matrix is ​​constructed; and then the pixel offset of the calibration target is solved.

[0031] The corresponding attitude compensation and correction model is composed of the overall target attitude rotation matrix, translation transformation matrix, and pixel offset.

[0032] Furthermore, the process of determining another matching method for the standard working distance range in which the actual working distance falls includes:

[0033] Choose any two adjacent standard working distances and set the boundary buffer distance value; based on the two adjacent standard working distances and the boundary buffer distance value, obtain the boundary buffer space; when the actual working distance falls into the boundary buffer space, dynamically allocate the boundary buffer weight according to the distance between the actual working distance and the two adjacent standard working distances in the boundary buffer space.

[0034] The geometric mapping relationship corresponding to the standard working distance between the two ends of the boundary buffer is retrieved, and the corresponding 8 boundary buffer independent matrix parameter components are obtained according to the corresponding boundary buffer weights. Then, according to the linear parameter fusion interpolation model, weighted interpolation calculation is performed based on the 8 boundary buffer independent matrix parameter components corresponding to the actual working distance, thereby generating the target perspective transformation matrix of the actual working distance.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. It eliminates the need for frame-by-frame feature extraction, feature matching, and perspective matrix solving, saving a significant amount of image processing computation and greatly reducing the computational cost of the algorithm. It can be adapted to low-computing-power hardware platforms such as embedded and edge devices, meeting the engineering application requirements for real-time registration.

[0037] 2. It completely eliminates the dependence on the texture features of infrared and visible light images, avoids problems such as large gray-level differences in cross-modal images, lack of features in weak texture areas, and mismatch of features, and greatly improves the robustness of registration in complex scenes.

[0038] 3. Only a small number of discrete standard working distances need to be calibrated offline, eliminating the need to calibrate the entire range of distances one by one. This simplifies the on-site calibration process and reduces the labor and time costs of equipment debugging and mass production maintenance.

[0039] 4. By linearly fusing and interpolating matrix parameters, a perspective transformation matrix for any intermediate working distance can be smoothly generated, enabling adaptive adaptation for continuous variable distance working conditions across the entire range. This solves the problem that traditional fixed matrices can only adapt to a single distance, and the registration accuracy drops sharply after distance offset.

[0040] 5. During continuous changes in working distance, the registration parameters transition smoothly without abrupt changes, and the registration accuracy remains stable throughout, effectively improving the imaging alignment effect under actual working conditions such as power line inspection, industrial testing, and vehicle sensing.

[0041] 6. The overall solution has a simple architecture and well-organized logic. The algorithm is easy to port, encapsulate, and implement in engineering. It is compatible with mass production integration of batch equipment and has strong versatility and scalability.

[0042] 7. Based on the offline calibration and online interpolation operation mode, the algorithm complexity in the online operation stage is greatly reduced, and the system operation stability is effectively improved while ensuring registration accuracy. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0044] Figure 1 This is a schematic diagram illustrating the steps of an image registration method based on multi-distance calibration and linear fusion of matrix parameters.

[0045] Figure 2 A flowchart is constructed for standard distance calibration and matrix mapping.

[0046] Figure 3 A flowchart for solving the target perspective matrix.

[0047] Figure 4 Schematic diagram of the registration and correction principle for infrared and visible light images. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] Infrared and visible light image registration falls under the category of heterogeneous multimodal image registration. For visible light and infrared thermal images simultaneously acquired in the same observation scene, a pixel spatial mapping relationship is constructed based on the differences in the imaging mechanisms of the two types of images. A suitable geometric transformation model is then solved to correct translation, rotation, scaling, and perspective deviations between the images, achieving precise alignment of the pixel positions of the same entity within the scene in both images. Visible light relies on reflected light from objects to form images, revealing rich texture and color information. Infrared images capture the thermal radiation signal of the object itself, reflecting temperature distribution characteristics. The grayscale distributions of the two are not directly correlated, making conventional grayscale matching criteria unsuitable. Furthermore, hardware installation deviations of the dual cameras, optical lens distortion, and imaging parallax further exacerbate image misalignment. The registration algorithm needs to overcome the matching difficulties caused by cross-modal grayscale inconsistencies.

[0051] In practical applications, such as all-weather security monitoring, visible light is used to identify the shape features of targets, while infrared light detects heat sources under conditions such as nighttime, fog, and backlight. After image registration, the two types of image information can be fused to eliminate target misalignment and achieve uninterrupted target tracking and vehicle / person classification day and night. Alternatively, in UAV-borne dual-light inspection scenarios, visible light and infrared payloads are widely used. After image registration and alignment, visual inspection of equipment appearance defects and detection of abnormal temperatures and thermal faults can be completed simultaneously, which can be applied to the inspection of power lines, photovoltaic power stations, oil and gas pipelines, and wind turbine blades. In vehicle-mounted dual-light assisted driving systems, the two image coordinate systems are unified through registration. Infrared light identifies pedestrians, animals, and other heat source obstacles at night, while visible light provides road markings and lane line information. The fused output provides a unified field of view to assist in hazard warnings.

[0052] Based on this, in a first aspect, this application discloses at least one embodiment that provides an image registration method based on multi-distance calibration and linear fusion of matrix parameters, so as to ensure registration efficiency and registration accuracy, and realize the registration of infrared and visible light images.

[0053] Please refer to the appendix. Figure 1 This application discloses an image registration method based on multi-distance calibration and linear fusion of matrix parameters, the method comprising:

[0054] Step S1: Pre-select multiple discrete standard working distances to form a standard working distance set, and simultaneously acquire infrared calibration images and visible calibration images at the standard working distances;

[0055] Optionally, in this embodiment, based on the actual operating distance range of the equipment, a lower limit and an upper limit of the actual operating distance range are determined, and several discrete and uniformly distributed standard working distances are selected within the actual operating distance range to form a standard distance set:

[0056] Among them, satisfying ; The selected standard working distances must cover the entire operating range of the corresponding equipment, in order to meet the preset number of calibration working distances.

[0057] Step S2: Based on each of the standard working distances, solve for the perspective transformation matrix of the corresponding standard working distance; decompose the perspective transformation matrix into independent matrix parameter components, and then construct the geometric mapping relationship between the corresponding standard working distance and the independent matrix parameter components;

[0058] Optionally, in this embodiment, the offline calibration process of pre-selecting multiple discrete standard working distances and acquiring infrared calibration images and visible light calibration images further includes adaptive intelligent selection of standard distances and calibration error compensation optimization, the specific process of which is as follows:

[0059] Based on the actual operating distance range of the equipment, determine the lower limit and upper limit of the actual operating distance range. Within the actual operating distance range, select several discrete and uniformly distributed standard working distances to form a standard distance set. ;

[0060] When the standard distance set Two adjacent standard working distances When the gradient of the independent matrix parameter components exceeds the gradient threshold, the adjacent standard working distance is determined. If the interpolation accuracy of the interval is insufficient, it will automatically adjust the interpolation at adjacent standard working distances. A new encrypted standard working distance has been added at the midpoint of the interval.

[0061] Optionally, in this embodiment, when acquiring infrared calibration images and visible light calibration images at various standard distances, the spatial pitch data, yaw attitude data, and rotation data of the calibration target plane itself are simultaneously acquired. The specific process of extracting corner position deviations for corner points of the calibration target that are not directly facing the device and constructing an attitude compensation correction model includes:

[0062] When the calibration target plane is completely perpendicular to the image acquisition module, a two-dimensional coordinate system of the target world is constructed. ;

[0063] Based on the spatial pitch data collected in real time by the image acquisition module Yaw attitude data And the rotation data of the calibration target plane itself. Construct the corresponding pitch and rotation matrix yaw rotation matrix and the roll rotation matrix The overall attitude rotation matrix of the target is obtained. .

[0064] It should be noted that the overall target attitude rotation matrix Describe the current tilted calibration target coordinate system relative to the target world two-dimensional coordinate system. Spatial rotation transformation relationship.

[0065] According to the corresponding standard working distance The horizontal and vertical offsets of the target to the image acquisition module are determined to generate a three-dimensional translation vector. And form a translation transformation matrix. ;in, The horizontal offset is the horizontal displacement of the target center relative to the optical axis of the image acquisition module. Right offset is positive and left offset is negative. The vertical axis offset represents the vertical displacement of the target center relative to the optical axis of the image acquisition module. A positive value indicates the target is raised and a negative value indicates it is lowered. This is the depth offset, the vertical straight-line distance from the calibrated target to the optical center of the image acquisition module, i.e., the current standard working distance. , is always a positive number;

[0066] Based on the overall attitude rotation matrix of the target and translation transformation matrix The original three-dimensional coordinates of the corner points of the calibration target that are not directly opposite the device are obtained. Based on the camera pinhole imaging model, the original three-dimensional coordinates are... Projected onto the image pixel plane, the uncorrected original corner pixel coordinates are obtained. ; target world two-dimensional coordinate system Direct projection yields standard, unbiased, ideal pixel coordinates. Solve for the pixel offset caused by the tilt of the calibration target. ;

[0067] The overall attitude rotation matrix of the target Translation transformation matrix Uncorrected original corner pixel coordinates and standard unbiased ideal pixel coordinates This forms the corresponding attitude compensation and correction model;

[0068] The corner points of non-directly aligned devices are pre-corrected based on the attitude compensation correction model to obtain the final corner point correction coordinates. ;in, The corner pixel coordinates of the tilted calibration target obtained from image detection.

[0069] It should be further explained that, in the specific implementation process, the specific process of solving for the perspective transformation matrix of each standard working distance includes:

[0070] Optionally, in this embodiment, the relative installation position, lens parameters, and overall hardware orientation between the infrared imaging device and the visible light imaging device are kept constant throughout the process; and the process is carried out sequentially at multiple discrete standard working distances. A calibration target was placed at the location, and all standard working distances were collected simultaneously. The infrared calibration image and the visible calibration image are denoted as follows: , The dual-modal calibration image data was obtained.

[0071] From each group of infrared calibration images and visible cursor fixed image The corner points of the calibrated target after compensation and correction are extracted as matching feature point pairs. Based on the least squares algorithm and the matching feature point pairs, the corresponding initial homography matrix is ​​obtained. The specific process is as follows:

[0072] Optionally, in this embodiment, based on the pixel mapping homogeneous equation of the third-order homogeneous homography matrix, each pair of matching feature points is transformed into two sets of linear constraint equations, and all pairs of matching feature points are combined to construct an overdetermined system of linear equations.

[0073] For the overdetermined system of equations, a coefficient matrix and a constant vector are constructed. Based on the least squares solution method, the optimal approximate solution of the system of equations is obtained by matrix singular value decomposition (SVD).

[0074] Apply homogeneous matrix scaling constraints to the original matrix parameter component solutions of the obtained 9-dimensional matrix, normalize the lower right corner element of the matrix to 1, eliminate the scaling degrees of freedom, and construct a complete third-order homogeneous homography matrix from the remaining 8 original matrix parameter components.

[0075] It should be noted that the third-order homography matrix does not remove the interference from mismatched points and is used as the initial homography matrix corresponding to the current standard distance.

[0076] Based on the initial homography matrix, and using the RANSAC algorithm, the reprojection error of each matching feature point pair is checked to filter out valid matching feature point pairs and cross-modal mismatched feature point pairs. The specific process is as follows:

[0077] Optionally, in this embodiment, a reprojection error threshold and the maximum number of RANSAC iterations are preset, and all matching feature point pairs are stored in the matching feature point pair set.

[0078] Randomly select the minimum number of matching feature point pairs, resolve the temporary homography matrix of the corresponding minimum number of matching feature point pairs based on least squares, and calculate the reprojection error of all matching feature point pairs in the set of matching feature point pairs based on the temporary homography matrix.

[0079] The single-point reprojection error is calculated as follows: the visible light pixel coordinates are substituted into the temporary homography matrix, the mapped infrared pixel coordinates are calculated, and the Euclidean distance between the mapped infrared pixel coordinates and the actual infrared corner coordinates is taken as the reprojection error of that point.

[0080] Optionally, in this embodiment, the single-point reprojection error is compared with a preset error threshold. If the single-point reprojection error is less than or equal to the preset error threshold, the matching feature point pair is determined to be a valid matching feature point pair (i.e., an interior point). If the single-point reprojection error is greater than the preset error threshold, it is determined to be a cross-modal mismatched feature point pair (i.e., an exterior point).

[0081] Optionally, in this embodiment, the total number of valid matching feature point pairs corresponding to this sampling is counted, and the current optimal temporary homography matrix and the corresponding set of valid matching feature point pairs are recorded; the process is iterated until the preset maximum number of iterations is reached.

[0082] After the iteration, the optimal homography matrix with the largest number of valid interior points is selected as the verification benchmark, and all valid matching feature point pairs and cross-modal mismatched feature point pairs corresponding to the optimal homography matrix are output to complete the point pair classification and screening.

[0083] Optionally, in this embodiment, the effective matching feature point pairs are iteratively optimized based on the least squares algorithm to output the perspective transformation matrix of the standard working distance. The perspective transformation matrix for:

[0084] .

[0085] It should be noted that the perspective transformation matrix is ​​based on the third-order homography matrix as the fundamental matrix.

[0086] Furthermore, the pixel mapping relationship under homogeneous coordinates satisfies:

[0087] In the formula, The input image pixel coordinates; The corresponding pixel coordinates of the mapped image.

[0088] It should be further explained that, in the specific implementation process, the specific process of decomposing the perspective transformation matrix into independent matrix parameter components, and then constructing the geometric mapping relationship between the corresponding standard working distance and the independent matrix parameter components includes:

[0089] Optionally, in this embodiment, the normalization condition of the perspective transformation matrix is ​​constrained based on the scale normalization property of the third-order homogeneous homography matrix. The matrix elements in the perspective transformation matrix, which are fixed as constants, are constrained and decomposed into eight independent matrix parameter components, which form a matrix parameter vector. ;

[0090] Each set of standard working distances is bound to its corresponding matrix parameter vector to obtain the geometric mapping relationship between all standardized distances and matrix parameter vectors. .

[0091] Step S3: Based on the two adjacent standard working distances and the geometric mapping relationship, construct a linear parameter fusion interpolation model.

[0092] It should be further explained that, in the specific implementation process, the process of constructing a linear parameter fusion interpolation model based on two adjacent standard working distances and the aforementioned geometric mapping relationship includes:

[0093] Optionally, in this embodiment, the standard working distance set is traversed. Let any two adjacent standard working distances be... , According to standard working distance Standard working distance The corresponding standard working distance range is divided as follows: ;

[0094] For each group of adjacent standard working distance intervals According to the standard working distance range Standard working distance between the two endpoints , The geometric mapping relationship is used to obtain the standard working distance between the two endpoints. , The corresponding 8 independent matrix parameter components are each treated as a one-dimensional variable and denoted as follows: , ;

[0095] Based on the linear interpolation algorithm, and according to the one-dimensional variable and the standard working distance interval Standard working distance corresponding to the two endpoints , Construct a linear parameter fusion interpolation model:

[0096] In the formula, This represents the current working distance.

[0097] Step S4: Obtain the actual working distance of the device, and solve the target perspective transformation matrix corresponding to the actual working distance through the linear parameter fusion interpolation model.

[0098] It should be further explained that, in the specific implementation process, the specific process of solving the target perspective transformation matrix corresponding to the actual working distance through the linear parameter fusion interpolation model includes:

[0099] Optionally, in this embodiment, when the device is working normally, the actual working distance output by the device's ranging module is read. Matching to determine the actual working distance The standard working distance range ; Retrieve the geometric mapping relationship corresponding to the standard working distances at both ends of the standard working distance interval. This leads to the corresponding 8 independent matrix parameter components;

[0100] Based on the linear parameter fusion interpolation model, weighted interpolation is performed to obtain the independent matrix parameter components of the actual working distance, which form the corresponding actual complete parameter vector. ;

[0101] Substitute the independent matrix parameter components from the actual complete parameter vector into the perspective transformation matrix, and satisfy the normalization constraint based on the scale normalization property of the third-order homogeneous homography matrix. The perspective transformation matrix is ​​reconstructed to generate the target perspective transformation matrix for the actual working distance. .

[0102] Optionally, in this embodiment, the actual working distance is determined. There is another matching method for the standard working distance range, specifically:

[0103] Choose any two adjacent standard working distances and denot them as follows: , Set the boundary buffer distance value, denoted as According to standard working distance Standard working distance and boundary buffer distance value , obtain the boundary buffer space When the actual working distance Falling into the boundary buffer zone At that time, based on the actual working distance Between the boundary buffer Two adjacent standard working distances , Near and far dynamic allocation of boundary buffer weights Instead of fixed linear weights, the boundary buffer is retrieved. Standard working distance at both ends Standard working distance The corresponding geometric mapping relationship is determined, and based on the corresponding boundary buffer weights, the eight independent boundary buffer matrix parameter components are obtained. Finally, according to the linear parameter fusion interpolation model, weighted interpolation is performed to calculate the actual working distance. Independent matrix parameter components;

[0104] It should be noted that the boundary buffer distance value It is half the difference between two adjacent standard working distances.

[0105] To further explain, the boundary buffer weight is based on the boundary buffer distance value. It is set, for example, the standard working distance. Standard working distance Therefore, the boundary buffer distance value If the actual working distance at this time This indicates the actual working distance. Compared to standard working distance Closer to the standard working distance Boundary buffer weights for Corresponding standard working distance The boundary buffer weight is ; corresponding boundary buffer independent matrix parameter components The specific unit of distance is not specified and will be determined based on different application scenarios.

[0106] Step S5: Perform perspective transformation correction on the target infrared image or the target visible light image according to the target perspective transformation matrix to achieve registration and alignment between the target infrared image and the target visible light image.

[0107] It should be further explained that, in the specific implementation process, the process of performing perspective transformation correction on the target infrared image or target visible light image based on the target perspective transformation matrix includes:

[0108] Optionally, in this embodiment, based on the target perspective transformation matrix Establish global mapping rules; wherein, the global mapping rules include global mapping relationships from visible light pixel coordinates to infrared pixel coordinates and global mapping relationships from infrared pixel coordinates to visible light pixel coordinates;

[0109] It should be noted that the global mapping relationship from visible light pixel coordinates to infrared pixel coordinates refers to mapping the pixel coordinates of the acquired visible light image to the pixel coordinates of the infrared image; the global mapping relationship from infrared pixel coordinates to visible light pixel coordinates refers to mapping the pixel coordinates of the acquired infrared image to the pixel coordinates of the visible light image.

[0110] To further explain, the pixel coordinate mapping formula for the global mapping rule is as follows:

[0111] ;

[0112] Iterate through all the original pixel coordinates in the image to be registered, line by line. The target pixel coordinates corresponding to the original pixel coordinates are calculated according to the global mapping rules. ;

[0113] The target pixel coordinates are resampled and filled using a bilinear interpolation algorithm to correct geometric distortion of the image to be registered. The specific process is as follows:

[0114] Optionally, in this embodiment, all integer original pixel coordinates of the image to be registered are traversed. The target pixel coordinates in floating-point form are obtained by mapping the target perspective transformation matrix. The target pixel coordinates There is no directly corresponding grayscale value on the integer pixel grid of the image;

[0115] Locate the floating-point target pixel coordinates For each of the four adjacent integer pixels, obtain the original grayscale value of the image corresponding to the four neighboring pixels.

[0116] First, perform linear interpolation on the gray levels of two adjacent pixels along the horizontal axis of the image to obtain the intermediate gray level values ​​of the two horizontal interpolations; then, perform a second linear interpolation on the two intermediate gray level values ​​along the vertical axis of the image to finally obtain the interpolated gray level value corresponding to the floating-point target coordinates.

[0117] The interpolated grayscale value obtained from the interpolation calculation is assigned to the target pixel coordinates of the output image to be registered.

[0118] Optionally, in this embodiment, edge pixels without complete four neighboring points at the boundary of the image to be registered are filled based on boundary grayscale copying to eliminate holes at the image edge.

[0119] Once all pixels of the image to be registered have been filled with grayscale resampling, all stretching, offset, jagged edges, and blank holes caused by perspective transformation in the image to be registered are eliminated. The output completes geometric distortion correction and achieves a registered image that is pixel-aligned with another band image, thus completing pixel-level matching and alignment between the infrared image and the visible light image.

[0120] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0121] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more electronic devices to execute all or part of the steps of the methods described in the various embodiments of this application.

[0123] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed application can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image registration method based on multi-distance calibration and linear fusion of matrix parameters, characterized in that, The method includes: Multiple discrete standard working distances are pre-selected, and the perspective transformation matrix of each standard working distance is solved respectively; the perspective transformation matrix is ​​decomposed into independent matrix parameter components, and then the geometric mapping relationship between the corresponding standard working distance and the independent matrix parameter components is constructed. Based on the adjacent standard working distances and the geometric mapping relationship, a linear parameter fusion interpolation model is constructed; The actual working distance of the device is obtained, and the target perspective transformation matrix corresponding to the actual working distance is solved by the linear parameter fusion interpolation model. The infrared image or visible light image of the target is corrected by perspective transformation according to the target perspective transformation matrix, so as to achieve registration and alignment between the infrared image and the visible light image of the target.

2. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1, characterized in that, The process of solving for the perspective transformation matrix of each standard working distance includes: Calibration targets are placed sequentially at multiple discrete standard working distances, and corresponding infrared calibration images and visible light calibration images are acquired simultaneously to obtain dual-modal calibration image data. The target corner points of the calibration image data are extracted as matching feature point pairs. The corresponding perspective transformation matrix is ​​fitted based on the least squares algorithm, and the RANSAC algorithm is used to remove erroneous feature point pairs.

3. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1, characterized in that, The process of constructing a linear parameter fusion interpolation model based on adjacent standard working distances and the geometric mapping relationship includes: Traverse the set of standard working distances and divide adjacent standard working distance intervals; For each group of standard working distance intervals, based on the geometric mapping relationship of the standard working distances of the two endpoints in the standard working distance interval, obtain the eight independent matrix parameter components corresponding to the standard working distances of the two endpoints, and treat the eight independent matrix parameter components as one-dimensional variables respectively. Based on the one-dimensional variable and the working distance, and the matrix parameters corresponding one-to-one with the eight independent matrix parameter components in the one-dimensional variable, and based on the linear interpolation algorithm, a linear variation model of the eight independent matrix parameter components in the one-dimensional variable with the matrix parameters corresponding one-to-one with the working distance is established, which is denoted as the linear parameter fusion interpolation model.

4. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1 or 3, characterized in that, The process of solving the target perspective transformation matrix corresponding to the actual working distance using the linear parameter fusion interpolation model includes: The geometric mapping relationship corresponding to the standard working distance at both ends of the standard working distance interval is retrieved, and then the corresponding 8 independent matrix parameter components are obtained; According to the linear parameter fusion interpolation model, weighted interpolation calculations are performed based on the eight independent matrix parameter components corresponding to the actual working distance to obtain the independent matrix parameter components of the actual working distance, forming the corresponding actual complete parameter vector; the independent matrix parameter components in the actual complete parameter vector are substituted into the perspective transformation matrix, and the perspective transformation matrix is ​​reconstructed according to the scale normalization property of the third-order homography matrix to generate the target perspective transformation matrix of the actual working distance.

5. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1 or 4, characterized in that, The process of performing perspective transformation correction on the infrared image or visible light image of the target based on the target perspective transformation matrix includes: A global mapping rule is established based on the target perspective transformation matrix; wherein, the global mapping rule includes a global mapping relationship from visible light pixel coordinates to infrared pixel coordinates and a global mapping relationship from infrared pixel coordinates to visible light pixel coordinates; Traverse all the original pixel coordinates in the image to be registered, and calculate the target pixel coordinates corresponding to the original pixel coordinates according to the global mapping rules.

6. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 5, characterized in that, The process of grayscale resampling and filling of target pixel coordinates based on the bilinear interpolation algorithm includes: Locate the four adjacent integer pixels of the target pixel coordinates, and obtain the original grayscale values ​​of the image corresponding to the four neighboring pixels respectively; First, perform linear interpolation on the gray levels of two adjacent pixels along the horizontal axis of the image to obtain the intermediate gray level values ​​of the two horizontal interpolations; then, perform a second linear interpolation on the two intermediate gray level values ​​along the vertical axis of the image to finally obtain the interpolated gray level value corresponding to the floating-point target coordinates. The interpolated grayscale value obtained by interpolation is assigned to the target pixel coordinates corresponding to the output image to be registered; for edge pixels of the image to be registered that do not have complete four neighboring points, the edge pixels are filled based on the boundary grayscale copy, thereby realizing the registration and alignment of the target infrared image and the target visible light image.

7. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1 or 2, characterized in that, For the corner point extraction of the calibration target not being directly aligned with the equipment, the process of constructing an attitude compensation correction model includes: Based on the real-time collected spatial pitch data, yaw attitude data, and the rotation data of the calibration target plane itself, a corresponding overall target attitude rotation matrix is ​​constructed; based on the corresponding standard working distance, the lateral offset of the calibration target to the device, and the longitudinal offset, a translation transformation matrix is ​​constructed; and then the pixel offset of the calibration target is solved. The corresponding attitude compensation and correction model is composed of the overall target attitude rotation matrix, translation transformation matrix, and pixel offset.

8. The image registration method based on multi-distance calibration and linear fusion of matrix parameters according to claim 1 or 4, characterized in that, The process of determining another matching method for the standard working distance range in which the actual working distance falls includes: Choose any two adjacent standard working distances and set the boundary buffer distance value; based on the two adjacent standard working distances and the boundary buffer distance value, obtain the boundary buffer space; when the actual working distance falls into the boundary buffer space, dynamically allocate the boundary buffer weight according to the distance between the actual working distance and the two adjacent standard working distances in the boundary buffer space. The geometric mapping relationship corresponding to the standard working distance between the two ends of the boundary buffer is retrieved, and the corresponding 8 boundary buffer independent matrix parameter components are obtained according to the corresponding boundary buffer weights. Then, according to the linear parameter fusion interpolation model, weighted interpolation calculation is performed based on the 8 boundary buffer independent matrix parameter components corresponding to the actual working distance, thereby generating the target perspective transformation matrix of the actual working distance.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image registration method based on multi-distance calibration and linear fusion of matrix parameters as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image registration method based on multi-distance calibration and linear fusion of matrix parameters as described in any one of claims 1 to 7.

Images