Thermal infrared image registration method based on region segmentation method

Abstract

The invention discloses a thermal infrared image registration method based on a region segmentation method, and belongs to the technical field of image processing. According to the method, the problem of insufficient registration precision caused by lens distortion between different cameras is solved through a dynamic feature point distribution and region segmentation technology. The method comprises the following specific steps: extracting feature points of RGB and infrared images and matching; establishing an efficient search structure based on a KD tree; dividing the image into a plurality of rectangular regions, and evaluating feature point distribution of each region; performing proximity search on a few-point region to enhance the quality of feature points; calculating a homography matrix for each region by using an RANSAC (Random Sample Consensus) algorithm; and detecting and correcting the outlier matrix in combination with PCA analysis. According to the method, non-rigid deformation of lens distortion is approximated through region segmentation, the local registration precision is improved, the algorithm robustness is enhanced, and the method is suitable for industrial production and robot vision systems.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular, to a thermal infrared image registration method based on a region segmentation method applied to industrial production and robot supporting software. Background Art

[0002] In the field of image registration, common methods mainly include feature-based registration methods and intensity-based registration methods.

[0003] Feature-based registration method: It relies on extracting distinguishable feature points, edges, corners and other features from the image, and then aligns the images by matching these features. Its common steps include: using a corner detector (such as Harris, SIFT, SURF) or a neural network method to extract key points; using descriptors (such as SIFT, ORB, etc.) for feature matching; calculating the geometric transformation between the images by aligning the matched feature points. However, the feature points extracted by the feature extraction algorithm relied on by the feature-based registration algorithm are not evenly distributed on the image, and there may even be some incorrect feature points, which will affect the stability of the region segmentation algorithm.

[0004] Intensity-based registration method: It directly uses the intensity information (pixel values) of the image for alignment without relying on feature points, and generally uses an optimization algorithm to find a transformation to make the pixel values of the reference image and the image to be registered as similar as possible. However, this method may not achieve ideal results when dealing with complex situations such as lens distortion.

[0005] For the registration of infrared and RGB images, its main task is to align the images in different spectral ranges so that they correspond to the same scene or target. The algorithm based on whole-image registration is difficult to make the registration matrix fit the high-order non-linear transformation of lens distortion, resulting in poor performance of the registered result in the image distortion area. Summary of the Invention

[0006] The present invention aims to overcome the problem that it is difficult for a single registration matrix to effectively achieve the coordinate transformation between two images due to different degrees of lens distortion between different cameras, and at the same time improve the registration accuracy and the robustness of the algorithm. Specifically, by improving the registration performance in the image distortion area, the infrared and RGB images can be aligned more accurately to meet the requirements of application scenarios such as industrial production and robot supporting software.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] Input data: Receive the RGB image and the infrared image from the cameras at the same position, and there is no limit on the image size. Such a setting can adapt to the images obtained under different shooting devices and scenarios, and improve the versatility of the method.

[0009] Feature extraction: Based on the SuperPoint model (a deep learning method), feature points are extracted from RGB images and infrared images. The SuperPoint model can effectively extract feature points in images, providing a basis for subsequent feature matching.

[0010] Feature pairing process: The LightGlue (a deep learning method) algorithm is used to match the feature points in the two types of images to obtain matching feature point pairs. The LightGlue algorithm can efficiently achieve the matching of feature points and improve the accuracy of matching.

[0011] KD tree construction: A KD tree is constructed based on all paired feature points on the RGB image. The process of constructing the KD tree is as follows: Input a two-dimensional point set, select the x-axis as the initial division dimension, sort by the x coordinate and take the middle point as the division point to create the root node, divide the point set into left and right subsets, and recursively construct subtrees with the y-axis as the division dimension respectively. In each recursion, the x-axis and y-axis are alternately selected as the division dimension until all points are processed. The construction of the KD tree helps with subsequent efficient feature point search and region division.

[0012] Dividing rectangular regions: Perform the template matching algorithm matchTemplate and minMaxLoc in the OpenCV library on the RGB image for the infrared image to obtain the four points of the best matching rectangle, and divide this rectangle into K small rectangles. The division steps include extracting the coordinates and dimensions of the original rectangle, initializing a list to store small rectangles, calculating the number of divisions in the x-axis and y-axis directions, calculating the width and height of the small rectangles, generating small rectangles through two nested loops and adding them to the list, and finally converting the list to a tuple for return. Traverse all feature points, find the matching feature points within each small rectangle, and establish a data structure for storing the position of the small rectangle and the matching feature points therein. By dividing the rectangular regions, the image can be subdivided into multiple small regions for separate processing to better adapt to the non-rigid deformation caused by lens distortion.

[0013] Evaluation stage: Let the total number of matching feature point pairs be P all , calculate the average value P average = P all / K. Traverse all matrices. If any matrix contains fewer than 1 / 3P average points or its number of points is less than 10, then mark this matrix as a matrix with few points. If there is no matrix with few points, directly calculate the homography transformation matrix for each small rectangle based on the matching feature points. This step can screen out areas that may have problems and provide a basis for subsequent processing.

[0014] Enhancement stage: If a sparse point matrix is detected, perform point quality enhancement on the sparse point rectangle. First, find the coordinate center point of the sparse point matrix. Starting from this center point, perform K-nearest neighbor search for a specified point in the KD tree, and set the number of nearest neighbors to search as 1 / 3P average , add the obtained nearest neighbor points to the matrix, and then calculate the homography transformation matrix in the small rectangle area. The process of K-nearest neighbor search for a specified point in the KD tree includes initializing a minimum heap of size K to store the K points closest to the target point and their distances, starting from the root node of the KD tree and recursively searching downwards, calculating the distance between the target point and the current node and processing according to the situation of the heap, determining which side of the current node's splitting dimension the target point is on and recursively searching the corresponding subtree, if the distance between the target point and the splitting hyperplane is less than the distance of the top element of the heap, search the other subtree, backtrack after reaching the leaf node, and check the other subtree according to the conditions during the backtracking process to ensure that all possible neighboring points are checked, and the search ends when backtracking to the root node. Through the enhancement stage, the quality of feature points in the sparse point matrix area can be improved, and thus the registration accuracy can be improved.

[0015] Calculating the homography matrix: Use the standard RANSAC algorithm to calculate the homography matrix for the matching feature point pairs from the RGB and infrared images. The steps of the RANSAC algorithm include assuming that the initial model is applicable to the inliers, testing other data with the model, judging whether the model is reasonable, re-estimating the model with all assumed inliers, and evaluating the error rate of the model. Set the maximum number of iterations of the RANSAC algorithm to 2000 and the matrix confidence to 0.999. Traverse the obtained K small rectangles, and perform the RANSAC algorithm on the matching feature point pairs inside them to obtain K homography matrices. The homography matrix is used to describe the geometric transformation between two planes, and the homography matrix calculated by the RANSAC algorithm can more accurately achieve image registration.

[0016] Outlier matrix detection and processing: Analyze the outlier matrices in the homography transformation matrix group through the principal component analysis method (PCA). The process of PCA includes standardizing the original data set, calculating the covariance matrix, calculating the eigenvalues and eigenvectors of the covariance matrix, selecting the eigenvectors corresponding to the first two largest eigenvalues to form the projection matrix, and transforming the data to the new space. After reducing the matrix to two dimensions, calculate its average coordinates, calculate the distances between all two-dimensional vectors and this coordinate, and judge whether the homography matrix is an outlier matrix according to the distance threshold. The calculation formula for the distance threshold threshold is threshold = μ + 2σ, where μ represents the mean of distance and σ represents the standard deviation of distances. If the distance between a certain matrix and the average coordinates is greater than the distance threshold, then this matrix is an outlier matrix. After finding the outlier matrix, perform a K-nearest neighbor search for this matrix and add points, and recalculate the homography transformation matrix of this matrix. The detection and processing of the outlier matrix can further improve the accuracy and stability of registration.

[0017] Beneficial effects

[0018] Improve registration accuracy: By approximately approaching the non-rigid deformation caused by lens distortion through region segmentation, calculate the homography matrix for each small rectangular region respectively. Compared with the traditional whole-image registration method, it can more accurately fit the high-order non-linear transformation of lens distortion, thereby improving the registration accuracy in the image distortion region.

[0019] Enhance the robustness of the algorithm: Design an algorithm process for the problems of uneven feature point extraction and partial errors. Through the processing of the matrix with few points in the evaluation stage and enhancement stage, as well as the detection and processing of the outlier matrix, the stability and robustness of the algorithm are improved, and the influence of wrong feature points on the registration result is reduced. Description of the drawings

[0020] Figure 1 It is a schematic diagram of the input data.

[0021] Figure 2 It is a schematic diagram of rectangular segmentation of the RGB image.

[0022] Figure 3 It is a schematic diagram of feature point visualization on the RGB image.

[0023] Figure 4 It is a flowchart of the present invention. Detailed implementation manners

[0024] To overcome the problem that it is difficult for a single registration matrix to effectively implement the coordinate transformation between two images due to different degrees of lens distortion among different cameras, and to improve the registration accuracy, the present invention proposes a region segmentation registration method based on dynamic feature point allocation. This method first divides the RGB image into several rectangular regions, and finds the corresponding matrix of each rectangular region on the infrared. Subsequently, perform the KD-TREE algorithm on the detected feature points. For the matrix that does not meet the pre-designed calculation prerequisite conditions, perform the nearest neighbor search of feature points to improve the quality of feature points within the region. Subsequently, calculate a dedicated homography matrix for each RGB-IR matrix pair for registration. By approximately approaching the non-rigid deformation caused by lens distortion through region segmentation, this method improves the registration ability in the region with large lens distortion on the image, and at the same time improves the registration accuracy within each region due to the refined feature point allocation.

[0025] Feature-based registration algorithms rely on feature extraction algorithms. The feature points extracted by feature extraction algorithms are not evenly distributed on the image, and even the feature points extracted by the algorithms may be partially incorrect. To overcome the problem that the above issues may lead to poor stability of the region segmentation algorithm, this solution designs an algorithm process to counteract this influence and improve the algorithm accuracy and robustness.

[0026] The specific technical solution includes:

[0027] 1. Input data: Receive RGB images and infrared images. The RGB images and infrared images should come from cameras at the same location. There are no requirements for the sizes of the RGB images and infrared images.

[0028] 2. Feature extraction: Based on the SuperPoint model (a deep learning method), extract feature points from the RGB images and infrared images.

[0029] 3. Feature pairing processing: Match the feature points in the two images based on the LightGlue (a deep learning method) algorithm to obtain matching feature point pairs.

[0030] 4. KD tree construction:

[0031] Build a KD tree based on all the paired feature points on the RGB image. Since the matching feature point pairs have been obtained, for any paired feature point on the RGB image, its corresponding feature point on the infrared image can be obtained. The specific algorithm process for building the KD tree is as follows:

[0032] (1) Start:

[0033] Input the two-dimensional point set D = {(x1, y1), (x2, y2),..., (xn, yn)}.

[0034] (2) Construct the root node:

[0035] Select the x-axis as the initial partitioning dimension. Sort the point set D by the x coordinate. Select the middle point as the partitioning point, for example, the point (p_x, p_y). Create a KD tree node with the data being the point (p_x, p_y).

[0036] (3) Partition the data set:

[0037] Divide the point set D into two subsets: D_left contains all points with x coordinates less than p_x, and D_right contains all points with x coordinates greater than or equal to p_x.

[0038] (4) Recursively construct subtrees:

[0039] For D_left, select the y-axis as the partitioning dimension and recursively execute the above steps to construct the left subtree.

[0040] For D_right, select the y-axis as the partitioning dimension and recursively execute the above steps to construct the right subtree.

[0041] (5) Repeat recursion:

[0042] In each recursion, alternately select the x-axis and y-axis as the partitioning dimensions. Each recursion creates a new KD-tree node until all points are processed.

[0043] (6) End:

[0044] When all points are processed, the KD-tree construction is complete.

[0045] 5. Partition the rectangular region:

[0046] Execute the template matching algorithm matchTemplate in the OpenCV library on the RGB image for the infrared image, and further execute minMaxLoc to obtain the four points min_val, max_val, min_loc, and max_loc of the rectangle with the best match. Then, divide this rectangle into K small rectangles. The division steps are as follows:

[0047] (1) First, extract the coordinates (x, y) and size (w, h) from the original rectangle rect.

[0048] (2) Initialize an empty list rects to store the divided small rectangles.

[0049] (3) Calculate num_x and num_y. These two variables represent how many parts the rectangle should be divided into in the x-axis and y-axis directions respectively. The function tries to find a more balanced division method (i.e., the aspect ratio of each small rectangle is close) by taking the square root of k.

[0050] (4) If the total number of divisions in the x-axis and y-axis directions is not enough to reach k, increase the number of divisions in the y-axis direction.

[0051] (5) Calculate dx and dy. These two variables represent the width and height of the small rectangles respectively.

[0052] (6) Use two nested loops to generate all the small rectangles. The loop variables i and j represent the positions in the x-axis and y-axis directions respectively.

[0053] (7) For each position (i, j), calculate the top-left coordinates (x0, y0) of the small rectangle and add the coordinates and size of this small rectangle to the list rects.

[0054] (8) If k small rectangles have been generated, exit the inner loop.

[0055] (9) Finally, convert the list rects into a tuple and return it.

[0056] After obtaining the coordinates of the K small matrices, traverse all feature points to find the matching feature points within each small rectangle. The specific steps are as follows:

[0057] (1) First, extract the coordinates (x, y) and size (w, h) of the rectangle from the rect parameter.

[0058] (2) The condition for the point (px, py) to be inside the rectangle is that px is between x and x + w (excluding x + w), and py is between y and y + h (excluding y + h). If these conditions are all met, then the point belongs to the matrix.

[0059] Establish a data structure that is used to save the position of a small rectangle and the matching feature points distributed therein. This data structure will be used in the subsequent processing.

[0060] 6. Evaluation stage:

[0061] Let P all be the total number of pairs of matching feature points. Then the average number of points P average in each small rectangle is P all / K. Traverse all matrices and calculate the number of points they contain. If any matrix contains a number of points less than 1 / 3P average or its number of points is less than 10, then mark this matrix as a matrix with few points. If there is none, then directly calculate the homography transformation matrix for each small rectangle based on the matching feature points.

[0062] 7. Enhancement stage:

[0063] If a matrix with few points is detected in the previous step, perform point quality enhancement on the rectangle with few points. First, find the center point of the coordinates of the matrix with few points. Subsequently, starting from this center point, perform a K-nearest neighbor search for a specified point in the KD tree. The steps are as follows:

[0064] (1) Initialization:

[0065] Create a minimum heap (Min-Heap) of size K to store the K points closest to the target point and their distances. This heap is sorted by distance, and the top element of the heap is the point with the farthest distance.

[0066] (2) Search for K nearest neighbor points:

[0067] Start from the root node of the KD tree and recursively search downward.

[0068] For each visited node, perform the following steps:

[0069] a. Calculate the distance between the target point and the current node.

[0070] b. If the heap is not full (i.e., the number of elements in the heap is less than K), add the current node and its distance to the heap.

[0071] c. If the heap is full and the distance of the current node is less than the distance of the top element of the heap, remove the top element of the heap and add the current node to the heap.

[0072] d. Determine on which side of the splitting dimension of the current node the target point is and recursively search the subtree on that side.

[0073] e. If the distance between the target point and the splitting hyperplane is less than the distance of the top element of the heap, then the subtree on the other side may also contain closer points and needs to be recursively searched.

[0074] (3) Backtracking:

[0075] When the search reaches a leaf node, start backtracking.

[0076] During backtracking, for each parent node, if it is possible to find closer points by crossing the splitting hyperplane of the parent node (i.e., the distance between the target point and the splitting hyperplane is less than the distance of the top element of the heap), then the subtree on the other side also needs to be checked.

[0077] (4) Checking boundaries:

[0078] During backtracking, if the heap is not full, ensure that all possible neighboring points are checked.

[0079] If the heap is full, only check the points that may be closer than the top element of the heap.

[0080] (5) Termination:

[0081] When backtracking reaches the root node, the search ends.

[0082] The elements in the heap are the K points closest to the target point.

[0083] (6) Result:

[0084] Take out all the elements from the heap, and these elements are the K nearest neighbors of the target point.

[0085] In the present invention, set the number of nearest neighbors searched to 1 / 3P average , after the K-nearest neighbor search is completed, add the obtained nearest neighbor points to the matrix, and then calculate the homography transformation matrix in the small rectangular area.

[0086] 8. Calculate the homography matrix

[0087] The Homography Matrix is mainly used to describe the geometric transformation between two planes. The Homography Matrix is a 3×3 matrix that can be used to represent the projective mapping from one plane to another.

[0088] If H is the Homography Matrix, point p is on the first image plane, and point p′ is on the second image plane, then their relationship can be expressed as:

[0089] p′ ∼ Hp

[0090] where ∼ means that the two vectors are proportional in homogeneous coordinates. Here, both p and p′ are homogeneous coordinate vectors.

[0091] The calculation of the Homography Matrix requires point sets from two different planes. The present invention uses the standard RANSAC (RAndom SAmple Consensus) algorithm to calculate the Homography Matrix for the matching feature point pairs from RGB and infrared images.

[0092] The steps of RANSAC are as follows:

[0093] (1) An initial model is applicable to the assumed inliers, that is, all unknown parameters can be calculated from the assumed inliers.

[0094] (2) Use the model obtained in step (1) to test all other data. If a point is applicable to the estimated model, it is considered an inlier.

[0095] (3) If enough points are classified as the assumed inliers, then the estimated model is reasonable enough.

[0096] (4) Then, use all the assumed inliers to re - estimate the model because it has only been estimated by the initial assumed inliers.

[0097] (5) Finally, evaluate the model by estimating the error rate between the inliers and the model.

[0098] Set the maximum number of iterations of the RANSAC algorithm to 2000 and the matrix confidence to 0.999.

[0099] Now traverse the obtained K small rectangles, perform the RANSAC algorithm on the matching feature point pairs inside them, and after completion, obtain K Homography Matrices.

[0100] 9. Outlier Matrix Detection and Processing:

[0101] An outlier matrix refers to a matrix whose ability to transform coordinates is significantly different from that of most other matrices. The reason for the appearance of an outlier matrix may be that there are few points in the region or the quality of the points is poor, resulting in a poor and unreliable calculated matrix. In the present invention, the principal component analysis method (PCA) is used to analyze the outlier matrices in the homography transformation matrix group.

[0102] Principal component analysis is a statistical method that transforms a set of potentially correlated variables into a set of linearly uncorrelated variables through an orthogonal transformation. This set of variables is called the principal components. The process of using PCA to reduce the dimension of a matrix to a 2D vector in the present invention is as follows:

[0103] (1) Data standardization: The original data set is standardized to ensure that the mean of each feature is 0 and the standard deviation is 1. This step is to eliminate the influence of different dimensions and numerical magnitudes on the results.

[0104] (2) Calculate the covariance matrix: The covariance matrix can reflect the correlation between data features. The calculation formula of the covariance matrix is where X is the standardized data matrix and n is the number of samples.

[0105] (3) Calculate the eigenvalues and eigenvectors of the covariance matrix: Perform eigenvalue decomposition on the covariance matrix to find the eigenvalues and corresponding eigenvectors. The eigenvalues represent the variance of the data in the directions of the eigenvectors, while the eigenvectors represent the distribution of the data in these directions.

[0106] (4) Select the principal components: Arrange them in descending order according to the magnitudes of the eigenvalues, and select the eigenvectors corresponding to the two largest eigenvalues. These two eigenvectors are the new coordinate axes of the data, which can retain the variance information of the data to the greatest extent.

[0107] (5) Form the projection matrix: Combine the two selected eigenvectors into a 2-column matrix, which is the projection matrix W.

[0108] (6) Transform the data into the new space: Use the projection matrix W to transform the original data, that is, Y = XW, where Y is the data after dimension reduction and it has only two dimensions.

[0109] After reducing the matrix to two dimensions, calculate its average coordinates The calculation formula is as follows:

[0110]

[0111] where n is the total number of matrices, and x and y are the components of the corresponding 2D vectors of the matrix on the X-axis and Y-axis respectively.

[0112] Obtain the average coordinates After that, calculate the distances between all two-dimensional vectors and this coordinate. The calculation formula for the distance distance:

[0113]

[0114] After obtaining the distances between the two-dimensional vectors and the average coordinate, determine whether the homography matrix corresponding to the two-dimensional vector is an outlier matrix according to the distance threshold. The calculation formula for the distance threshold threshold:

[0115] threshold = μ + 2σ

[0116] where μ represents the mean of distance, and σ represents the standard deviation of distances. If the distance between a certain matrix and the average coordinate is greater than the distance threshold, then this matrix is an outlier matrix. After finding the outlier matrix, perform a K-nearest neighbor search for this matrix, add points of 1 / 2P average and recalculate the homography transformation matrix of this matrix.

[0117] 10. End

[0118] After the above process is completed, K homography transformation matrices can be obtained, which can be used for the registration operation.

Claims

1. A thermal infrared image registration method based on a region segmentation method, characterized in that Including the following steps: Extract the feature points of the RGB image and the thermal infrared image, and perform matching to obtain the matching feature point pairs; Based on the matching feature point pairs, establish a KD-tree structure and divide the image into multiple rectangular regions; Evaluate the number of feature points in each rectangular region, and perform a proximity search on the low-point regions to enhance the quality of the feature points; Use the RANSAC algorithm to calculate the homography matrix for each region; Detect the outlier matrix through principal component analysis (PCA), and recalculate the corrected homography matrix.

2. The method according to claim 1, wherein The feature extraction uses the SuperPoint deep learning model, and the feature matching uses the LightGlue algorithm.

3. The method according to claim 1, wherein The determination condition for the low-point region is that the number of feature points in the region is less than 1 / 3 of the total average or less than 10.

4. The method according to claim 1, wherein The proximity search is based on the KD-tree structure, and the search number is 1 / 3 of the average value of the feature points in the low-point region.

5. The method according to claim 1, wherein The detection of the outlier matrix is determined by calculating the distance between the matrix and the average coordinates in the two-dimensional projection space and based on the threshold μ + 2σ.

6. The method according to claim 1, wherein In the calculation of the homography matrix, the maximum number of iterations of the RANSAC algorithm is set to 2000, and the confidence level is 0.

999.

7. The method according to claim 1, wherein The region division uses the template matching algorithm of the OpenCV library to determine the initial rectangle and divides it into K sub-rectangles.

Images