A cross-modal feature point matching method between infrared and RGB

Abstract

The present invention relates to a method for matching infrared and RGB cross-modal feature points, comprising: offline training a deep learning model based on collected original RGB and IR images to obtain a trained matching model; inputting the data to be tested into the matching model to extract feature descriptors and output corresponding matching results. Compared with the existing technology, the present invention focuses on the fusion of multispectral images, fusing visible light images (RGB) and thermal imaging images (IR). Through model training, it is possible to accurately extract feature points in a multimodal environment, better perform cross-modal feature matching tasks, and thus improve the accuracy of camera pose estimation in scenes with drastic lighting changes and in dark scenes. It can provide a reliable perception front end for many applications, pave the way for subsequent research on the fusion of multispectral sensors under the traditional SLAM framework, and will also facilitate the mapping, positioning, matching, depth estimation, and three-dimensional mapping of the same scene across daytime (based on RGB images) and nighttime (based on IR images).

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method for matching infrared and RGB cross-modal feature points. Background Art

[0002] Feature extraction is crucial for autonomous driving perception tasks. However, in environments with drastic lighting changes, even complete darkness and inclement weather, traditional machine vision often faces large feature extraction errors or even failure. For example, in tasks like SLAM (simultaneous localization and mapping), SfM (Structure from motion), camera calibration, and image registration, the extracted features primarily consist of points of interest. Poor quality or a small number of points of interest extracted due to environmental influences inevitably lead to subsequent feature point matching failures, making subsequent computations impossible.

[0003] Traditional feature point extraction methods, including SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features), and ORB (Oriented Fast and Rotated Brief), are mostly based on relatively stable local image features. While these traditional methods offer high precision in manual point extraction, they lack robustness in many scenarios. For example, in dark and low-texture scenes, pixel gradient information is minimal and noisy, making it difficult to accurately and quickly extract feature points for matching.

[0004] With the development of deep learning networks, deep learning-based feature point extraction methods have begun to emerge. However, these methods are all targeted at a single modality and do not consider the feature differences between modalities. Due to the perception defects of visible light cameras in low-light scenes, the accuracy of camera pose estimation cannot be guaranteed in scenes with drastic lighting changes and darkness. This poses a huge challenge to mapping, positioning, matching, depth estimation, and 3D mapping of the same scene across day and night. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an infrared and RGB cross-modal feature point matching method, so as to accurately extract feature points in multimodality and better match cross-modal feature matching tasks, thereby improving the accuracy of camera pose estimation in scenes with drastic lighting changes and dark scenes, which is conducive to achieving mapping, positioning matching, or depth estimation and three-dimensional mapping of the same scene across day and night.

[0006] The purpose of the present invention can be achieved by the following technical solution: A method for matching infrared and RGB cross-modal feature points, comprising the following steps:

[0007] S1. Collect original RGB images and original IR images, and perform offline training on the deep learning model based on the original RGB images and original IR images to obtain a trained matching model;

[0008] S2. Input the data to be tested into the trained matching model to extract feature descriptors and output the corresponding matching results.

[0009] Furthermore, the step S1 specifically collects the original RGB image through the visible light camera and collects the original IR image through the thermal imaging camera.

[0010] Furthermore, the deep learning model in step S1 is specifically an UnsuperPoint neural network model.

[0011] Furthermore, the specific process of offline training in step S1 is as follows:

[0012] S11, preprocessing the collected original RGB image and original IR image to obtain paired images;

[0013] S12. Construct an UnsuperPoint neural network model, input the paired images into the UnsuperPoint neural network model for offline training, and obtain a trained matching model.

[0014] Furthermore, the step S11 specifically performs pixel alignment processing on the original RGB image and the original IR image to ensure that the original RGB image and the original IR image are completely aligned at the pixel level.

[0015] Furthermore, the paired images are specifically an original RGB image and an IR image with perspective transformation added.

[0016] Furthermore, the paired images are specifically the original IR image and the RGB image with perspective transformation added.

[0017] Furthermore, the UnsuperPoint neural network model constructed in step S12 includes a backbone network, which is used to perform the joint tasks of point confidence estimation, point coordinate regression, and descriptor extraction. The backbone network is divided into two branches: one branch is used to process the original image, and the other branch is used to process the image after the homography matrix transformation. The extracted points are projected into the same image coordinate system through the true value of the homography matrix, and the distance between each pair of points is calculated. Point pairs with a distance less than 4 pixels are regarded as valid point pairs, and point correspondences are constructed for self-supervised learning.

[0018] The UnsuperPoint neural network model uses a convolutional network layer with a convolution kernel size of 3 and a stride of 2 to process edge blur caused by temperature radiation in IR images.

[0019] Furthermore, the learning loss function of the UnsuperPoint neural network model is specifically:

[0020] L=α score L score +α rep L rep +α pos L pos +α uni L uni +α des L des +α des_coor L des_coor

[0021]

[0022]

[0023] sim(z i , z j )=z i T z j / ||z i ||||z j ||

[0024] Among them, A is the logo of RGB image, B is the logo of IR image, L is the total loss function, L score is the point confidence loss, which is represented by the square of the score difference between the same points in A and B, α score For L score corresponding weights;

[0025] L rep is the repeatability loss based on the point pair distance, s is the confidence of the extracted point, d is the distance between the point pairs, is the mean distance of all point pairs, α rep For L rep corresponding weights;

[0026] L pos is the point-to-point Euclidean distance loss, α pos For L pos corresponding weights;

[0027] L uni is the loss of coordinate homogenization, α uni For L uni corresponding weights;

[0028] L des is the descriptor loss, which is represented by the square of the difference between the descriptors of the same points A and B, α des For L des corresponding weights;

[0029] L des_coor Used to improve the compactness of the descriptor in space. The loss is represented by the sum of the correlation coefficients of the descriptors at different positions. des_coor For L des_coor corresponding weights;

[0030] z i , z j are two descriptor vectors, sim(z i , z j ) is z i , z j The similarity calculation value of , τ is the temperature hyperparameter used to control the intensity of learning negative examples.

[0031] Furthermore, the data to be tested includes a pair of an RGB image to be tested and an IR image to be tested.

[0032] Compared to existing technologies, this invention focuses on the fusion of multispectral images, specifically, the integration of visible light cameras and thermal imaging cameras. This approach leverages the thermal imaging camera's indifference to illumination variations to compensate for the visible light camera's perception deficiencies in low-light scenarios. Based on a neural network, this invention learns a multimodally adaptable extraction point and descriptor model. This paves the way for subsequent research on the fusion of multispectral sensors within the traditional SLAM framework, improving the accuracy of camera pose estimation in scenes with drastic illumination changes and darkness. This facilitates mapping, positioning, matching, depth estimation, and 3D mapping of the same scene across day and night.

[0033] The present invention mixes RGB and IR data sets to train a model, so that the matching model obtained after training can perform feature matching in both RGB and IR modes. Through this training mode, the model with dual-modal feature matching function will have significantly improved matching accuracy in both modalities. At the same time, a large amount of data is used in unsupervised learning tasks, and the model trained with large amounts of data is more robust in the tasks of extracting features and defining repeated points.

[0034] This invention is based on the UnsuperPoint network model infrastructure, and achieves complete unsupervised operation through self-supervision of image pairs, eliminating the dependence of SuperPoint pre-training on synthetic data; at the same time, the position is used as a regression term to make it differentiable, so that the position can be optimized and moved; in order to solve the problem of feature points appearing at the grid boundary, a heuristic distribution approximation algorithm is adopted, through which the distribution of feature points is made more uniform; in addition, the cross-correlation between the various dimensions of the descriptor is reduced to reduce the redundancy between the various dimensions of the descriptor, thereby improving the expressive power of the descriptor; by making innovative adjustments such as using convolution instead of maximum pooling and changing the learning loss function, it can better match the cross-modal feature matching task. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the method flow of the present invention;

[0036] Figure 2 This is a schematic diagram of the architecture of the deep learning network in the embodiment;

[0037] Figure 3 Schematic diagram of the self-supervised learning framework;

[0038] Figure 4 Example diagrams of descriptor matching under different modalities. DETAILED DESCRIPTION

[0039] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Example

[0041] like Figure 1 As shown, a method for matching infrared and RGB cross-modal feature points includes the following steps:

[0042] S1. Collect original RGB images and original IR images, and perform offline training on the deep learning model based on the original RGB images and original IR images to obtain a trained matching model;

[0043] Since the UnsuperPoint model has already proven its excellent performance in RGB mode, this embodiment mainly uses this model as the main framework (e.g. Figure 2 As shown in ), based on it, it is adjusted for the cross-modal feature matching task;

[0044] In this embodiment, the UnsuperPoint neural network model is used for offline training. The specific process of offline training is as follows:

[0045] S11. Preprocessing the collected original RGB image (which can be collected by a visible light camera) and the original IR image (which can be collected by a thermal imaging camera) to obtain paired images, where the paired images specifically include the original RGB image and the IR image with a perspective transformation, or the original IR image and the RGB image with a perspective transformation, and the original RGB image and the original IR image are completely aligned at the pixel level;

[0046] It should be noted that the model training process in this invention is the same as the model training idea of ​​UnsuperPoint under single modality. According to the idea under single modality, the transformation relationship between the original image and the transformed image is randomly generated and known. Therefore, the input data under cross-modality must ensure that the original RGB and original IR images are completely aligned at the pixel level to achieve cross-modal matching;

[0047] S12. Construct an UnsuperPoint neural network model and input paired images into the UnsuperPoint neural network model for offline training to obtain a trained matching model. The UnsuperPoint neural network model includes a backbone network. UnsuperPoint demonstrates excellent performance in RGB mode. The points extracted by this network avoid artificially defining corners or image gradients, achieving good performance in terms of repeatability and positioning error. However, because infrared information originates from thermal radiation, the edge features of infrared imaging are noisy and can differ significantly from the edge features of RGB images. Therefore, selecting key points based on repeatability, rather than visual corner or edge information, can minimize errors caused by noise.

[0048] UnsuperPoint is composed of a lightweight network to extract low-level features. The network is used to perform the joint tasks of point confidence estimation, point coordinate regression and descriptor extraction. For each 8×8 grid at the original image scale, the network outputs a point and its descriptor. The self-supervised learning process is as follows Figure 3 As shown in Figure 2, the network is divided into two branches: one branch processes the original image, and the other branch processes the image after the homography matrix transformation. The extracted points are projected into the same image coordinate system using the true value of the homography matrix. The distance between each pair of points is calculated. Point pairs with a distance less than 4 pixels are considered valid point pairs, and point correspondences are constructed for self-supervised learning.

[0049] At the same time, considering that the original UnsuperPoint network uses a parameter-free network structure called maximum pooling for downsampling, and that images in the infrared mode will have blurred edges due to temperature radiation, in order to allow the network to better learn the underlying features without excessively affecting computing resources, the present invention uses a convolutional network layer with a convolution kernel size of 3 and a stride of 2 to process edge blur;

[0050] In addition, in terms of learning loss functions, the descriptor loss part adopts the contrastive learning loss SimCLR based on negative examples. SimCLR is a self-supervised learning framework that autonomously constructs positive and negative examples for contrastive learning. This loss function adopts a twin structure and constructs positive examples through transformation enhancement, forcing the network to learn certain invariant properties in image features. In the descriptor learning task, the positive example is the point pair descriptor whose distance is within the threshold. Therefore, the contrastive loss can be used to encourage higher similarity between positive examples, while expanding the distance with any negative examples. The negative examples are used to supervise the network to retain independent features, thereby preventing model collapse, that is, to make the negative example features as evenly distributed as possible. For two descriptor vectors z i ,z j The similarity calculation is obtained by the dot product of the vector after regularization and normalization:

[0051] sim(z i ,z j )=z i T z j / ||z i ||||z j ||

[0052] The descriptor loss function is obtained as:

[0053]

[0054] Where, is a prompt that is multiplied by the subsequent natural exponential operation. If k is not equal to i, the prompt's value is 1, otherwise it is 0. τ is the temperature hyperparameter, which controls the intensity of learning negative examples. First, negative examples also vary, ranging from relatively similar ones to completely unrelated ones. Therefore, learning from spatially close negative examples is more difficult, and it is more difficult to distance negative examples. The temperature hyperparameter adjusts the penalty distribution for difficult negative examples. A smaller setting for this hyperparameter emphasizes pushing difficult negative examples further away, making the spatial distribution of samples more uniform. However, a smaller temperature hyperparameter is not necessarily better. When learning descriptors, the network determines whether a descriptor pair is positive or negative based on the distance prior between the point pairs. If the extraction points are incorrectly positioned in the early stages of network learning, setting the temperature hyperparameter too low may push descriptors that should actually be similar apart, making it more difficult to close their spatial distances later.

[0055] The learning loss function of the UnsuperPoint neural network model in the present invention is specifically:

[0056] L=α score L score +α rep L rep +α pos Lpos +α uni L uni +α des L des +α des_coor L des_coor

[0057] Where, L score Point confidence loss is used to ensure that the scores of the same points in image A (RGB image) and image B (infrared image) are similar, that is, the confidence of the points under different viewing angles is consistent. This loss is represented by the square of the difference in the scores of the same points in image A and image B.

[0058] L rep It is a repeatability loss based on point-to-point distance, and its loss function is:

[0059]

[0060] Here we consider the twin graph of A after homography transformation, s is the confidence of the extracted point. Here we only consider the point pairs with a pixel distance less than 4, d is the distance between the point pairs, is the mean distance of all point pairs. This loss is to reduce the confidence of the point when the distance is large and to increase the confidence of the point when the distance is small.

[0061] L pos It is the Euclidean distance loss of the point pair. Its main goal is to ensure that the key point positions detected in image A (after homography transformation) are the same as those detected in image B, that is, the stability of the point positions under different perspectives, which makes the point pairs closer. uni It is the loss of coordinate homogenization. Here, only the points in the 8×8 grid are considered. When calculating this loss, the points are sorted by coordinates and the variance of the interval between the coordinates is calculated as the loss. des L is the descriptor loss, which is represented by the square of the difference between the descriptors of the same point in image A and image B. This loss ensures that the descriptors of the point pairs should be similar. des_coor It is used to improve the compactness of the descriptor in space. The loss is represented by the sum of the correlation coefficients of the descriptors at different positions.

[0062] α score For L score The corresponding weight; α rep For L rep The corresponding weight; α pos For L pos The corresponding weight; α uni For L uni The corresponding weight; α des For L des The corresponding weight; α des_coor For Ldes_coor The corresponding weights.

[0063] Therefore, the present invention improves the UnsuperPoint network structure by adopting convolutional layers to process edge blur and contrast loss to improve matching and descriptor extraction quality.

[0064] S2. Input the data to be tested into the trained matching model to extract feature descriptors and output corresponding matching results. Similarly, the data to be tested includes a pair of RGB images to be tested and IR images to be tested.

[0065] In summary, this technical solution uses visible light cameras and thermal imaging cameras to collect initial images, and trains the network model offline based on the collected initial image data; then the data set to be tested is fed into the trained model, feature descriptors are extracted, and matching is performed. Figure 4 In the figure, the left side is the RGB image and the right side is the IR image. Figure 4 The figure shows an example of descriptor matching between the RGB modality and the IR modality in this embodiment.

[0066] The model training process in the present invention is the same as the model training idea under single modality. The difference is that the input pair of images is the original RGB image and the IR image with added perspective transformation (or the original IR image and the RGB image with added perspective transformation), and it must be ensured that the original RGB and the original IR images are completely aligned at the pixel level.

[0067] The qualitative indicators in training are expressed as follows: RS is the repetition rate, LE is the position error, HE is the homography transformation estimation error, where ∈ is the threshold of the average error after homography transformation of the four corner points, and MS is the matching score.

[0068] The robustness of the extracted points is evaluated by the repetition rate. Let O represent the original image, C represent the transformed image, and the transformation matrix is ​​known. At this time, the points extracted from O are transformed to the perspective of C by the homography transformation matrix, which is recorded as P true_warped , the point extracted from C is recorded as P warped , the repetition rate calculation can be written as:

[0069]

[0070] The distance threshold is 3 pixels, and points smaller than the distance threshold are considered to be matching point pairs.

[0071] Another evaluation index is the position error, which evaluates the accuracy of the extracted point position. It is the same as the repetition rate calculation. At this time, the points with a distance less than the threshold are recorded as G true_warped and G warped , the position error is calculated as:

[0072]

[0073] Descriptors cannot be evaluated individually because the evaluation method follows the same principle of first detecting points and then calculating descriptors. Therefore, the performance of both extracted points and descriptors can only be evaluated comprehensively. This comprehensive evaluation is performed using homography estimation. Descriptors are first matched using brute force matching, with L2 distance used to measure their similarity. The matching results are then combined with the RANSAC (Random Sample Consensus) algorithm to estimate the homography matrix between the two images.

[0074] The matching score mainly reflects the performance of the descriptor. RANSAC will screen the inliers (corresponding to matching point pairs within the error range of the homography matrix) and outliers in the sample. The outliers are matching point pairs that cause excessive errors in the calculation of the homography matrix. The matching score calculation can be written as:

[0075]

[0076] Homography accuracy comprehensively evaluates the accuracy of extracted point positions and matching performance. Given the coordinates of four edge points of a given image size, the average error distance between the four points under the estimated homography and the true value of the transformation matrix is ​​calculated. The accuracy of the estimate is determined using different thresholds. Finally, accuracy is evaluated by the proportion of correct estimates, a metric labeled HE. In this example, thresholds of 1, 3, 5, and 10 pixels are used as evaluation thresholds.

[0077] Regarding the choice of backbone network, different from single-modal features, the pair of features output by the backbone network under cross-modality are RGB modality and IR modality. UnsuperPoint's backbone network is very lightweight. Unlike more complex network architectures such as VGG or ResNet, its migration performance may be relatively poor, and it cannot be used as a backbone network shared across modalities. Therefore, the present invention mixes RGB and IR data sets to train a model, that is, the model can perform feature matching in each modality under the two modalities of RGB and IR. Through this training mode, the model with dual-modal feature matching function will have a significant improvement in matching accuracy in the two modalities. At the same time, a large amount of data is used in unsupervised learning tasks, and the model trained with large amounts of data is more robust in the tasks of extracting features and defining repeated points.

[0078] This invention can provide a reliable perception front end for autonomous driving, paving the way for subsequent research on the integration of multispectral sensors within the traditional SLAM framework, thereby improving the accuracy of camera pose estimation in scenes with dramatic lighting changes and in darkness. This work will also facilitate mapping, positioning, matching, or depth estimation and three-dimensional mapping of the same scene across day and night. For example, it can address the failure of RGB cameras to track feature points when lighting changes dramatically. Using cross-modal feature matching, multimodal sensor conversion can be achieved, reducing the impact of lighting on the stability of the SLAM system.

Claims

1. A method for matching infrared and RGB cross-modal feature points, characterized in that: The following steps are involved: S1. Collect original RGB images and original IR images, and perform offline training on the deep learning model based on the original RGB images and original IR images to obtain a trained matching model; S2. Input the data to be tested into the trained matching model to extract feature descriptors and output the corresponding matching results; The deep learning model in step S1 is specifically an UnsuperPoint neural network model; The specific process of offline training in step S1 is: S11, preprocessing the collected original RGB image and original IR image to obtain paired images; S12, constructing an UnsuperPoint neural network model, inputting paired images into the UnsuperPoint neural network model for offline training, and obtaining a trained matching model; Step S11 specifically performs pixel alignment processing on the original RGB image and the original IR image to ensure that the original RGB image and the original IR image are completely aligned at the pixel level. The paired images specifically include the original RGB image and the IR image with the perspective transformation, or the original IR image and the RGB image with the perspective transformation. The UnsuperPoint neural network model constructed in step S12 includes a backbone network, which is used to perform the joint tasks of point confidence estimation, point coordinate regression, and descriptor extraction. The backbone network is divided into two branches: one branch is used to process the original image, and the other branch is used to process the image after the homography matrix transformation. The extracted points are projected into the same image coordinate system through the true value of the homography matrix, and the distance between each pair of points is calculated. Point pairs with a distance less than 4 pixels are regarded as valid point pairs, and point correspondences are constructed for self-supervised learning. The UnsuperPoint neural network model uses a convolutional network layer with a convolution kernel size of 3 and a stride of 2 to handle edge blurring caused by temperature radiation in IR images. The learning loss function of the UnsuperPoint neural network model is specifically: ， ， ， ， in, A is the identifier of the RGB image, B is the identifier of the IR image, is the total loss function, is the point confidence loss, which is given by A and B The square of the score difference of the same point is expressed as, for corresponding weights; is the repeatability loss based on the point-to-point distance, is the confidence of the extracted point, is the distance between the point pairs, is the mean distance of all point pairs, for corresponding weights; is the Euclidean distance loss of the point pair, for corresponding weights; is the loss of coordinate homogenization, for corresponding weights; is the descriptor loss, which is determined by A and B The square of the difference between the descriptors of the same point is expressed as, for corresponding weights; Used to improve the compactness of the descriptor in space. The loss is represented by the sum of the correlation coefficients of the descriptors at different positions. for corresponding weights; are two descriptor vectors, for The similarity calculation value of is a temperature hyperparameter that controls the intensity of learning negative examples.

2. The infrared and RGB cross-modal feature point matching method according to claim 1, characterized in that: Specifically, step S1 involves collecting the original RGB image through a visible light camera and collecting the original IR image through a thermal imaging camera.

3. The infrared and RGB cross-modal feature point matching method according to claim 1, characterized in that: The data to be tested includes a pair of an RGB image to be tested and an IR image to be tested.

Images