Multi-source image registration method based on deep learning

Abstract

The invention provides a multi-source image registration method based on deep learning, and relates to the technical field of image processing. The method comprises the following steps: firstly, respectively carrying out image preprocessing on infrared and visible light images; then constructing and training a feature point extraction model, and respectively extracting feature points and descriptors from the preprocessed infrared and visible light images; carrying out feature matching on the extracted feature points by adopting a feature point matching model to obtain high-precision matching point pairs; robust matching point screening and registration optimization are carried out on the matching point pairs, and a geometric transformation matrix between the infrared image and the visible light image is calculated; and finally, performing geometric transformation on each pixel point of the infrared image through the geometric transformation matrix between the infrared image and the visible light image, and converting the pixel points into a coordinate system of the visible light image to obtain a final registration result of the infrared image and the visible light image. According to the method, when the modal difference between the multi-modal images is large, a stable registration effect can still be obtained, and the robustness of multi-modal image registration is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-source image registration method based on deep learning. Background Art

[0002] Image registration is a key technology in the field of computer vision and image processing. Its goal is to align multiple images acquired at different times, different perspectives, or different sensors so that they are spatially consistent. This process has a wide range of applications in medical imaging, remote sensing image processing, computer graphics, and other fields. In medical imaging, image registration can be used for image fusion of different modalities, such as aligning CT images with MRI images to provide more comprehensive diagnostic information; in remote sensing image processing, image registration can be used for change detection in multi-temporal images to monitor environmental changes or disaster assessment.

[0003] In recent years, with the increasing demand for multi-source image fusion, multi-source image registration methods have received extensive attention and development. The purpose of multi-source image fusion is to integrate image information from different sensors to generate images containing more comprehensive information, thereby enhancing the image's explanatory power and application value. In this process, image registration is a crucial step, which determines the quality and effect of the fused image. Using traditional algorithms (including mutual information-based algorithms, phase correlation algorithms, and image gradient information algorithms) to solve this problem will face severe challenges, mainly manifested in: (1) It is necessary to traverse a large number of pixel values, and the process of calculating mutual information is very time-consuming, which is difficult to meet the requirements of real-time registration; (2) The phase correlation method assumes that there is a rigid transformation between images, and the effect is poor when there are transformations such as rotation and scaling; (3) In multi-modal images, gradient features may be inconsistent, especially the edge responses of infrared images and visible light images will be very different; (4) Although these traditional methods perform well in single-modal registration, they are usually not robust enough when facing multi-modal images due to factors such as sensor differences, noise interference, and initial alignment errors. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a multi-source image registration method based on deep learning. By combining the feature extraction and matching technology of deep learning, the problems of unstable feature matching and low registration accuracy in the registration of infrared images and visible light images are solved.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-source image registration method based on deep learning, comprising the following steps:

[0006] Step 1: Perform image preprocessing on infrared and visible light images respectively to reduce modality differences and enhance the consistency of image features;

[0007] Step 1.1: Use Gaussian filtering to smooth the infrared image and reduce the noise between the infrared image and the visible light image;

[0008] Step 1.2: The filtered infrared image and visible light image are normalized separately so that the pixel values ​​of the two images are normalized to the same range to reduce the influence of the difference in light intensity and sensor response;

[0009] Step 2: Build and train a feature point extraction model to extract feature points and descriptors from the preprocessed infrared and visible light images respectively;

[0010] The feature point extraction model extracts feature information from the infrared and visible light images after standardization respectively; the feature information includes feature points and descriptors;

[0011] The feature point extraction model adopts the SuperPoint network, which extracts feature points and descriptors from the preprocessed infrared and visible light images respectively; the SuperPoint network includes a shared encoder and two decoders; the two decoders are respectively an interest point decoder and a descriptor decoder; in the feature point extraction process, the interest point decoder is self-supervised trained and fine-tuned to further improve the feature point detection capability of the feature point extraction model in infrared and visible light multimodal images;

[0012] First, the interest point decoder is used as a pre-training model. Then, a small number of labeled samples are manually marked in the aligned infrared and visible light image datasets as supervisory signals for preliminary training. In order to make full use of the infrared and visible light image datasets, a self-supervised learning algorithm is used to further train unlabeled samples based on labeled samples, and the locations of feature points are predicted using unlabeled samples. During the training process, a loss function is designed based on the consistency of feature points at corresponding positions in infrared and visible light images, and the feature point detection results are optimized by minimizing the difference in the locations of feature points in the two images.

[0013] The loss function is defined as follows:

[0014]

[0015] in, and Respectively represent the corresponding feature point positions in infrared and visible light images;

[0016] Through multiple rounds of iterative training, the model weights are continuously optimized so that the feature point extraction model can better predict feature points on unlabeled samples. Finally, after fine-tuning, the optimized weights are applied to feature point detection in infrared and visible light images, significantly improving the performance of the feature point extraction model in actual scenarios.

[0017] Step 3: Use the feature point matching model to perform feature matching on the feature points extracted in step 2 to obtain high-precision matching point pairs;

[0018] The feature point matching model uses the LightGlue network to perform feature matching on feature points, and replaces the global self-attention mechanism with local self-attention. In the self-attention unit, the feature point of each image only interacts with the top K feature points that are most similar in space in the image.

[0019] In the self-attention unit, each feature point in an image obtains information only from the top K most spatially similar feature points in the same image; by calculating the query vector q for each point i and key vector k i Linear transformation of , and obtain the self-attention score between feature points:

[0020]

[0021] Among them, R(p j -p i ) is the point p j With p i The rotation encoding of the relative position between ij is a weight coefficient used to represent the local correlation of adjustment point i to feature point j;

[0022] In the cross-attention unit, the points in each image obtain information from the points in another image, and only the similarity between the key vectors is calculated to obtain the cross-attention score between the feature points:

[0023]

[0024] in, are the key vectors of infrared image feature point i and visible light image feature point j, is the weight of infrared image point i and visible light image point j, indicating the local similarity between them;

[0025] After several layers of self-attention and cross-attention units, the feature point matching network generates a set of updated feature representations;

[0026] Then the matching relationship is predicted by calculating the similarity score between points;

[0027] First, calculate the similarity score for each pair of points in image A and image B:

[0028]

[0029] Among them, S ij The midpoint of image A and the midpoint of image B Similarity score, Linear() is a linear transformation with a bias term;

[0030] Then assign a matching score σ to each point i , used to indicate whether the point can be matched:

[0031] σ i =Sigmoid(Linear(x i )) (5)

[0032] The score represents the probability of finding a match for point i in another image;

[0033] Finally, the similarity score and the matching score are combined to generate a matching probability matrix M, which represents the matching probability M of each pair of points. ij :

[0034]

[0035] Step 4: Perform robust matching point screening and registration optimization on the matching point pairs and calculate the geometric transformation matrix H between the infrared and visible light images;

[0036] The MAGSAC algorithm is used to perform robust matching point screening and registration optimization on matching point pairs and calculate the geometric transformation matrix between infrared and visible light images, specifically:

[0037] Input the feature matching points of the two different modal images obtained in step 3; set the maximum noise scale σ max is the maximum noise level allowed by the algorithm; select the geometric model to be estimated and determine the maximum number of iterations N; randomly sample a subset T from the matching point pairs, estimate the geometric model θ through the sampled subset, and fit the geometric model using the least squares method to generate an initial geometric model estimate; calculate the residual r for each matching point pair i (θ), residual r i (θ) reflects the fitting error of the geometric model θ for each point pair; by marginalizing the noise scale σ, it is determined whether this feature point is an internal point, and the model quality is optimized; the weighted least squares method is used to estimate the geometric model;

[0038] In each iteration, the quality of the current geometric model and the probability of the internal points determine whether to continue sampling, and finally the optimal geometric model is found and the geometric transformation matrix H is output;

[0039] Step 5: Through the geometric transformation matrix H between the infrared and visible light images, perform geometric transformation on each pixel of the infrared image and convert it to the coordinate system of the visible light image to obtain the final registration result of the infrared image and the visible light image.

[0040] The beneficial effects of the above technical solution are as follows: the multi-source image registration method based on deep learning provided by the present invention, (1) through deep learning adaptive feature extraction and matching, does not rely on traditional manually designed features, so when the modal differences between multimodal images are large, a stable registration effect can still be obtained, which greatly improves the robustness of multimodal image registration; (2) by using the registration image data set to train the feature point detection network, it is possible to extract high-quality feature points, combined with the optimized attention mechanism, using the efficient lightweight feature matching network, accurately find the corresponding points between multimodal images, and the registration accuracy is significantly better than the traditional key point matching method. At the same time, the matching points are screened and optimized using a robust algorithm based on random consistency sampling, which further improves the accuracy of the registration results; (3) the network structure of the present invention is also optimized in terms of computational efficiency, and the efficient process of feature point detection and feature matching enables the method to meet real-time requirements in practical applications, and is suitable for multimodal image fusion scenarios such as autonomous driving, intelligent monitoring, and drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a multi-source image registration method based on deep learning is provided in an embodiment of the present invention.

[0042] Figure 2 A schematic diagram of the network structure for extracting image feature points provided by an embodiment of the present invention.

[0043] Figure 3 A schematic diagram of the image feature point matching network structure provided by an embodiment of the present invention.

[0044] Figure 4 This is a diagram of an image registration operation example provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0046] This embodiment takes infrared and visible light images as examples, and uses the multi-source image registration method based on deep learning of the present invention to register the infrared and visible light images.

[0047] In this embodiment, a multi-source image registration method based on deep learning is provided. Figure 1 As shown, the following steps are included:

[0048] Step 1: Image preprocessing: Gaussian filtering and image standardization are performed on infrared and visible light images respectively to reduce the modal difference between infrared images and visible light images, enhance similar features, and prepare for subsequent feature extraction and matching;

[0049] Step 1.1: Use Gaussian filtering to smooth the infrared image and reduce the noise between the infrared image and the visible light image;

[0050] In this embodiment, the convolution kernel for Gaussian filtering of infrared images is defined as:

[0051]

[0052] Among them, G(x,y) is the Gaussian kernel, I(x,y) is the pixel value at the position (x,y) of the infrared image, and I max is the maximum pixel value of the infrared image, α is the coefficient that controls the intensity of weight adjustment, and x and y represent the horizontal and vertical displacement of a point on the convolution kernel relative to the center σ g is the standard deviation of the infrared image;

[0053] By convolving the infrared image I(x, y) with the Gaussian kernel G(x, y), a smoothed infrared image is obtained;

[0054] I smooth (x,y)=I(x,y)*G(x,y) (2)

[0055] Gaussian filtering can effectively eliminate random noise in infrared images, making the images smoother and helping to improve the robustness of subsequent feature extraction.

[0056] Step 1.2: The filtered infrared image and visible light image are normalized separately so that the pixel values ​​of the two images are normalized to the same range to reduce the influence of the light intensity and sensor response differences.

[0057] The standardized formula is:

[0058]

[0059] Among them, I norm (x, y) is the normalized infrared or visible light image, I'(x, y) is the filtered infrared or visible light image, μ is the mean value of the infrared or visible light image, σ n is the standard deviation of the infrared or visible light image;

[0060] Through the normalization process, the pixel value range of the infrared or visible light image is adjusted to a standard normal distribution, so that the illumination difference between the infrared and visible light images is eliminated.

[0061] Step 2: Use the SuperPoint network to build a feature extraction network model to extract feature information from the infrared and visible light images after standardization. Figure 2 As shown, it includes a shared encoder and two decoders; the two decoders are an interest point decoder and a descriptor decoder respectively; in the feature point extraction process, the present invention further improves the model's feature point detection ability in infrared and visible light multimodal images by performing self-supervisory training and fine-tuning on the interest point decoder; first, the interest point decoder is used as a pre-training model, and its feature point detection ability that has been trained on a general data set is utilized; then, in the aligned infrared and visible light image data sets, a small number of labeled samples are manually marked as supervisory signals for preliminary training; in order to make full use of the data, a self-supervised learning algorithm is used, and on the basis of the labeled samples, unlabeled samples are further trained, and the feature point positions are predicted using the unlabeled samples; during the training process, a loss function based on the consistency of feature points at corresponding positions in the infrared and visible light images is designed, and the feature point detection results are optimized by minimizing the difference in the feature point positions in the two images;

[0062] The loss function is defined as follows:

[0063]

[0064] in, and Represent the corresponding feature point positions in infrared and visible light images respectively; by minimizing the position difference, the model can better learn the consistency of cross-modal feature points. Subsequently, through multiple rounds of iterative training, the model weights are continuously optimized so that it can obtain better feature point prediction results on unlabeled samples; finally, the model is fine-tuned and its weights are applied to the accurate detection of feature points in infrared and visible light images to improve the performance of the network in actual scenes; the network can autonomously learn features and adapt to the differences between multimodal images;

[0065] The specific method of SuperPoint network to extract feature points and descriptors from the input image is:

[0066] Step 2.1: Input the infrared or visible light image of size H×W×1 or H×W×3 (if it is a grayscale image, the input is a single channel, otherwise it is an RGB three-channel image) into the shared encoder to extract features of the image; the image first passes through multiple 3×3 convolution kernels to extract local information such as edges and corners in the image; then use three maximum pooling layers, each pooling passes through a 2x2 non-overlapping window, and gradually reduces the image size to 1 / 8, that is, H c =H / 8,W c =W / 8, forming a "cell unit" (Cells), that is, an area of ​​8x8 pixels; finally, nonlinearity is introduced through activation functions such as ReLU, making the network more expressive; the shared encoder outputs an intermediate tensor B∈R with a smaller spatial dimension Hc×Wc×F , where F is the number of channels, which provides rich multi-scale feature representation for subsequent interest point detection and descriptor extraction, while reducing computational complexity;

[0067] Step 2.2: Share the intermediate tensor B∈R output by the encoder Hc×Wc×F The input interest point decoder is used to detect key points from the tensor output by the shared encoder. First, a tensor of size Hc×Wc×65 is generated through a convolution operation. 64 of the 65 channels are used to represent the 8x8 local grid area, and the last channel is used to represent the category of "non-interest point". Then, a channel-level softmax operation is performed on the 64 local grid areas to normalize the pixel values ​​in each area to probability, indicating the probability of each pixel being an interest point. Finally, the 64-channel tensor is converted to H×W size through a reshape operation to generate the final interest point probability map. By analyzing the interest point probability map, the interest points that meet the probability threshold condition are feature points, thereby retaining the significant and robust information in the image and ensuring the accuracy and stability of subsequent feature matching.

[0068] Step 2.3: Share the intermediate tensor B∈R output by the encoder Hc×Wc×F Input descriptor decoder, used to extract the descriptor of each interest point; first, a semi-dense descriptor graph P∈R is generated through convolution operation Hc×Wc×D , that is, a descriptor is generated for every 8 pixels; the descriptor map P is then upsampled to a resolution H×W×D of the same size as the original image through bicubic interpolation, that is, a 128-dimensional descriptor is generated for each pixel; finally, the generated descriptors are L2 normalized and a descriptor map of size H×W×D is output, where D represents the dimension of the descriptor, so that the length of each descriptor is 1, which is convenient for the subsequent similarity calculation between feature points.

[0069] Step 3: Use the LightGlue network as the feature point matching network, perform feature matching on the feature points extracted in step 2, modify the LightGlue network, and replace the original global self-attention mechanism with local self-attention; in the self-attention unit, the feature points of each image only interact with the top K feature points that are most similar in space in the image. This improvement limits the attention calculation to the local area, greatly reducing the burden of calculating the correlation between global feature points. Through the efficient feature association ability of the network, high-precision matching point pairs are obtained; Figure 3 As shown, the feature point matching network consists of a stack of L identical layers that jointly process a pair of images. Each layer consists of self-attention units and cross-attention units that update the representation of each point. The classifier then decides at each layer whether to stop reasoning, thereby avoiding unnecessary calculations. The lightweight head calculates the match based on the representation set. Specifically, it includes the following steps:

[0070] Step 3.1: Input the infrared image and visible light image and the two-dimensional coordinate position p of each feature point i obtained in step 2 i =(x i ,y i ) and descriptor d i ∈R d ; In the self-attention unit, each feature point in an image (such as image A) only obtains information from other points in the same image; by calculating the query vector q for each point i and key vector k i Linear transformation of , and obtain the self-attention score between feature points:

[0071]

[0072] Among them, R(p j -p i ) is the point p j With p i The rotation encoding of the relative position between ij is a weight coefficient used to represent the local correlation of adjustment point i to feature point j;

[0073] In the cross-attention unit, the points in each image obtain information from the points in another image, and only the similarity between the key vectors is calculated to obtain the cross-attention score between the feature points:

[0074]

[0075] in, are the key vectors of infrared image feature point i and visible light image feature point j, is the weight of infrared image point i and visible light image point j, indicating the local similarity between them;

[0076] Step 3.2: After several layers of self-attention and cross-attention units, the feature point matching network generates a set of updated feature representations; next, the matching relationship is predicted by calculating the similarity score between points;

[0077] First, calculate the similarity score for each pair of points in image A and image B:

[0078]

[0079] Among them, S ij The midpoint of image A and the midpoint of image B Similarity score, Linear() is a linear transformation with a bias term;

[0080] Then assign a matching score σ to each point i , used to indicate whether the point can be matched:

[0081] σ i =Sigmoid(Linear(x i )) (8)

[0082] The score represents the probability of finding a match for point i in another image;

[0083] Finally, the similarity score and the matching score are combined to generate a matching probability matrix M, which represents the matching probability M of each pair of points. ij :

[0084]

[0085] Step 4: Use the MAGSAC algorithm to perform robust matching point screening and registration optimization on the matching point pairs and calculate the geometric transformation matrix;

[0086] Input the feature matching points of the two different modal images obtained in step 3; set the maximum noise scale σ max is the maximum noise level allowed by the algorithm; select the geometric model to be estimated and determine the maximum number of iterations N; randomly sample a subset T from the matching point pairs, estimate the geometric model θ through the sampled subset, and fit the geometric model using the least squares method to generate an initial geometric model estimate; calculate the residual r for each matching point pair i (θ), residual r i (θ) reflects the fitting error of the geometric model θ for each point pair; by marginalizing the noise scale σ, it is determined whether this feature point is an internal point, and the model quality is optimized; the weighted least squares method is used to estimate the geometric model;

[0087] In each iteration, the quality of the current geometric model and the probability of the internal points determine whether to continue sampling, and finally the optimal geometric model is found and the geometric transformation matrix H is output;

[0088] Step 5: Through the geometric transformation matrix H, each pixel of the infrared image is geometrically transformed and converted into the coordinate system of the visible light image to obtain the final registration result of the infrared image and the visible light image; this method can be executed in real time and shows high robustness and registration accuracy in multimodal images.

[0089] Through the implementation methods described in detail above, the multi-source image registration method proposed in the present invention can effectively solve the registration problem of images under different spectra, and is particularly suitable for complex registration tasks between infrared and visible light images. The multimodal image registration method in the present invention has been comprehensively improved in the steps of image preprocessing, feature extraction, feature matching, model optimization, etc., and can adapt to a variety of complex situations of different spectra, lighting conditions and geometric transformations. Compared with traditional image registration methods, the present invention has higher accuracy, stronger robustness and better adaptability. Its end-to-end architecture based on deep learning not only reduces the need for manual adjustment of parameters, but also greatly reduces the calculation time, and can achieve efficient multi-spectral image registration in real-time applications. The present invention is particularly suitable for scenes that need to process infrared and visible light images, such as unmanned driving, remote sensing monitoring, and military reconnaissance, and has broad application prospects. In this embodiment, the image registration operation example is as follows: Figure 4 shown.

[0090] In summary, the present invention proposes an innovative and effective multi-source image registration solution, which not only solves many limitations of traditional methods, but also has high application value and technical prospects. In future technological development, the present invention is expected to become an important foundation in the field of multimodal image processing, providing a more reliable solution for image registration in various complex scenes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-source image registration method based on deep learning, characterized by: The following steps are involved: Step 1: Perform image preprocessing on infrared and visible light images respectively to reduce modality differences and enhance the consistency of image features; Step 2: Build and train a feature point extraction model to extract feature points and descriptors from the preprocessed infrared and visible light images respectively; Step 3: Use the feature point matching model to perform feature matching on the feature points extracted in step 2 to obtain high-precision matching point pairs; Step 4: Perform robust matching point screening and registration optimization on the matching point pairs and calculate the geometric transformation matrix H between the infrared and visible light images; Step 5: Through the geometric transformation matrix H between the infrared and visible light images, perform geometric transformation on each pixel of the infrared image and convert it to the coordinate system of the visible light image to obtain the final registration result of the infrared image and the visible light image.

2. The multi-source image registration method based on deep learning according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1: Use Gaussian filtering to smooth the infrared image and reduce the noise between the infrared image and the visible light image; Step 1.2: The filtered infrared image and visible light image are normalized separately so that the pixel values ​​of the two images are normalized to the same range to reduce the influence of the light intensity and sensor response differences.

3. The multi-source image registration method based on deep learning according to claim 2, characterized in that: The feature point extraction model in step 2 extracts feature information from the infrared and visible light images after standardization respectively; the feature information includes feature points and descriptors; The feature point extraction model adopts the SuperPoint network, and the SuperPoint network extracts feature points and descriptors from the preprocessed infrared and visible light images respectively.

4. The multi-source image registration method based on deep learning according to claim 3, characterized in that: The SuperPoint network includes a shared encoder and two decoders; the two decoders are an interest point decoder and a descriptor decoder; in the feature point extraction process, the interest point decoder is self-supervised trained and fine-tuned to further improve the feature point detection capability of the feature point extraction model in infrared and visible light multimodal images; First, the interest point decoder is used as a pre-training model. Then, a small number of labeled samples are manually marked in the aligned infrared and visible light image datasets as supervisory signals for preliminary training. In order to make full use of the infrared and visible light image datasets, a self-supervised learning algorithm is used to further train unlabeled samples based on labeled samples, and the locations of feature points are predicted using unlabeled samples. During the training process, a loss function is designed based on the consistency of feature points at corresponding positions in infrared and visible light images, and the feature point detection results are optimized by minimizing the difference in the locations of feature points in the two images. The loss function is defined as follows: in, and Respectively represent the corresponding feature point positions in infrared and visible light images; Through multiple rounds of iterative training, the model weights are continuously optimized so that the feature point extraction model can better predict feature points on unlabeled samples; finally, after fine-tuning, the optimized weights are applied to feature point detection in infrared and visible light images, significantly improving the performance of the feature point extraction model in actual scenarios.

5. The multi-source image registration method based on deep learning according to claim 4, characterized in that: The feature point matching model described in step 3 uses the LightGlue network to perform feature matching on feature points, and replaces the global self-attention mechanism with local self-attention. In the self-attention unit, the feature point of each image only interacts with the top K feature points that are most similar in space in the image. In the self-attention unit, each feature point in an image obtains information only from the top K most spatially similar feature points in the same image; by calculating the query vector q for each point i and key vector k i Linear transformation of , and obtain the self-attention score between feature points: Among them, R(p j -p i ) is the point p j With p i The rotation encoding of the relative position between ij is a weight coefficient used to represent the local correlation of adjustment point i to feature point j; In the cross-attention unit, the points in each image obtain information from the points in another image, and only the similarity between the key vectors is calculated to obtain the cross-attention score between the feature points: in, are the key vectors of infrared image feature point i and visible light image feature point j, is the weight of infrared image point i and visible light image point j, indicating the local similarity between them; After several layers of self-attention and cross-attention units, the feature point matching network generates a set of updated feature representations; Then the matching relationship is predicted by calculating the similarity score between points; First, calculate the similarity score for each pair of points in image A and image B: Among them, S ij The midpoint of image A and the midpoint of image B Similarity score, Linear() is a linear transformation with a bias term; Then assign a matching score σ to each point i , used to indicate whether the point can be matched: σ i =Sigmoid(Linear(x i )) (5) The score represents the probability of finding a match for point i in another image; Finally, the similarity score and the matching score are combined to generate a matching probability matrix M, which represents the matching probability M of each pair of points. ij :

6. The multi-source image registration method based on deep learning according to claim 5, characterized in that: The step 4 adopts the MAGSAC algorithm to perform robust matching point screening and registration optimization on the matching point pairs and calculate the geometric transformation matrix between the infrared and visible light images, specifically: Input the feature matching points of the two images of different modalities obtained in step 3; Set the maximum noise scale σ max is the maximum noise level allowed by the algorithm; select the geometric model to be estimated and determine the maximum number of iterations N; randomly sample a subset T from the matching point pairs, estimate the geometric model θ through the sampled subset, and fit the geometric model using the least squares method to generate an initial geometric model estimate; calculate the residual r for each matching point pair i (θ), residual r i (θ) reflects the fitting error of the geometric model θ for each point pair; by marginalizing the noise scale σ, it is determined whether this feature point is an internal point, and the model quality is optimized; the weighted least squares method is used to estimate the geometric model; In each iteration, the quality of the current geometric model and the probability of the internal points determine whether to continue sampling, and finally the optimal geometric model is found and the geometric transformation matrix H is output.

Images