Visible light and infrared thermal imaging image registration method based on graph neural network

Abstract

The application discloses a visible light and infrared thermal imaging image registration method based on a graph neural network, first, corresponding original infrared thermal imaging and visible light heterogeneous image data are collected, and image data is processed to form an unlabeled data set; then, a feature point and a corresponding descriptor of an input image are extracted in a self-supervised manner through an inserted feature point extraction network; then, a two-stage matching method from coarse to fine is adopted to obtain a one-stage and two-stage matching point pair, confidence-based sampling weight calculation is performed according to the one-stage and two-stage matching point pair, a suboptimal transformation model is estimated according to the sampling weight, and an optimal transformation model is obtained, thereby completing the image registration task, improving the accuracy of feature point extraction under the graph neural network and the accuracy of feature point matching, and guaranteeing the registration effect of the infrared thermal imaging and the visible light heterogeneous image.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision image processing, and mainly relates to a method for registering visible light and infrared thermal imaging images based on graph neural networks. Background Technology

[0002] With advancements in computer vision technology and the continuous development of internet technology, object detection, the most crucial technology in computer vision, has entered a new era. As the application of object detection becomes increasingly widespread, in complex scenarios, such as detecting partially weathered metal sheets in the Gobi Desert, images generated by a single camera are insufficient to provide enough information to support the object detection task. In such cases, it is necessary to introduce images of the same target from different devices. This not only provides richer information but also overcomes the limitations of single-modal images. To fuse information from these dissimilar images, image registration is the first step.

[0003] Image registration is a fundamental task in computer vision, primarily aiming to fit the transformation parameters between two images. These transformation parameters reflect the mapping relationship between the images. Images used for registration typically come from cameras capturing the same scene from different angles. Heterogeneous image registration is a special case within image registration tasks, where heterogeneous refers to images with significant modal differences or originating from different sensors.

[0004] For same-source image registration, SIFT and its variants such as SURF, which employ manually designed descriptors, have achieved good results. However, due to the inherent characteristics of heterogeneous images, these traditional feature extraction methods often face significant challenges in heterogeneous image registration. The discriminative power of descriptors greatly affects matching performance. Given the significant appearance differences and low correlation between heterogeneous images, obtaining sufficiently discriminative feature descriptors remains a problem that needs further investigation. Compared to the difficulties faced in traditional same-source image registration, such as geometric deformation and illumination variations, heterogeneous image registration also needs to address the significant appearance differences caused by sensors with different imaging mechanisms. Traditional registration methods, limited by manually designed feature extraction techniques, often fail to obtain discriminative feature descriptors for the more challenging heterogeneous image registration.

[0005] In recent years, convolutional neural networks have demonstrated powerful feature representation capabilities in many computer vision tasks. Furthermore, using graph neural networks to aggregate the relationships between feature points inside and outside an image can yield more accurate feature point pairings, resulting in better registration outcomes. Summary of the Invention

[0006] This invention addresses the problem of insufficient accuracy in feature point extraction and matching in existing methods for heterogeneous image registration under complex environments. It provides a graph neural network-based method for registering visible light and infrared thermal imaging images. First, it acquires corresponding raw infrared thermal imaging and visible light heterogeneous image data, processing the image data to form an unlabeled dataset. Then, it extracts feature points and corresponding descriptors from the input images using a self-supervised approach by inserting a feature point extraction network. Next, a coarse-to-fine two-stage matching method is used to obtain matching point pairs for the first and second stages. Based on the first and second stage matching point pairs, confidence-based sampling weights are calculated. A suboptimal transformation model is estimated based on the sampling weights to obtain the optimal transformation model, completing the image registration task. This method improves the accuracy of feature point extraction and matching under the graph neural network, ensuring the effectiveness of infrared thermal imaging and visible light heterogeneous image registration.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a visible light and infrared thermal imaging image registration method based on graph neural networks, comprising the following steps:

[0008] S1, Data Processing: Collect the corresponding raw infrared thermal imaging and visible light heterogeneous image data, and process the image data to form an unlabeled dataset;

[0009] S2, Feature Extraction: Feature points and corresponding descriptors of the input image are extracted using a self-supervised method via a feature point extraction network. In this step, the encoder of the feature point extraction network processes the input image, and the output feature map is input into the decoder of the feature extraction network to obtain a score map representing the probability that each pixel is a feature point. Maximum suppression is applied to the score map to obtain the feature point distribution. The feature map output by the encoder is multi-convolved to output an initial descriptor matrix. The initial descriptor matrix is ​​expanded using a double interpolation method and normalized to obtain the descriptor for each pixel. This descriptor is then jointly encoded with the descriptor corresponding to the feature point to obtain a descriptor of uniform length.

[0010] S3, Feature point matching:

[0011] S31: The feature points and corresponding descriptors obtained in step S2 are processed by a graph neural network, and the matching matrix between heterogeneous images is obtained by iteratively solving the Sinkhorn algorithm, and the first-stage matching point pairs are output.

[0012] S32: After filtering the first-stage matching point pairs obtained in step S31, estimate the common viewing area. Repeat steps S2 and S31 on the common viewing area to obtain the second-stage matching point pairs.

[0013] S4, Image Registration: Based on the first and second stage matching point pairs obtained in step S3, calculate the sampling weights based on confidence, estimate the suboptimal transformation model based on the sampling weights, and obtain the optimal transformation model to complete the image registration task.

[0014] As an improvement of the present invention, the image data processing in step S1 specifically includes: performing pixel-symmetric transformations on each image along the x-axis and y-axis of the image center, flipping to expand the data volume, stretching, and uniformly adjusting the image size to form a grayscale image with a resolution of H*W, a single channel, and a pixel value range of 0-255. The dataset D is:

[0015] D = {(x1,y1),(x2,y2)...(x...} n ,y n )}

[0016] Where (x,y) is the processed infrared thermal image and visible light image pair, and the image is a 1*H*W tensor; n represents the amount of data.

[0017] As another improvement of the present invention, in step S2, the encoder of the feature point extraction network includes convolutional layers, pooling layers, and nonlinear activation function layers. The non-activation function adopts the ReLU function, the mathematical expression of which is as follows:

[0018]

[0019] The pooling layer places a max pooling operation after every two convolutional layers, with a pooling kernel size of 2×2 and a stride of 2.

[0020] The decoder of the feature point extraction network contains two convolutional layers with 256 and 65 channels respectively, multiple BN layers and activation function layers, and outputs an 80×60×65 tensor. The normalized probability distribution is obtained by classification through the Softmax layer, and the size is restored by the ReShape layer to obtain the score map. The pixel values ​​of the score map are distributed between 0 and 1, representing the probability that each pixel in the input image is a feature point.

[0021] As another improvement of the present invention, a channel spatial attention mechanism module is further inserted into the feature point extraction network encoder in step S2.

[0022] The channel attention mechanism module performs global average pooling and global max pooling based on the size of the network feature map, obtains weights through a multi-perceptron, sums the weights item by item, and finally normalizes the weights using the sigmoid function and performs a channel-by-channel multiplication weighted summation before fusing the output into the input feature map. Specifically:

[0023] M c(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0024] The spatial attention mechanism module performs global average pooling and global max pooling operations on the feature map output by the channel attention mechanism module. After convolution to reduce the dimensionality to 1 dimension, it obtains attention features through the Sigmoid function and finally fuses them with the input feature map before outputting the final feature map. Specifically:

[0025] M s (F)=σ(∫ 7×7 (MLP(AvgPool(F)); MLP(MaxPool(F))).

[0026] As another improvement of the present invention, step S31 specifically includes:

[0027] S311: Input infrared thermal imaging image I thermal and visible light image I visible Feature points p and descriptors d are extracted from the two images. M and N feature points are extracted from each image, respectively, denoted as T = {1, ..., M} and V = {1, ..., N}. For the infrared thermal imaging image I... thermal Extracted feature points and descriptors The initial feature matching vector is formed by fusion. The mathematical expression for a vector is:

[0028]

[0029] MLP stands for Multilayer Perception Encoder;

[0030] For I visible Extracted feature point information and descriptors The same operation described above is performed to obtain the initial feature matching vector.

[0031] S312: Input the obtained initial feature matching vector into a multilayer graph neural network. In the graph neural network, each node of the complete graph represents the feature matching vector of each feature point. or Aggregate edge information for each feature point i to obtain m ε→i The mathematical expression is:

[0032]

[0033] Where α ij is the attention weight, representing the similarity between the queried feature point i and all its neighboring feature points j;

[0034] S313: After obtaining the aggregated feature point information of each layer, the initial matching vector is combined with the layer-by-layer calculation and update. The vector calculation formula is as follows:

[0035]

[0036] Where [·||·] represents the concatenation operation between two vectors;

[0037] After passing through a graph neural network with a total of L layers, the final feature point matching vector f is obtained. i T :

[0038]

[0039] Where W and b are the weights and biases, respectively, the visible light image also undergoes the above steps to obtain the feature point matching vector f. i V ;

[0040] S314: Regarding the f obtained above i T f i V Perform the scoring matrix S∈R M×N Construct, and by maximizing the overall score ∑ i，j S i，j P i，j To obtain the optimal matching matrix P, the mathematical structure of the score matrix is ​​as follows:

[0041]

[0042] S315: Iteratively solve ∑ using the Sinkhorn algorithm. i，j S i，j P i，j To maximize the overall score, the final matching matrix is ​​thresholded, and each element in the matching matrix below the threshold t is considered a mismatch. The first-stage matching point pair set is then output.

[0043] As another improvement of the present invention, step S32 specifically includes:

[0044] S321: Sort the first-stage matching point pairs output in step S31 by confidence from largest to smallest, and take the top n% of the sorted matching point pair set as the common view region estimation point set;

[0045] S322: Based on the estimated point set of the common viewing area, sort the pixel horizontal and vertical coordinates to obtain the maximum and minimum coordinate values: x max x min y max y minThe selected area is cropped using the coordinate extrema: the starting point is (x... min y min ), with a length of (x) max -x min ), width is (y max -y min ), cropping to obtain heterogeneous image pairs in the common viewing region;

[0046] S323: Scale the selected area obtained by cropping in step S322 to H*W, and repeat steps S2 and S31 to obtain the two-stage matching point set:

[0047] S324: Perform an affine transformation based on the transformation matrix F on the two-stage matching point set, transforming it to the corresponding point set of the one-stage input image, so that the matching point pairs of the one-stage and two-stage stages are concatenated, and the concatenated matching results are filtered by NMS to obtain the final matching point pair set.

[0048] As a further improvement of the present invention, step S4 specifically includes:

[0049] S41: The sampling weight is calculated based on the confidence level corresponding to the matching point pair. The mathematical expression for the sampling weight corresponding to the matching point pair i is as follows:

[0050]

[0051] Among them, S i Represents the confidence level of each matching pair;

[0052] S42: Randomly sample the set of matched point pairs according to the sampling weights and estimate the model to obtain a suboptimal model with low confidence. Output the interior point set m corresponding to this suboptimal model. s The output conditions of the suboptimal model are:

[0053]

[0054] Where k is the iteration number, p is the confidence level of the model at this time, ε is the proportion of the number of interior points corresponding to this model to the total number of samples, and n is the minimum number of samples.

[0055] S43: Through the interior point set m s Continue random sampling based on sampling weights and estimate the model. When the obtained model can find more inliers, replace the original sampling set with this inlier set. Otherwise, continue sampling and estimation on the original sampling set. When the proportion of inliers is close to 100% or no longer needs to be transformed, output the optimal transformation model and complete the image registration.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) This invention uses graph neural networks to aggregate feature points and feature descriptors to obtain the relationship information between feature points and other feature points, and uses the relationship information to guide the accurate matching of feature points;

[0058] (2) The method of the present invention achieves the task of extracting the common viewing area of ​​infrared thermal imaging and visible light image by screening the first-stage matching point pairs, and then extracts and matches the secondary feature points of the common viewing area to complete the image matching from coarse to fine. This method obtains more highly reliable matching point pairs to reduce the impact of high background repetition and low texture.

[0059] (3) This invention uses feature point confidence to estimate the parameters of the image transformation model based on sampling weight. By first obtaining a suboptimal model with low confidence, the sample set is screened to obtain a high-quality sampling point set. By continuously approximating the model with the correct registration of high-quality matching point pairs, the optimal registration model is obtained. This method can quickly and accurately obtain a high-precision image perspective transformation model, and ultimately improve the image registration quality. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the overall steps of the method of the present invention.

[0061] Figure 2 This is a schematic diagram of the feature point extraction network in step S2 of the method of the present invention;

[0062] Figure 3 This is a structural framework diagram of the attention mechanism in the feature point extraction network in step S2 of the method of the present invention;

[0063] Figure 4 This is a flowchart illustrating the two-stage image registration process in step S3 of the method of the present invention.

[0064] Figure 5 This is a flowchart of step S4 of the method of the present invention. Detailed Implementation

[0065] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0066] Example 1

[0067] A method for registering visible light and infrared thermal imaging images based on graph neural networks is proposed. This method uses an improved feature extraction network with an attention mechanism to extract feature points from heterogeneous images, and employs a graph neural network to construct feature matching vectors, thereby improving the accuracy of feature point extraction and matching. A two-stage matching method from coarse to fine is used to further improve the accuracy of feature point matching, and a sampling optimization algorithm is applied to feature point matching results with high redundancy. Figure 1 As shown, the specific steps include the following:

[0068] S1: Data Processing: Collect the corresponding raw infrared thermal imaging and visible light heterogeneous image data, and process the image data to form an unlabeled dataset.

[0069] A simulated scene was built, requiring the target object to be indistinguishable from the background. A binocular camera was constructed by setting up an infrared thermal imaging camera and a visible light camera at fixed positions to collect data. The corresponding raw infrared thermal imaging and visible light heterogeneous image data were collected to form dataset D1.

[0070] Processing heterogeneous image data to form an unlabeled dataset: After flipping each image pair in dataset D1 to increase the data volume, a uniform image resizing operation is performed. Flipping refers to performing pixel-symmetrical transformations along the x-axis and y-axis of the image center, respectively. Image resizing refers to using bilinear interpolation to obtain the pixel values ​​of each pixel after resizing. The final result is a grayscale image with a resolution of H*W, single channel, and pixel values ​​ranging from 0 to 255. The final dataset D2 is as follows:

[0071] D2={(x1,y1),(x2,y2)…(x n ,y n )}

[0072] Where (x,y) is the processed infrared thermal image and visible light image pair, and the image is a 1*H*W tensor; n represents the amount of data.

[0073] S2, Feature Extraction: Feature points and corresponding descriptors of the input image are extracted using a self-supervised method by inserting a feature point extraction network. A schematic diagram of the feature point extraction network is shown below. Figure 2 As shown.

[0074] S21: Transfer the image to be registered (x i ,y iThe encoder of the feature point extraction network using the improved attention module proposed in this invention includes convolutional layers, pooling layers, and nonlinear activation function layers. The main function of the convolutional layer is to first fill the boundaries of the input image, and then perform convolution operations on the input image using convolution kernels to extract the image features and output a feature map. The function of the nonlinear activation function layer is to increase the nonlinearity of the neural network. The main function of the pooling layer is to downsample the feature map obtained by the convolutional layer, reduce the size of the feature map output by the convolutional layer, and reduce the computational load of the network.

[0075] S22: Combine the channel attention mechanism module and the spatial attention mechanism module to construct an attention mechanism module, and insert the constructed attention mechanism module into the encoder. After a series of operations by the encoder, a feature map of H / 8×W / 8×128 is finally obtained.

[0076] The structural framework diagram of the attention mechanism is as follows: Figure 3 As shown, the channel attention mechanism involves: global average pooling and global max pooling based on the size of the network feature map; obtaining weights through a multi-perceptron; summing each weight; and finally normalizing the weights using the sigmoid function and multiplying them channel-by-channel into the input feature map. Its mathematical expression is:

[0077] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0078] The spatial attention mechanism module process involves performing global average pooling and global max pooling operations on the feature maps output by the channel attention module, followed by convolution to reduce the dimensionality to 1D. Finally, the attention features are obtained through the sigmoid function and fused with the input feature map. The mathematical expression is as follows:

[0079]

[0080] S23: The feature map obtained in step S22 is input into the feature point decoder of the feature point extraction network. The decoder contains two convolutional layers with 256 and 65 channels, multiple BN layers, and non-linear activation function layers, outputting a tensor of size H / 8×W / 8×65. Each 65-dimensional tensor represents 65 cases where the i-th pixel in a non-overlapping 8×8 pixel window of the original image is a feature point or the pixel window does not contain a feature point. The normalized probability distribution of the 65 cases is obtained by classification through a Softmax layer, and then the size is restored to H×W×1 through a ReShape layer to obtain a score map of size H×W. The pixel values ​​of the score map are distributed between 0 and 1, representing the probability that each pixel in the input image is a feature point.

[0081] S24: Apply the Non-Maximum Suppression (NMS) algorithm to each N×N window in the output score map obtained in step S23. Each window retains only one maximum value. Then, perform a threshold judgment on the entire map and regard the points above the threshold as feature points to obtain M feature points.

[0082] S25: Input the feature map obtained in step S22 into the descriptor decoder of the feature point extraction network to normalize the image size, and at the same time, normalize the feature points to their corresponding positions; then list the normalized feature points to construct a 1×1×K×2 tensor, where K represents the number of feature points and 2 represents the X and Y coordinates of the feature points respectively; perform inverse normalization on the positions of the feature points: scale and translate according to the ratio to obtain the positions of the inverse normalized feature points in the tensor. Since the positions may be non-integer pixels, they need to be filled in by a bilinear interpolation algorithm to insert the descriptor into the position of the corresponding key point; finally, normalize by L2 norm standard to obtain a descriptor of uniform length.

[0083] S3, Feature point matching: such as Figure 4 As shown, it is divided into two parts:

[0084] S31: The feature points and corresponding descriptors obtained in step S2 are processed by a graph neural network, and the matching matrix between heterogeneous images is obtained by iteratively solving the Sinkhorn algorithm, and the first-stage matching point pairs are output.

[0085] S311: Step S2 can be used to process the input infrared thermal imaging image I. t h ermal and visible light image I visible Feature points p and descriptors d are extracted. M and N feature points are extracted from the two images respectively, denoted as T={1,…,M} and V={1,…,N}.

[0086] Infrared thermal imaging image I t h ermal Extracted feature points and descriptors The initial feature matching vector is formed by fusion. The mathematical expression for a vector is:

[0087]

[0088] MLP stands for Multilayer Perceptron Encoder, and its main function is to process feature point location information. After dimensionality increase, it can be compared with descriptors Perform an addition operation;

[0089] For I visible Extracted feature point information and descriptors The same operation described above is performed to obtain the initial feature matching vector.

[0090] S312: Input the obtained series of initial feature matching vectors into a multilayer graph neural network. Each node in the complete graph of the graph neural network represents the feature matching vector of each feature point. or There are two types of undirected edges between nodes: one is ε self One represents the relationship between feature points within an infrared thermal imaging image or a visible light image; the other is ε. cross This represents the relationship between feature points in infrared thermal imaging and feature points in visible light images. The edge information of each feature point i is aggregated to obtain m. ε→i The mathematical expression is:

[0091]

[0092] Where α ij The attention weight represents the similarity between the query feature point i and all its neighboring feature points j, and its mathematical expression is as follows:

[0093]

[0094] The Softmax algorithm will obtain q i and k j The similarity between them.

[0095] Let feature point i be located on input image Q, and feature point j be located on input image S, where (Q,S)∈{T,V} 2 The mathematical expressions for q, k, and v in the above two equations are as follows:

[0096]

[0097]

[0098] Where l is the l-th layer in a multi-layer graph neural network, and W1, W2, W3, b1, b2, b3 represent the weights and biases shared by the l-th layer, respectively.

[0099] S313: After obtaining the aggregated feature point information of each layer, the initial matching vector is combined with the layer-by-layer calculation and update. The vector calculation formula is as follows:

[0100]

[0101] Where [·‖·] denotes the concatenation operation between two vectors, and when l is odd, ε is only ε. self When l is even, ε is only ε cross .

[0102] After passing through a graph neural network with a total of L layers, the final feature point matching vector is obtained.

[0103]

[0104] Where W and b are the weights and biases, respectively, the feature point matching vector f can also be obtained from a visible light image through the above steps. i V .

[0105] S314: Regarding the f obtained above i T f i V Perform the scoring matrix S∈R M×N Construct, and by maximizing the overall score ∑ i,j S i,j P i,j To obtain the optimal matching matrix P, the mathematical structure of the score matrix is ​​as follows:

[0106]

[0107] S315: Solving ∑ using the Sinkhorn algorithm i,j S i,j P i,j To maximize the overall score, after obtaining the score matrix S, a distribution vector is defined. and The regularization term λ, and the specific algorithm steps are as follows:

[0108] Initialize the matching matrix P = e -λS Begin the following iterations:

[0109] Step 1: Calculate the sum of each row of the current matrix P;

[0110] Step 2: Divide each element in a row by the sum of the above rows and multiply by the distribution vector. Corresponding value;

[0111] Step 3: Calculate the sum of each column of the current matrix P;

[0112] Step 4: Divide each element in each column by the sum of the above columns and multiply by the distribution vector. Corresponding value;

[0113] Repeat the above steps until convergence.

[0114] Threshold the final matching matrix P, and treat each element of the matching matrix below the threshold t as a mismatch, and output the set of matching point pairs m1.

[0115] S32: After filtering the first-stage matching point pairs obtained in step S31, estimate the common viewing area. Repeat steps S2 and S31 on the common viewing area to obtain the second-stage matching point pairs.

[0116] S321: Sort the confidence scores of the matching point pairs output in step S31 from largest to smallest, and take the top n% of the sorted matching point pairs as the common view region estimation point set.

[0117] S322: Based on the estimated point set of the common viewing area obtained in S321, sort the pixels by their horizontal and vertical coordinates to obtain the maximum and minimum coordinate values: x max x min y max y min The selected area is cropped using the coordinate extrema: the starting point is (x... min ,y min ), with a length of (x) max -x min ), width is (y max -y min );

[0118] S323: For the selected area obtained by cropping in step S322, scale the image resolution to H*W, and repeat steps S2 and S31 to obtain the two-stage matching point set m2.

[0119] S324: Perform an affine transformation based on the transformation matrix F on the two-stage matching point set m2, transforming it to the corresponding point set m′2 of the one-stage input image. Its mathematical representation is as follows:

[0120] m′2=Fm2

[0121] Where m′2 is the coordinate transformation matrix F of the matching point pair, as shown below:

[0122]

[0123] The parameters in the transformation matrix can be calculated from the coordinate extrema of S52:

[0124]

[0125] The matching point pairs in the first and second stages are concatenated, and the concatenated matching results are filtered by NMS to obtain the final set of matching point pairs m.

[0126] S4, Image Registration: Based on the first and second stage matching point pairs obtained in step S3, confidence-based sampling weights are calculated. Suboptimal transformation model estimation is performed based on the sampling weights to obtain the optimal transformation model, thus completing the image registration task. The specific process is as follows: Figure 5 As shown.

[0127] S41: The sampling weight is calculated based on the confidence level corresponding to the matching point pair. The mathematical expression for the sampling weight corresponding to the matching point pair i is as follows:

[0128]

[0129] S42: Randomly sample the set of matched point pairs according to the sampling weights and estimate the model to obtain a suboptimal model with low confidence. Output the interior point set m corresponding to this suboptimal model. s The output conditions of the suboptimal model are shown in the following equation:

[0130]

[0131] Where k is the iteration number, p is the confidence level of the model at this time, ε is the proportion of the number of interior points corresponding to this model to the total samples, and n is the minimum number of samples, which is 4 in the image transformation model estimation task.

[0132] S43: Through the interior point set m s Continue random sampling based on sampling weights and estimate the model. When the obtained model can find more inliers, replace the original sampling set with this inlier set. Otherwise, continue sampling and estimation on the original sampling set. In this way, the number of inliers in the sampling set will continue to increase. When the proportion of inliers is close to 100%, or when no more transformation is needed, the optimal transformation model is output, and the image registration is completed.

[0133] Example 2

[0134] A method for registering visible light and infrared thermal imaging images based on graph neural networks includes the following steps:

[0135] S1: Infrared thermal imaging and visible light images of the same scene are acquired from a multimodal imaging system constructed from infrared thermal imaging and visible light. These images are then flipped and stretched to obtain grayscale images with a resolution of 640*480, single-channel, and pixel values ​​ranging from 0-255. Dataset D:

[0136] D = {(x1,y1),(x2,y2)...(x...} 400 ,y 400 )}

[0137] S2: The feature point extraction network is used to extract feature points and corresponding descriptors from the input image in a self-supervised manner.

[0138] S21: The encoder processes the input image. In this embodiment, the image size is reduced from 640×480×1 to 80×60×128, resulting in a smaller spatial size and a larger channel depth. The encoder structure consists of 8 convolutional layers, 3 pooling layers, and a non-linear activation function layer. The first four convolutional layers contain 64 3×3 convolutional kernels with a stride of 1, and the last four convolutional layers contain 128 3×3 convolutional kernels with a stride of 1. The non-activation function used is the ReLU function, whose mathematical expression is as follows:

[0139]

[0140] The pooling layer places a max pooling operation after every two convolutional layers, with a pooling kernel size of 2×2 and a stride of 2.

[0141] S22: Insert a channel spatial attention mechanism module into the encoder described above. The channel attention mechanism process involves: global average pooling and global max pooling based on the network feature map size; obtaining weights through a multi-perceptron; summing each weight; normalizing the weights using the Sigmoid function; and multiplying and weighting each channel before fusing them into the input feature map for output. The feature map output by the channel attention module is then subjected to global average pooling and global max pooling, followed by convolution to reduce the dimensionality to 1D. Finally, the attention features are obtained using the Sigmoid function and fused with the input feature map before output.

[0142] S23: Feature point extraction is performed on the feature map output by the encoder. The feature point extraction part includes two convolutional layers with 256 and 65 channels respectively, multiple BN layers, and activation function layers, outputting an 80×60×65 tensor. Each 65-dimensional tensor represents 65 cases where the i-th pixel in a non-overlapping 8×8 pixel window of the original image is a feature point or the pixel window does not contain a feature point. The normalized probability distribution of the 65 cases is obtained by classification through a Softmax layer, and then the size is restored to 640×480×1 through a ReShape layer to obtain a 640×480 score map. The pixel values ​​of the score map are distributed between 0 and 1, representing the probability that each pixel in the input image is a feature point.

[0143] S24: Perform maximum suppression on the score map to obtain the final feature point distribution;

[0144] S25: After multiple convolution of the feature map output by the encoder, an initial descriptor matrix with a dimension of 80×60×256 is output. Then, the initial descriptor matrix is ​​expanded to 640×480×256 by double interpolation. The 256 channels are normalized by L2 regularization to obtain the descriptor of each pixel in the original image. Then, it is jointly encoded with the descriptor corresponding to the feature point.

[0145] S3: An attention-based graph neural network is used to aggregate the characteristics of feature points and descriptor joint vectors to obtain matching vectors with better spatial location and feature characteristics. The matching matrix between heterogeneous images is obtained by iteratively solving the Sinkhorn algorithm, and the matching point pairs are output.

[0146] S31: The feature points and corresponding descriptors obtained in step S2 are processed by a graph neural network to obtain feature matching descriptors, and a first-stage matching point pair is obtained through similarity measurement;

[0147] The registered image feature points are coupled with their corresponding feature descriptors to form a feature matching vector, the expression of which is shown below:

[0148]

[0149] The feature point location information is obtained through an MLP multilayer sensing encoder. After dimensionality increase, with descriptors Since they have the same dimensions, they can be added together.

[0150] Information is stored in the form of nodes. Feature matching vectors are constructed by aggregating the results of operations on nodes within an image and the node relationships between registered images. These feature matching vectors are continuously passed and aggregated within the multi-layer graph neural network. The formula for passing the feature matching vector from one layer to the next is shown below:

[0151]

[0152] Where [·‖·] denotes the concatenation operation between two vectors, m ε→i This represents the aggregated edge information for each specific matching vector. When l is odd, ε is only ε. self When l is even, ε is only ε cross .

[0153] S32: Sort the matching results from largest to smallest according to the values ​​in the matching matrix, take the top 40% as the reference point set for common view region estimation, and perform common view region estimation; calculate the common view region based on the reference point set, and crop to obtain heterogeneous image pairs of the common view region; after cropping the estimated common view region, scale it to 640×480, perform secondary feature point extraction and matching, perform affine transformation on the two-stage matching point pairs, project the matching point pairs onto the input image, and perform NMS operation together with the one-stage matching point pairs. Perform non-maximum suppression on the surrounding one pixel of each matching point pair, select the matching point pair with the highest confidence, and output the final matching point pair.

[0154] S4: Based on the first and second stage matching results, perform confidence-based sampling weight calculation on the matching point pairs, and estimate the suboptimal transformation model based on the sampling weight. Use the interior point set obtained from the suboptimal transformation model estimation to estimate the number of sampling points to approximate the correct transformation model, obtain the optimal transformation model, and complete the image registration task.

[0155] S41: Calculate the sampling weights for the corresponding confidence levels based on the matching points;

[0156] S42: Estimate the suboptimal model and sampling point set according to the following formula.

[0157]

[0158] The model confidence level p is set to 0.6 to obtain the suboptimal model and the corresponding set of sampling points;

[0159] S43: Perform random consistent sampling based on sampling weights on the sample point set obtained in step S42. When the obtained model can find more inliers, replace the original sample set with this inlier set; otherwise, continue sampling and estimation on the original sample set. In this way, the number of inliers in the sample set will continuously increase. When the proportion of inliers approaches 100% or no longer changes, the optimal transformation model is output. Image registration is complete.

[0160] This invention employs an attention mechanism for targeted feature extraction, yielding more accurate feature points. A graph neural network is used to aggregate relationships between feature points within and between the registered images, enabling the feature matching vector to acquire contextual information and thus obtain more discriminative feature point matching pairs. Finally, a two-stage image registration method is used to improve the accuracy of feature point extraction and matching under the graph neural network. Faced with a large number of matching point pairs generated in the initial matching, this invention adopts a coarse-to-fine model estimation method. Sampling weights are constructed using the confidence of matching point pairs, and based on these weights, a set of sampling points with high overall reliability is selected. Image transformation model parameter estimation based on the sampling point set is then used for correct model approximation. This method is suitable for infrared thermal imaging and visible light heterogeneous image registration in challenging scenarios with high background repetition and blurred details, ultimately ensuring the effectiveness of infrared thermal imaging and visible light heterogeneous image registration.

[0161] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for registering visible light and infrared thermal imaging images based on graph neural networks, characterized in that, Includes the following steps: S1, Data Processing: Collect the corresponding raw infrared thermal imaging and visible light heterogeneous image data, and process the image data to form an unlabeled dataset; S2, Feature Extraction: Feature points and corresponding descriptors of the input image are extracted using a self-supervised method through a feature point extraction network. In this step, the encoder of the feature point extraction network processes the input image, and the output feature map is input into the decoder of the feature extraction network to obtain a score map representing the probability that each pixel is a feature point. Maximum suppression is applied to the score map to obtain the feature point distribution. The feature map output by the encoder is multi-convolved to output an initial descriptor matrix. The initial descriptor matrix is ​​expanded by double interpolation and normalized to obtain the descriptor of each pixel. This descriptor is then jointly encoded with the descriptor corresponding to the feature point to obtain a descriptor of uniform length. S3, Feature point matching: S31: The feature points and corresponding descriptors obtained in step S2 are processed by a graph neural network, and the matching matrix between heterogeneous images is obtained by iteratively solving the Sinkhorn algorithm, and the first-stage matching point pairs are output. S32: After filtering the first-stage matching point pairs obtained in step S31, estimate the common viewing area. Repeat steps S2 and S31 on the common viewing area to obtain the second-stage matching point pairs. S4, Image Registration: Based on the first and second stage matching point pairs obtained in step S3, calculate the sampling weights based on confidence, estimate the suboptimal transformation model based on the sampling weights, and obtain the optimal transformation model to complete the image registration task.

2. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 1, characterized in that: The image data processing in step S1 specifically includes: performing pixel-symmetric transformations on each image along the x-axis and y-axis of the image center, flipping to expand the data volume, stretching, and uniformly adjusting the image size to form a grayscale image with a resolution of H*W, a single channel, and a pixel value range of 0-255. The dataset D is: ； Among them, (x) n ,y n ) represents the processed infrared thermal image and visible light image pair, with the image format being a 1*H*W tensor; n represents the amount of data.

3. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 1, characterized in that: In step S2, the encoder of the feature point extraction network includes convolutional layers, pooling layers, and non-linear activation function layers. The non-activation function is the ReLU function, whose mathematical expression is as follows: ； The pooling layer places a max pooling operation after every two convolutional layers, with a pooling kernel size of 2×2 and a stride of 2; The decoder of the feature point extraction network contains two convolutional layers with 256 and 65 channels respectively, multiple BN layers and activation function layers, and outputs an 80×60×65 tensor. The normalized probability distribution is obtained by classification through the Softmax layer, and the size is restored by the ReShape layer to obtain the score map. The pixel values ​​of the score map are distributed between 0 and 1, representing the probability that each pixel in the input image is a feature point.

4. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 3, characterized in that: The feature point extraction network encoder in step S2 also incorporates a channel spatial attention mechanism module. The channel spatial attention mechanism module performs global average pooling and global max pooling based on the size of the network feature map, obtains weights through a multi-perceptron, sums the weights item by item, and finally normalizes the weights using the sigmoid function and performs a channel-by-channel multiplication weighted summation before fusing the output into the input feature map. Specifically: ； The spatial attention mechanism module performs global average pooling and global max pooling operations on the feature map output by the channel attention mechanism module. After convolution to reduce the dimensionality to 1 dimension, it obtains attention features through the Sigmoid function and finally fuses them with the input feature map before outputting the final feature map. Specifically: 。 5. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 4, characterized in that: Step S31 specifically includes: S311: Input infrared thermal imaging image and visible light images Feature points p and descriptors d are extracted from the two images. M and N feature points are extracted from each image, respectively, denoted as T={1,…,M} and V={1,…,N}. For the infrared thermal imaging image... Extracted feature points and descriptors The initial feature matching vector is formed by fusion. The mathematical expression for a vector is: = ； MLP stands for Multilayer Perception Encoder; for Extracted feature point information and descriptors The same operation described above is performed to obtain the initial feature matching vector. ; S312: Input the obtained initial feature matching vector into a multilayer graph neural network. In the graph neural network, each node of the complete graph represents the feature matching vector of each feature point. or For each feature point i, the edge information is aggregated to obtain... The mathematical expression is: ； in is the attention weight, representing the similarity between the query feature point i and all its neighboring feature points j; S313: After obtaining the aggregated feature point information of each layer, the initial matching vector is combined with the layer-by-layer calculation and update. The vector calculation formula is as follows: ； in This represents a concatenation operation between two vectors; After passing through a graph neural network with a total of L layers, the final feature point matching vector is obtained. : ； Where W and b are the weights and biases, the visible light image also obtains the feature point matching vector through steps S311-S313. ; S314: Regarding the above results , Perform a scoring matrix Build, and maximize the overall score To obtain the optimal matching matrix The mathematical structure of the score matrix is ​​shown below: , ； S315: Solved iteratively using the Sinkhorn algorithm. To maximize the overall score, the final matching matrix is ​​thresholded, and each element in the matching matrix below the threshold t is considered a mismatch. The first-stage matching point pair set is then output.

6. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 5, characterized in that: Step S32 specifically includes: S321: Sort the first-stage matching point pairs output in step S31 by confidence from largest to smallest, and take the top n% of the sorted matching point pair set as the common viewing region estimation point set; S322: Based on the estimated point set of the common viewing area, sort the pixel horizontal and vertical coordinates to obtain the maximum and minimum coordinate values: , , , The selected area is cropped using the coordinate extrema: the starting point is... , length is , width is Cropping yields heterogeneous image pairs in the shared viewing region; S323: Scale the selected area obtained by cropping in step S322 to H*W, and repeat steps S2 and S31 to obtain the two-stage matching point set; S324: Perform an affine transformation based on the transformation matrix F on the two-stage matching point set, transforming it to the corresponding point set of the one-stage input image, so that the matching point pairs of the one-stage and two-stage stages are concatenated, and the concatenated matching results are filtered by NMS to obtain the final matching point pair set.

7. The visible light and infrared thermal imaging image registration method based on graph neural networks as described in claim 6, characterized in that: Step S4 specifically includes: S41: The sampling weight is calculated based on the confidence level corresponding to the matching point pair. The mathematical expression for the sampling weight corresponding to the matching point pair i is as follows: ； in, Represents the confidence level of each matching pair; S42: Randomly sample the set of matched point pairs according to the sampling weights and estimate the model to obtain a suboptimal model with low confidence. Output the set of interior points corresponding to this suboptimal model. The output conditions of the suboptimal model are: ； in, For iteration rounds, This is the confidence level of the model at this point. Therefore, the proportion of interior points corresponding to this model to the total sample is... Minimum number of samples; S43: Through the set of interior points Continue random sampling based on sampling weights and estimate the model. When the obtained model can find more inliers, replace the original sampling set with this inlier set. Otherwise, continue sampling and estimation on the original sampling set. When the proportion of inliers is close to 100% or no longer needs to be transformed, output the optimal transformation model and complete the image registration.

Images