A feature point matching method across multispectral images
By combining adversarial generative networks and convolutional neural networks, feature point matching across spectral images was achieved, solving the problem of insufficient environmental perception capability of autonomous vehicles under low light conditions, improving image fusion quality and matching accuracy, and enhancing the safety of autonomous vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2023-01-19
- Publication Date
- 2026-06-02
AI Technical Summary
In low-light scenarios, existing technologies struggle to accurately register visible light and infrared images, leading to a decline in the environmental perception capabilities of autonomous vehicles. This is especially true at night and under other low-light conditions, where image fusion quality is severely affected by camera shake and displacement, making it impossible to accurately capture corresponding features in the images.
An adversarial generative network model is used to convert visible light images into pseudo-non-visible light images. Convolutional neural networks and graph neural networks are combined for key point extraction and matching. By training the generator and discriminator alternately and optimizing, outliers are eliminated using a random sampling consensus algorithm, thus achieving reliable feature point matching across spectral images.
This improves the perception capabilities of autonomous vehicles in low-light scenarios, reduces the dependence on annotations across spectral datasets, enhances the similarity and matching stability between images, and ensures the perception capabilities and safety of autonomous vehicles under low-light conditions.
Smart Images
Figure CN116051872B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent autonomous driving technology, and in particular to a feature point matching method for cross-spectral images. Background Technology
[0002] Environmental perception in low-light scenarios is a technical challenge for autonomous vehicles. Due to poor lighting conditions, the images captured by cameras have low contrast and significant loss of environmental information, which reduces the accuracy of many downstream perception tasks (such as lane detection and target recognition) and results in weak perception capabilities of vehicles at night and in other low-light scenarios.
[0003] Non-visible light images, such as infrared images (wavelength range: 780nm~1mm), capture the relative intensity information of thermal radiation generated by the internal energy of the object's molecules. Therefore, their image quality is less correlated with lighting conditions, and they can still provide stable perception information in low-light scenes. However, non-visible light images have the disadvantage of less texture information than visible light images. By fusing visible light images with infrared images, rich and stable scene information can be obtained, thereby improving the vehicle's perception capability in low-light scenes.
[0004] However, due to limitations in manufacturing and assembly precision, the calibrated visible light and infrared images, under moving conditions, will exhibit registration errors due to camera shake and displacement, thus affecting the quality of image fusion. Therefore, how to correct registration errors is a problem that must be considered when building a multispectral sensor platform.
[0005] Furthermore, besides image fusion, the relative camera pose obtained during image matching is a crucial component of the robust perception system's front-end, indispensable for many downstream tasks (such as SLAM, object detection, and semantic segmentation). Therefore, researching cross-spectral image matching is a necessary prerequisite for conducting research on downstream perception tasks.
[0006] Existing approaches to cross-spectral image registration research include: extracting shared features across different spectra, such as object edges, for matching; and designing algorithms to address the issue of inconsistent intensity gradient directionality, incorporating frequency information based on the phase consistency assumption for matching. However, these studies typically employ manually designed algorithms. For scenarios with drastic changes in lighting conditions (such as entering or exiting tunnels) or insufficient brightness (such as nighttime scenes), due to significant differences in modal information and information loss, they cannot accurately capture corresponding features in the image, thus hindering precise registration. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a feature point matching method for cross-spectral images. This method can reliably extract and match feature points from cross-spectral images (visible light images and non-visible light images) with large modal differences, thereby obtaining rich and stable scene information and improving the perception ability of autonomous vehicles in low-light scenes.
[0008] The objective of this invention can be achieved through the following technical solution: a feature point matching method for cross-spectral images, comprising the following steps:
[0009] S1. Input the calibrated and aligned transspectral image pairs into the adversarial generative network model, and obtain the trained mode conversion model through offline training. The input of the mode conversion model is a visible light image and the output is the corresponding pseudo non-visible light image.
[0010] S2. Input the single-modal non-visible light image into the key point extraction and matching model composed of convolutional neural network and graph neural network for training to obtain the trained matching model;
[0011] S3. Input the image pairs to be matched into the mode conversion model and the matching model, extract key points and feature descriptors from the image pairs to be matched, and perform matching to obtain the matching relationship of key points between images.
[0012] Then, the random sampling consensus algorithm is used to remove outliers, and the matching relationship between images with a confidence level greater than a preset threshold is obtained.
[0013] Furthermore, the transspectral image pairs in step S1 are specifically visible light and non-visible light image pairs under different scenes and lighting conditions.
[0014] Furthermore, the adversarial generative network model in step S1 includes a generator and a discriminator. The generator has a U-Net structure and is used to generate pseudo images in different spectra from the input image through an encoding-decoding process.
[0015] The discriminator structure adopts a fully convolutional network structure to infer the probability that each sub-image block in the input image is a real image, and then calculates the average value as the overall realness probability of the image.
[0016] The generator generates images with labels of 0 and real images with labels of 1. During training, the generator and discriminator are trained alternately, allowing them to compete with each other. The resulting generator model can convert the input visible light image into a realistic pseudo-non-visible light image.
[0017] Furthermore, the loss function used during the training of the adversarial generative network model includes a GAN (Generative Adversarial Network) loss term and an edge loss term, specifically:
[0018]
[0019]
[0020]
[0021] in, Let D(·) be the GAN loss term, D(·) be the discriminator, G(·) be the generator, E(·) be the expectation term, x be the visible light image in the input image pair, y be the non-visible light image in the image pair (i.e., the conversion target), and z be random noise. Thus, when training the generator, minimizing the GAN loss term enables it to generate realistic images sufficient to deceive the discriminator, while maximizing the GAN loss term when training the discriminator improves its ability to distinguish pseudo-generated data from the data.
[0022] The edge loss is λ1, which is the corresponding weight. The edge loss is used to reduce the difference in edge features between the generated pseudo image and the ground image. Sobel(·) includes the Sobel gradient operator, which is used to calculate the sum of squares of the differences in the intensity of the input image in the x and y directions. It uses the smooth-L1 loss form.
[0023] Furthermore, the key point extraction and matching model in step S2 includes a feature extraction network and a feature matching network.
[0024] Furthermore, the feature extraction network consists of three parts: a fully convolutional network backbone for image feature extraction;
[0025] The convolutional keypoint detection head is used to infer the confidence of keypoints at each pixel position based on the feature tensor, and uses Top-K sorting to extract positions with higher confidence as keypoints;
[0026] The convolutional descriptor prediction head is used to infer the descriptor vector at each position based on the feature tensor.
[0027] Furthermore, the feature matching network specifically combines a graph attention network with an optimal transport layer. First, the key point positions and descriptors output by the feature extraction network are used as nodes in the graph. The neural network learns the mapping relationship between feature vectors and attention parameters (key, query, value): linear transformation matrix W, bias vector b.
[0028] Then, attention-weighted summation is used to interact with key point information between image pairs to obtain a matching score matrix;
[0029] The Sinkhorn algorithm is then used to optimize the matching score matrix to satisfy the assignment constraints, thus obtaining the final matching relationship.
[0030] Furthermore, the parameters of the feature extraction network are trained using supervised learning, and the loss function is:
[0031]
[0032]
[0033]
[0034] l d (d, d′; s) = λ d *s*max(0,m) p -d T d′)+(1-s)*max(0,d) T d'-m n )
[0035]
[0036] in, Y represents the input image, descriptor vector, and ground truth values of keypoint locations, respectively, and (·)′ represents the quantity after homography transformation. The keypoint loss is used to supervise the network to accurately predict the location and confidence of keypoints in an image. The feature extraction network maps an image of size (H, W, 3) to a feature tensor of size (H / 8, W / 8, 65). It consists of H / 8*W / 8 vectors of length 65. The first 64 bits of the vector represent the probability values of the corresponding 64 pixels in the original image being keypoints, and the last bit represents the probability value that there are no keypoints in that region of the original image. hw ∈R 65 ,y hw ∈[0,64] represents the slice vector of the feature tensor at coordinates (h,w) and the index of the keypoint, respectively, indicating which pixel the keypoint is located in out of 64 pixels. c W c These represent the length and width of the image, respectively, and x... hwy ∈R represents the eigenvector at coordinates (h,w) with index y as the true value. hw The value at;
[0037] To denote the loss, Hinge Loss is used to differentiate the positive and negative example denote vectors and to monitor their invariance under different homography transformations. Here, d is the denote vector, and m... p m n For the safety factor, λ d Here, s represents the weighting coefficient, and p is used to determine whether a specific keypoint is a positive example. hwLet (h, w) represent the coordinates of a point on the original graph. This represents a homography transformation that projects points from the original image onto the coordinate system of the target image. h′w′ Let (h′, w′) represent the coordinates of the matching point on the target image. The pixels of the output image are then transformed to the target coordinate system, and the projection error is calculated. If the error is less than 8 pixels, then s... hwh′w′ A value of 1 indicates a positive example, while a value of 0 indicates a negative example.
[0038] Furthermore, the specific working process of the feature matching network is as follows:
[0039] Considering that key points in images from different perspectives cannot be guaranteed to correspond one-to-one, a partial allocation method is adopted. That is, the allocation relationship between the N key points in image A and the M key points in image B is defined as an allocation matrix P∈[0,1]. M×N And make it satisfy the allocation constraints:
[0040]
[0041] Among them, 1 M Let P be a vector of length M with all values of 1. This constraint means that the assignment matrix P makes the N key points in graph A correspond one-to-one with the M key points in graph B. There are key points that do not correspond to each other, and there is no one-to-many relationship.
[0042] Obtaining the score matrix using a graph attention network and an optimal transport layer Make S i,j This represents the score for matching point i in image A with point j in image B. The graph attention network treats the keypoints output by the feature extraction network as nodes in the graph, using attention to achieve the interaction of keypoint information between images. The graph attention layer learns the mapping from feature vectors to attention parameters (query, value, key).
[0043]
[0044]
[0045] Where (Q,S)∈{A,B} 2 Q and S are sets of all keypoints in one of images A or B, indicating that the calculation of attention weights includes both attention weights between different images and attention weights within the same image. Let j be the descriptor vector of the j-th layer in the Query graph. Let W be the descriptor vector of the j-th layer in the Source graph. i ,b i (i = 1, 2, 3) are the linear mapping parameters, q i k j vj These are attention parameters. After obtaining the relevant parameters, we use attention-weighted summation to obtain the message that needs to be transmitted between key points:
[0046]
[0047]
[0048] Where, α ij As weight, m ε→i The message passed from the other nodes in the graph to node i;
[0049] The collected information is then concatenated with the descriptors and input into the multilayer perceptron, with the output serving as the input for the next layer.
[0050]
[0051] Where MLP(·) represents a multilayer perceptron, Let i be the descriptor vector of key point i on image A, i.e. node i at the l-th layer, and || is the vector concatenation;
[0052] Then, the inner product operation is performed on the feature vectors corresponding to different key points of the input images A and B:
[0053]
[0054] in, Let S be the eigenvector of the i-th point in graph A. i,j To match the value of the score matrix S at position (i,j), which represents the matching probability between the i-th keypoint in image A and the j-th keypoint in image B, These represent the sets of key points in Figure A and Figure B, respectively.
[0055] The above allocation constraint can be expressed as:
[0056]
[0057] In order to include cases where keypoints are not assigned, the assignment matrix is expanded to an enhanced assignment matrix of size (M+1, N+1). An extra bit is used to explicitly indicate that a key point is not assigned.
[0058] Obtain the fractional matrix S i，j The optimal allocation matrix P that satisfies the constraints is obtained through the optimal transport layer. * :
[0059]
[0060] To maximize this expression, calculate the first derivative of the objective function with respect to each position in the allocation matrix and set it to zero:
[0061]
[0062]
[0063]
[0064] P i，j =u i K i，j v j
[0065] Where, d λ M(a,b) is the Sinkhorn distance:
[0066]
[0067] The second term in this distance is an entropy constraint, which encourages the allocation term to shift to as many objectives as possible, ensuring sufficient density of the solution. α and β are related to P and are intermediate variables to be optimized, while λ is a hyperparameter.
[0068] Introduce the conservation condition for optimal transmission:
[0069] u⊙(Kv)=a
[0070] v⊙(K T u)=b
[0071] Where ⊙ represents the Hadamard product of vector operations, i.e., element-wise multiplication, and P is optimized by iterating through u and v:
[0072]
[0073]
[0074] Among them, u (l+1) This represents the (l+1)th iteration value of u;
[0075] The loss function used by the feature matching network is to minimize the negative log-likelihood of the assignment matrix P to optimize the matching model:
[0076]
[0077] To train this matching model, supervision is required: ground truth of matching relationships. Thus, the allocation matrix is obtained. The truth label in the formula, This represents the set of key points in graph A. I represents the set of key points in graph B, and I represents the set of unmatched points in graph A. Let the set of unmatched points in graph B be represented by the negative log-likelihood function Loss. nll for:
[0078]
[0079] In the formula i gt This indicates the ordinal number corresponding to the truth value label. This represents the value at the truth label of the assignment vector predicted by the network, through... Calculate the sum, average, and then calculate the negative logarithm.
[0080] Further, in step S3, the visible light image in the image pair to be matched is input into the mode conversion model, and the corresponding pseudo non-visible light image is output; then the pseudo non-visible light image and the non-visible light image in the image pair to be matched are input into the matching model to obtain the matching relationship of key points between the images.
[0081] Compared with the prior art, the present invention has the following advantages:
[0082] I. This invention addresses feature point matching for cross-spectral images. On one hand, it trains an adversarial generative network to obtain a mode conversion model, capable of converting visible light images into corresponding pseudo-non-visible light images. On the other hand, it trains a keypoint extraction and matching network composed of convolutional neural networks and graph neural networks to obtain a matching model, capable of directly registering single-modality non-visible light images. Thus, the mode conversion model generates pseudo-non-visible light images that are extremely similar to real non-visible light images, and the matching model is then used to extract key points and feature descriptors to complete the matching. This enables reliable feature point extraction and matching for cross-spectral images with significant modal differences.
[0083] Second, this invention specifically improves the loss function of the Generative Adversarial Network (GAN) when training the modality conversion model. This loss function includes a GAN loss term and an edge loss term. The GAN loss term supervises the generator to produce pseudo-images that the discriminator cannot distinguish, while simultaneously supervising the discriminator to differentiate between pseudo-images generated by the generator. The edge loss term supervises the generator to produce pseudo-non-visible light images with edges similar to the target non-visible light image. This enables pixel-level modality conversion of heterogeneous image pairs with significant modal differences, greatly improving the similarity between images. The converted image pairs can be directly input into a matching network trained on single-modality image data. This not only reduces the dependence on the annotation of scarce cross-spectral datasets but also effectively improves the network's ability to capture common edge features between modalities, thereby generating high-quality pseudo-non-visible light images, which is beneficial for improving the stability and accuracy of subsequent matching.
[0084] III. When training the matching model, the feature extraction network of this invention includes: a fully convolutional backbone network for converting the input image into a high-dimensional feature tensor; a fully convolutional keypoint detection head for inferring the probability and location of keypoints in each region of the image; and a fully convolutional descriptor head for inferring descriptor vectors for each region of the image.
[0085] The feature matching network adopts a modeling approach that combines graph attention network and optimal transport layer. The key point positions and descriptors output by the feature extraction network are used as nodes in the graph. The neural network learns the mapping relationship between feature vectors and attention parameters. Attention weighted summation is used to interact key point information between image pairs. The resulting matching score matrix is then optimized using the Sinkhorn algorithm to satisfy the assignment constraints, thus obtaining the final matching relationship.
[0086] Therefore, a model adapted to extract key points and descriptors from cross-spectral images is obtained based on neural network learning. This model can perform accurate registration processing, thereby ensuring the stability of the perception capability of autonomous vehicles in low-light scenarios and improving the overall safety of autonomous vehicles. Attached Figure Description
[0087] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0088] Figure 2 This is a schematic diagram illustrating the application process of an example.
[0089] Figure 3 This is a schematic diagram of the dataset used when training the modality transition model in the embodiment;
[0090] Figure 4 This is a schematic diagram of the modality transition model architecture;
[0091] Figure 5 This is a schematic diagram of edge loss in a mode transition model;
[0092] Figure 6a and 6b This is a schematic diagram of the image feature extraction and matching model architecture;
[0093] Figure 7a and 7b This is a schematic diagram of cross-spectral image feature matching in an embodiment. Detailed Implementation
[0094] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0095] Example
[0096] like Figure 1As shown, a feature point matching method for cross-spectral images is characterized by comprising the following steps:
[0097] S1. Input the transspectral image pairs (calibrated and aligned visible light and non-visible light image pairs) into the adversarial generative network model and train it offline to obtain the trained mode conversion model.
[0098] S2. Input the single-modal non-visible light image into the key point extraction and matching model composed of convolutional neural network and graph neural network for training to obtain the trained matching model;
[0099] S3. Input the image pairs to be matched into the mode conversion model and the matching model, extract key points and feature descriptors from the visible light and non-visible light image pairs to be matched, and perform matching to obtain the matching relationship of key points between images.
[0100] Then, the random sampling consensus algorithm is used to remove outliers, and the matching relationships between matching images with a confidence level greater than a preset threshold are obtained.
[0101] In this embodiment, considering the promotion of infrared cameras in autonomous driving, non-visible light images are set as infrared images. The process of applying the above technical solution in this embodiment is as follows: Figure 2 As shown, specifically, during step S1, it is first necessary to acquire transspectral image pairs for training the modality conversion model, such as... Figure 3 As shown, a transspectral image pair dataset was collected and calibrated for training and validation testing of the modality conversion model. To ensure the algorithm's robustness to illumination changes and scenes, transspectral image pairs under different scenes and illumination conditions were collected.
[0102] The modality conversion model trained in step S1 uses an adversarial generative network to convert the input visible light image into a pseudo infrared image, which has a higher similarity to the real infrared image.
[0103] like Figure 4 As shown, the modality conversion model uses the encoder-decoder structure of U-Net as a pseudo-image generator to convert the input of the visible light image into a pseudo-infrared image. By supervising the alignment of image pairs, it is trained in the form of a generative adversarial network to obtain pseudo-images with a high degree of similarity to real infrared images. In addition, the discriminator adopts a fully convolutional network to output the probability value of each image patch as a real image. The average value of each region is used to obtain the overall authenticity probability value of the image.
[0104] In this embodiment, the generator has 16 network layers: 8 layers each for the encoder and decoder. Each encoder layer includes a convolutional layer, a normalization layer, and an activation layer, while the decoder includes a convolutional layer, a normalization layer, an activation layer, and an upsampling layer. The number of convolutional kernels in the last 4 encoder layers and the first 4 decoder layers is 512, and the number of convolutional kernels in the first 4 encoder layers and the last 4 decoder layers are 64, 128, 256, 512, 512, 256, 128, and 64, respectively. The convolutional kernel size is [4,4], with a stride of 2, used to reduce the size. The discriminator uses a fully convolutional network structure, with each layer containing a convolutional layer, a regularization layer, and an activation layer. The number of convolutions in each layer is 128, 256, 512, and 1, respectively, and the output is the realism of the image region.
[0105] like Figure 5 As shown, this technical solution employs edge loss during generator training to minimize the edge differences between the generated image and the ground truth image, thereby improving the similarity between the generated image and the real infrared image. In this technical solution, the adversarial generative network model requires calibrated and aligned infrared-visible image pairs during training. The loss function used for neural network parameter optimization includes a GAN loss term and an edge loss term.
[0106]
[0107] in The loss of GAN is as follows:
[0108]
[0109] In the formula, D(·) is the discriminator, G(·) is the generator, E(·) is the expectation, x is the input image, y is the output image, and z is random noise. This loss supervises the generator to generate pseudo-images that the discriminator cannot distinguish, while simultaneously supervising the discriminator to distinguish pseudo-images generated by the generator. The specific training process involves alternately optimizing both, improving performance through a game between them.
[0110] The edge loss is defined as follows, with λ1 as its weight:
[0111]
[0112] In the formula, Sobel(·) includes the Sobel operator, which is used to calculate the difference of image intensity values. It uses the smooth-L1 loss form, which is used to supervise the generator to generate pseudo-infrared images that are similar to the edges of the target infrared image.
[0113] During step S2, the feature extraction and matching network is divided into a feature extraction network and a feature matching network. The feature extraction network consists of three parts: a fully convolutional network backbone for image feature extraction, a convolutional keypoint detection head for inferring the confidence of keypoints at each pixel position based on the feature tensor and using Top-K sorting to extract positions with higher confidence as keypoints, and a convolutional descriptor prediction head for inferring descriptor vectors at each position based on the feature tensor.
[0114] The parameters of the feature extraction network are trained using supervised learning, and the loss function is:
[0115]
[0116] in Y represents the input image, descriptor vector, and ground truth values of keypoint locations, respectively, and (·)′ represents the quantity after homography transformation. Losses at critical points:
[0117]
[0118] This is used to supervise the network's accurate prediction of keypoint locations and confidence levels in images. The network maps an image of size (H, W, 3) to a feature tensor of size (H / 8, W / 8, 65). It consists of H / 8*W / 8 vectors of length 65. The first 64 bits of the vector represent the probability values of the corresponding 64 pixels in the original image being keypoints, and the last bit represents the probability value that there are no keypoints in that area of the original image. hw ∈R 65 ,y hw ∈[0,64] represents the slice vector of the feature tensor at coordinates (h,w) and the index truth value of the keypoint position (indicating which pixel the keypoint is located in out of 64 pixels), respectively. c W c The x and x values represent the length and width of the image, respectively. hwy ∈R represents the eigenvector at coordinates (h,w) with index y as the true value. hw The value at that location.
[0119] Descriptor loss function Hinge Loss is used to separate positive and negative examples to describe the difference between subvectors, and its invariance under different homography transformations is monitored:
[0120]
[0121] in
[0122] l d (d,d′;s)=λ d *s*max(0,m p-d T d′)+(1-s)*max(0,d T d′-m n )
[0123] In the formula, d is the descriptor vector; for simplicity, the variable subscripts are omitted here. p m n For the safety factor, λ d s represents the weighting coefficient. s is used to determine whether a specific keypoint is a positive example, i.e.:
[0124]
[0125] p hw Let (h, w) represent the coordinates of a point on the original graph. This represents a homography transformation that projects points from the original image onto the coordinate system of the target image. h′w′ This represents the coordinates (h′, w′) of the matching point on the target image. The above formula calculates the projection error after transforming the pixels of the output image to the target coordinate system. If the error is less than 8 pixels, then s... hwh′w′ A value of 1 indicates a positive example, while a value of 0 indicates a negative example.
[0126] The feature matching network combines a graph attention network with an optimal transport layer. It extracts the key point locations and descriptors from the network output as nodes in the graph, and uses a neural network to learn the mapping relationship between feature vectors and attention parameters (key, query, value): linear transformation matrix W and bias vector b. It uses attention weighted summation to interact key point information between image pairs, and the resulting matching score matrix is optimized using the Sinkhorn algorithm to satisfy the assignment constraints, thus obtaining the final matching relationship.
[0127] Considering that key points in images from different perspectives cannot be guaranteed to correspond one-to-one, a partial allocation method is adopted. That is, an allocation matrix P∈[0,1] is defined to represent the allocation relationship between the N key points in image A and the M key points in image B. M×N And make it satisfy the allocation constraints:
[0128]
[0129] In the formula 1 M Let P be a vector of length M, with all values being 1. This constraint means that the assignment matrix P ensures that the N keypoints in graph A correspond one-to-one with at most M keypoints in graph B, with some keypoints having no corresponding values and no one-to-many relationship. To obtain assignment-related information using the keypoints and descriptor information obtained from the feature extraction model, this invention utilizes a graph attention network and an optimal transport layer to obtain a score matrix. Make S i,jThis represents the score for matching point i in graph A with point j in graph B.
[0130] Specifically, graph attention networks treat keypoints output by feature extraction networks as nodes in a graph, using attention to enable the interaction of keypoint information between images. Specifically, the graph attention layer learns the mapping from feature vectors to attention parameters (query, value, key):
[0131]
[0132]
[0133] In the formula, (Q,S)∈{A,B} 2 , where Q and S are sets of all key points in one of the images A or B, indicating that the calculation of attention weights includes both attention weights between different images and attention weights within the same image. Let j be the descriptor vector of the j-th layer in the Query graph. Let W be the descriptor vector of the j-th layer in the Source graph. i ,b i (i = 1, 2, 3) are the linear mapping parameters, q i k j v j These are attention parameters. After obtaining the relevant parameters, we use an attention-weighted summation method to obtain the message that needs to be transmitted between key points:
[0134]
[0135]
[0136] In the formula, α ij As weight, m ε→i This represents the message passed from the other nodes in the graph to node i. Finally, the collected information is concatenated with the descriptor and input into the multilayer perceptron, with the output serving as the input for the next layer.
[0137]
[0138] In the formula, MLP(·) represents a multilayer perceptron. Let be the descriptor vector of keypoint i (i.e. node i) in image A at layer l, and || be the vector concatenation.
[0139] Next, the inner product operation is performed on the feature vectors corresponding to different key points of the two input images (named A and B respectively):
[0140]
[0141] In the formula, Let S be the eigenvector of the i-th point in graph A. i,j The value of the matching score matrix S at position (i,j) is used to represent the matching probability between the i-th key point in image A and the j-th key point in image B. These represent the sets of key points in Figure A and Figure B, respectively.
[0142] The above allocation constraint can be expressed as:
[0143]
[0144] In the formula, to include cases where keypoints are not assigned, the assignment matrix is expanded to an enhanced assignment matrix of size (M+1, N+1). An extra bit is used to explicitly indicate that a key point is not assigned.
[0145] Obtain the fractional matrix S i，j The optimal allocation matrix P that satisfies the constraints is obtained through the optimal transport layer. * :
[0146]
[0147] To maximize this expression, calculate the first derivative of the objective function with respect to each position in the allocation matrix and set it to zero:
[0148]
[0149]
[0150]
[0151] P i,j =u i K i,j v j
[0152] In the formula d λ M(a,b) is the Sinkhorn distance:
[0153]
[0154] The second term in this distance is an entropy constraint, which encourages the allocation term to shift to as many objectives as possible, ensuring sufficient density of the solution. α and β are related to P and serve as intermediate variables to be optimized, while λ is a hyperparameter. Introduce the conservation condition for optimal transmission:
[0155] u⊙(Kv)=a
[0156] v⊙(KT u)=b
[0157] Where ⊙ represents the Hadamard product of vector operations, i.e., element-wise multiplication, and P is optimized by iterating through u and v:
[0158]
[0159]
[0160] In the formula u (l+1) This represents the (l+1)th iteration value of u.
[0161] The matching network uses a loss function that minimizes the negative log-likelihood of the assignment matrix P to optimize the matching model.
[0162]
[0163] To train this matching model, supervision is required: ground truth of matching relationships. Thus, the allocation matrix is obtained. The truth label in the formula, This represents the set of key points in graph A. I represents the set of key points in graph B, and I represents the set of unmatched points in graph A. Let the set of unmatched points in graph B be represented by the negative log-likelihood function Loss. nll :
[0164]
[0165] In the formula i gt This indicates the ordinal number corresponding to the truth value label. This represents the value at the truth label of the assignment vector predicted by the network, through... Calculate the sum, average, and then calculate the negative logarithm.
[0166] In this embodiment, as Figure 6a and 6b As shown, a fully convolutional keypoint feature extraction model is adopted, consisting of a VGG backbone network, a keypoint detection part, and a descriptor inference part. The backbone network consists of four layers. Except for the last layer, each layer has the following structure: convolutional layer, non-linear activation layer, convolutional layer, activation layer, and max-pooling layer (2x2). The number of channels is doubled every two layers, while the feature map size is halved. Specifically, the number of convolutional kernels is 64, 64, 128, and 128, with a size of [3,3]. An image of size (H×W×1) is compressed into a feature tensor of size H / 8×W / 8×256. The keypoint detection part maps the feature tensor to a keypoint score map of the original image size, representing the confidence that each pixel is a keypoint. To reduce computation, the network does not upsample the feature map but instead regresses a tensor X∈R.(H / 8×W / 8×65) The last dimension contains 65 elements, representing each pixel block (the original image has 64 pixels corresponding to the features). Figure 1 The confidence scores of 64 pixels (in 64 pixels) are used to represent keypoints, plus a confidence score representing the absence of keypoints. Then, a Softmax operation is performed on the channels, and the shape of the tensor is changed to X'∈R^(H×W). The descriptor decoder maps the feature map to a descriptor map and restores the descriptor map to the original image size using bicubic interpolation. The specific network structure is: convolutional layer, non-linear activation layer, convolutional layer, and regularization layer. The number of convolutional kernels is 256, 256 (the length of the descriptor), and the size is [3,3]. Afterwards, the graph attention layer extracts the relationships between descriptors and converges them to keypoints in the form of message passing. The message passing is calculated using an attention mechanism. After 9 graph attention layers, enhanced descriptor vectors are output. The inner product of each enhanced descriptor vector is calculated to obtain a fraction matrix. The final result is obtained using the Sinkhorn algorithm. Figure 7a and 7b The accurate matching relationship is shown. The evaluation metrics for key points include:
[0167] 1) Key point repetition rate (Rep):
[0168]
[0169] Match high-quality ={(p1,p2)||H gt p1-p2| L2 <threshold}
[0170] If keypoints satisfy the coplanarity assumption (lie on the same plane), then keypoints can be mapped between images using a homography matrix. Therefore, the truth homography matrix H... gt The keypoint p1 on image a is mapped to image b, and the distance between the mapped point and its corresponding point p2 is calculated. By setting a distance threshold, it is determined whether the keypoint is valid, or in other words, a high-quality keypoint. The ratio of high-quality keypoints to the number of matches is the repetition rate. It is worth noting that this method calculates the repetition rate based on the matching results, not just the detection results. Therefore, this metric measures the combined performance of the detector and the matcher. The larger this value, the closer it is to 1, the better the matching algorithm performance. The distance threshold is set to 3.
[0171] 2) LE (Localization Error):
[0172] LE = Average pixel distance between point pairs
[0173] Point pairs with a distance of less than 4 pixels are considered valid.
[0174] 3) Homography Accuracy (HA) estimation accuracy:
[0175]
[0176] First, the similarity of descriptors is measured using L2 distance. Then, based on the matching results and combined with a random sample consensus algorithm, the homography transformation matrix between the two images is estimated. On the image obtained after performing an inverse F transform on the original image, the coordinates of four edge points are obtained. The average error distance between these four points under the estimated homography transformation and the true value of the transformation matrix is calculated. This metric is also called the Homography Error (HE). Different thresholds are used to determine whether the estimation is correct; in this embodiment, 1, 3, 5, 10, 20, and 50 pixels are used as evaluation thresholds.
[0177] 4) MS (Matching Score):
[0178]
[0179] A match with a distance less than 1.2 is considered a correct match.
[0180] The verification results of this embodiment demonstrate that the proposed technical solution can reliably extract and match feature points from cross-spectral images with significant modal differences. This solution employs a generative adversarial network (GAN) and optimizes it using an edge similarity loss function. It performs pixel-level modality conversion on heterogeneous image pairs with significant modal differences, greatly improving the similarity between images. The converted image pairs can be directly input into a matching network trained on single-modal image data, reducing the dependence on annotations in scarce cross-spectral datasets. This enhances the network's ability to capture common features among visually diverse cross-spectral images, improves the network's learning performance, and bridges the gap for the widespread application of infrared cameras in the field of autonomous driving perception.
[0181] This technical solution focuses on feature point matching of cross-spectral images. The solution generates corresponding pseudo images based on visible light images through a mode conversion module, which has a very high similarity to infrared images. Therefore, feature extraction and matching machines trained on single-modal datasets can be directly used for registration, reducing the dependence on cross-spectral datasets and annotation work.
[0182] This technical solution makes targeted improvements to the loss function of the generative adversarial model, enhancing the network's ability to capture common edge features between modalities, thereby generating high-quality pseudo-infrared images and improving the stability and accuracy of matching.
[0183] This technical solution is based on neural network learning to create a model that is adapted to extracting points and descriptors from cross-spectral images, thus bridging the gap for the widespread application of infrared cameras in the field of autonomous driving perception. The goal is to achieve stable perception capabilities for the system in low-light scenarios and improve the overall safety of the system.
Claims
1. A method of feature point matching across multispectral images, the method comprising: Includes the following steps: S1. Input the calibrated and aligned transspectral image pairs into the adversarial generative network model, and obtain the trained mode conversion model through offline training. The input of the mode conversion model is a visible light image and the output is the corresponding pseudo non-visible light image. S2. Input the single-modal non-visible light image into the key point extraction and matching model composed of convolutional neural network and graph neural network for training to obtain the trained matching model; The key point extraction and matching model includes a feature extraction network and a feature matching network. The feature extraction network consists of three parts: a fully convolutional network backbone for image feature extraction. The convolutional keypoint detection head is used to infer the confidence of keypoints at each pixel position based on the feature tensor, and uses Top-K sorting to extract positions with higher confidence as keypoints; The convolutional descriptor prediction head is used to infer the descriptor vector at each position based on the feature tensor; The parameters of the feature extraction network are trained using supervised learning, and the loss function is: ， in, , , These are the input image, descriptor vector, and ground truth values of keypoint locations, respectively. This represents the quantity after homography. The keypoint loss is used to supervise the network to accurately predict the location and confidence of keypoints in an image. The feature extraction network maps an image of size (H, W, 3) to a feature tensor of size (H / 8, W / 8, 65). It consists of H / 8 * W / 8 vectors, each of length 65. The first 64 bits of the vector represent the probability values of the corresponding 64 pixels in the original image being keypoints, and the last bit represents the probability value that there are no keypoints in that area of the original image. The feature tensor is in coordinates of The slice vector at a given location and the ground truth index of the keypoint indicate which pixel the keypoint is located in out of 64 pixels. These are the length and width of the image, respectively. Representing coordinates The eigenvector at the index of the truth value The value at; To describe the sub-loss, Hinge Loss is used to differentiate between positive and negative example sub-vectors, and its invariance under different homography transformations is monitored. To describe the subvector, , For safety reasons, These are the weighting coefficients. s Used to determine whether a specific key point is a positive example. Represents the coordinates of a point on the original image. , This represents a homography transformation that projects points from the original image onto the coordinate system of the target image. Represents the coordinates of the matching point on the target image. This process transforms the pixels of the output image to the target coordinate system and calculates the projection error. If the error is less than 8 pixels... A value of 1 indicates a positive example, while a value of 0 indicates a negative example. S3. Input the image pairs to be matched into the mode conversion model and the matching model, extract key points and feature descriptors from the image pairs to be matched, and perform matching to obtain the matching relationship of key points between images. Then, the random sampling consensus algorithm is used to remove outliers, and the matching relationship between images with a confidence level greater than a preset threshold is obtained.
2. The feature point matching method for transspectral images according to claim 1, characterized in that, In step S1, the transspectral image pairs specifically refer to visible light and non-visible light image pairs under different scenes and lighting conditions.
3. The feature point matching method for transspectral images according to claim 1, characterized in that, In step S1, the adversarial generative network model includes a generator and a discriminator. The generator structure is U-Net, which is used to generate pseudo-images in different spectra from the input image through encoding and decoding. The discriminator structure adopts a fully convolutional network structure to infer the probability that each sub-image block in the input image is a real image, and then calculates the average value as the overall realness probability of the image. The generator generates images with labels of 0 and real images with labels of 1. During training, the generator and discriminator are trained alternately, allowing them to compete with each other. The resulting generator model can convert the input visible light image into a realistic pseudo-non-visible light image.
4. The feature point matching method for transspectral images according to claim 3, characterized in that, The loss function used during the training of the adversarial generative network model includes a GAN (Generative Adversarial Network) loss term and an edge loss term, specifically: in, Let D(·) be the GAN loss term, D(·) be the discriminator, G(·) be the generator, E(·) be the expectation term, x be the visible light image in the input image pair, y be the non-visible light image in the image pair (i.e., the conversion target), and z be random noise. Thus, when training the generator, minimizing the GAN loss term enables it to generate realistic images sufficient to deceive the discriminator, while maximizing the GAN loss term when training the discriminator improves its ability to distinguish pseudo-generated data from the data. For edge loss, For the corresponding weights, the edge loss is used to reduce the difference in edge features between the generated pseudo-image and the ground truth image. Sobel(·) includes the Sobel gradient operator, which is used to calculate the sum of squares of the differences in intensity of the input image in the x and y directions, using the smooth-L1 loss form.
5. The feature point matching method for transspectral images according to claim 1, characterized in that, The feature matching network specifically combines a graph attention network with an optimal transport layer. First, the key point positions and descriptors output by the feature extraction network are used as nodes in the graph. The neural network learns the mapping relationship between feature vectors and attention parameters (key, query, value): linear transformation matrix W, bias vector b. Then, attention-weighted summation is used to interact with key point information between image pairs to obtain a matching score matrix; The Sinkhorn algorithm is then used to optimize the matching score matrix to satisfy the assignment constraints, thus obtaining the final matching relationship.
6. The feature point matching method for a transspectral image according to claim 5, characterized in that, The specific working process of the feature matching network is as follows: Considering that key points in images from different perspectives cannot be guaranteed to correspond one-to-one, a partial allocation method is adopted, that is, defining the allocation matrix to represent the distribution relationship between the N key points in image A and the M key points in image B. And make it satisfy the allocation constraints: in, Let P be a vector of length M with all values of 1. This constraint means that the assignment matrix P makes the N key points in graph A correspond one-to-one with the M key points in graph B. There are key points that do not correspond to each other, and there is no one-to-many relationship. Obtaining the score matrix using a graph attention network and an optimal transport layer , making This represents the score for matching point i in image A with point j in image B. The graph attention network treats the keypoints output by the feature extraction network as nodes in the graph, using attention to achieve the interaction of keypoint information between images. The graph attention layer learns the mapping from feature vectors to attention parameters (query, value, key). in, Q and S are sets of all keypoints in one of images A or B, indicating that the calculation of attention weights includes both attention weights between different images and attention weights within the same image. Let j be the descriptor vector of the j-th layer in the Query graph. Let j be the descriptor vector of the j-th layer in the Source graph. ) represents the linear mapping parameters. , , These are attention parameters. After obtaining the relevant parameters, we use attention-weighted summation to obtain the message that needs to be transmitted between key points: in, For the weights, Softmax(z) i ) = , The message passed from the other nodes in the graph to node i; The collected information is then concatenated with the descriptors and input into the multilayer perceptron, with the output serving as the input for the next layer. Where MLP(·) represents a multilayer perceptron, Let key point i on image A, i.e., node i at the th... The layer's descriptor vector, where || represents vector concatenation; Then, the inner product operation is performed on the feature vectors corresponding to different key points of the input images A and B: in, Let i be the feature vector of the i-th point in graph A. To match the score matrix The value at position (i,j) represents the matching probability between the i-th keypoint in graph A and the j-th keypoint in graph B. These represent the sets of key points in Figure A and Figure B, respectively. The above allocation constraint can be expressed as: In order to include cases where keypoints are not assigned, the assignment matrix is expanded to an enhanced assignment matrix of size (M+1, N+1). An extra bit is used to explicitly indicate that a key point is not assigned. ; Obtain the fraction matrix The optimal allocation matrix that satisfies the constraints is obtained through the optimal transport layer. : To maximize this expression, calculate the first derivative of the objective function with respect to each position in the allocation matrix and set it to zero: in, Sinkhorn distance: The second term in this distance is an entropy constraint, which encourages the allocation term to shift to as many objectives as possible, ensuring that the solution has sufficient density. , All variables related to P are intermediate variables to be optimized. For hyperparameters, , , ; Introduce the conservation condition for optimal transmission: in, The Hadamard product is a vector operation, which is the element-wise multiplication, performed iteratively. , optimization : in, Indicates u's first The value of the next iteration; The loss function used by the feature matching network is to minimize the negative log-likelihood of the assignment matrix P to optimize the matching model: To train this matching model, supervision is required: ground truth of matching relationships. Thus, the allocation matrix is obtained. The truth label in the formula, This represents the set of key points in graph A. This represents the set of key points in graph B. This represents the set of unmatched points in graph A. Let the set of unmatched points in graph B be represented by the negative log-likelihood function. for: In the formula This indicates the ordinal number corresponding to the truth value label. This represents the value at the truth label of the assignment vector predicted by the network, through... Calculate the sum, average, and then calculate the negative logarithm.
7. A feature point matching method for transspectral images according to any one of claims 1 to 6, characterized in that, In step S3, the visible light image in the image pair to be matched is input into the mode conversion model, and the corresponding pseudo non-visible light image is output. Then, the pseudo non-visible light image and the non-visible light image in the image pair to be matched are input into the matching model to obtain the matching relationship of key points between the images.