An image matching method based on progressive neighbor consistency mining

Abstract

The application discloses an image feature matching method based on gradual neighbor consistency mining, and comprises the following steps: 1, all features are constructed into an initial image feature matching set S; 2, the initial image feature matching set S is pruned to obtain a reliable candidate image feature matching set and corresponding inlier weight; 3, the corresponding essential matrix is predicted according to the candidate image feature matching set and the corresponding inlier weight; 4, the inlier probability w of all input initial image feature matching is obtained according to the essential matrix; finally, the essential matrix of two matching images and the probability of correct matching inliers are estimated. The application has obtained significant performance improvement in image matching.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision technology, and specifically relates to an image matching method. Background Technology

[0002] Image feature matching aims to obtain the accurate correspondence between feature point sets of two given matching images and recover the corresponding camera pose. As a link between lower-level vision and higher-level vision, image feature matching technology has wide applications in many vision fields, such as image stitching, image retrieval, 3D reconstruction, and simultaneous localization and mapping (SMR). Image feature matching is essentially a process of pairing feature point sets; it is a complex NP-complete combinatorial optimization problem with high computational complexity. Furthermore, due to issues such as changes in viewpoint and lighting, repetitive structures, occlusion, and blurring between matched images, incorrect feature matching (outliers) can significantly degrade the performance of image feature matching methods, severely hindering the implementation of subsequent applications. Therefore, researching a more robust, accurate, and faster image feature matching method is of great significance.

[0003] Most image feature matching algorithms consist of three basic steps: feature point extraction, feature point description, and feature matching. Feature point extraction is used to obtain matchable regions in an image, and point features are the most widespread and stable. Image point features are typically key points with significant structural information or strong discriminative power extracted from local image regions, including corner points, endpoints of line segments, intersections of lines, and center points of morphological regions. Feature point detection operators are the most commonly used methods for extracting image feature points. Feature point description utilizes the local regions surrounding the feature points, which usually possess more discriminative information, making it more conducive to establishing accurate matching relationships later. After obtaining the feature points and their corresponding feature descriptors, a preliminary matching relationship can be established based on the similarity criteria of the feature descriptors. Matching pruning can be used to further select more accurate matches for estimating more accurate camera pose. Matching pruning, as a post-processing step, plays a crucial role in improving the performance of image feature matching.

[0004] With the development of deep learning technology, some researchers have used learning-based methods to replace one or more steps in image feature matching algorithms. The powerful feature representation capabilities of deep neural networks in identifying the correctness of feature matching have attracted widespread attention. However, how to design effective network modules to explore feature matching context information for matching pruning remains to be studied. Summary of the Invention

[0005] To address the shortcomings and deficiencies of existing image feature matching technologies, this invention proposes an image feature matching method based on progressive neighbor consistency mining. The method takes the initial feature matching established by the feature extraction method as input, mines the consistency information of three different types of neighbors, and uses a series of carefully designed network structures to obtain features with stronger representation capabilities for neural network learning. Finally, it estimates the essential matrix of the two matching images and the probability of a correct match.

[0006] This invention is achieved using the following technical solution:

[0007] An image matching method based on progressive neighbor consistency mining, comprising the following steps:

[0008] Step 1: Use a feature extraction algorithm to extract features from the given matching image pair I and I'. Each image feature is represented by keypoint coordinates and the corresponding descriptor. Construct all features into an initial image feature matching set S:

[0009]

[0010] Among them, s i This indicates the i-th initial feature match, and N represents the total number of matches. i ,v i ) and (u' i ,v' i ) respectively represent s i In the image, normalized coordinates of pairs I and I';

[0011] Step 2: Prune the initial image feature matching set S based on the inlier probabilities learned by the neighbor consistency mining network to obtain the candidate image feature matching set. and the corresponding interior point weights

[0012] Step 3: Match the candidate image features based on the set obtained in Step 2 and the corresponding interior point weights Predict the corresponding essential matrix As shown in the following formula:

[0013]

[0014] Where we() represents the weighted eight-point algorithm, This represents the essential matrix based on the prediction;

[0015] Step 4: Based on the essential matrix obtained in Step 3 The inlier probability w of the initial image feature matching for all inputs is obtained as shown in the following equation:

[0016]

[0017] Where v() represents the full-size validation operation;

[0018] Finally, estimate the essential matrix of the two matching images and output the probability of correctly matching interior points.

[0019] Compared to existing methods, this invention can obtain a more reliable candidate matching set and estimate more accurate image feature matching results and camera pose, achieving a significant performance improvement in image matching. Attached Figure Description

[0020] Figure 1 Here is a flowchart of an image matching method based on progressive neighbor consistency mining according to the present invention: (a) overall flowchart of the method, (b) pruning flowchart.

[0021] Figure 2 A schematic diagram of a network model for mining neighbor consistency;

[0022] Figure 3 The structure diagram of the self-context extraction layer (SCE layer) and cross-context interaction layer (CCI layer) proposed in this invention is shown in the figure.

[0023] Figure 4 This is a schematic diagram illustrating the actual image matching effect of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0025] like Figure 1 As shown, this invention provides an image feature matching method based on progressive neighbor consistency mining, which specifically includes the following steps:

[0026] Step 1: Use a feature extraction algorithm to extract image features from the given matching image pair I and I'. Each image feature is represented by keypoint coordinates and the corresponding descriptor. Construct all features into an initial image feature matching set S:

[0027]

[0028] Among them, s i This indicates the i-th match, and N represents the total number of matches. i ,v i ) and (u' i ,v' i ) respectively represent s i In the image, normalized coordinates of pairs I and I';

[0029] Step 2: Prune the initial image feature matching set S based on the inlier probabilities learned by the neighbor consistency mining network to obtain a reliable candidate image feature matching set; the specific operation is as follows:

[0030] Step 2-1: Process the initial image feature matching set S through 3 residual blocks (ResNet) to obtain the intermediate feature mapping set. Where d represents the channel dimension and N represents the total number of matches, three types of neighbor spaces are obtained; wherein, the coordinate space is the initial feature matching The feature space is an intermediate feature map. The global graph space Used to reflect the consistency of internal points at the global level;

[0031] Step 2-2: Calculate the global graph space, the process is as follows:

[0032] Calculate the initial interior point weights w based on the intermediate feature map F. p =ReLU(tanh(MLP((F))), where MLP() represents a multilayer perceptron layer used to reduce the channel dimension to 1, and tanh() and ReLU() represent activation functions;

[0033] Construct a weighted global graph in, Represents all image feature matching nodes, ε g Representing undirected edges, through their corresponding weights Implement feature matching and connection for every two images, where , These represent the initial inlier weights for matching the i-th and j-th image features, respectively. The weights of the connecting edges are represented to obtain an adjacency matrix. To explicitly describe the long-range dependencies between image feature matches;

[0034] The global graph space F is obtained using spectral convolution operations. g As shown in the following formula:

[0035]

[0036] in, Let F represent the graph Laplacian matrix, and let F represent the intermediate eigenmap. I represents an adjacency matrix with self-connections. N Let A denote the diagonal identity matrix, and let A denote the adjacency matrix. Let W represent the angle matrix. g Indicates trainable weights;

[0037] Step 2-3: Based on the three neighbor spaces obtained in Step 2-1, use the k-nearest neighbor search strategy to find three different types of neighbors for each image feature matching: spatial k-neighbors, feature space k-neighbors, and global graph k-neighbors.

[0038] Steps 2-4: Construct spatial neighbor embeddings for the three different types of neighbors respectively. Feature space neighbor embedding and global graph neighbor embedding Embedded by spatial neighbors For example, node Indicates the initial image feature matching s i k-nearest neighbors in space, undirected edge Connect its coordinate space neighbors to construct edges. As shown in the following formula:

[0039]

[0040] Among them, f i and These represent the i-th initial image feature matches s, respectively. i and its j-th (j = 1, 2, ..., k) spatial neighbor In the feature map of the intermediate feature space F, [] represents the connection operation along the channel dimension;

[0041] Construct coordinate space neighbor embeddings for all feature matches Similarly, obtain the feature space neighbor embeddings of all matching features. and global graph neighbor embedding

[0042] Steps 2-5: To adapt to complex matching scenarios, a new neighbor consistency block is used to progressively mine the consistency of three different types of neighbors, such as... Figure 2 The diagram shows a schematic of the neighbor consistency mining network model proposed in this invention. As an end-to-end trained network model, this model takes the initial feature matching established by the feature extraction method as input. It mines consistency information from three different types of neighbors and uses a series of carefully designed network structures to obtain features with stronger representational capabilities for neural network learning. The model utilizes a self-context extraction layer to capture relationships within neighbors and aggregate contextual information. The specific steps included in the neighbor consistency block algorithm are described below:

[0043] Spatial neighbor embedding constructed in steps 2-4 Feature space neighbor embedding and global graph neighbor embedding To mine neighbor consistency information, a Self-Context Extraction (SCE) layer was designed to process each neighbor embedding. Considering that neighbor nodes are ranked according to similarity principles in different spaces, grouped convolutions were used to dynamically extract contextual features between neighbors along the graph nodes. The k neighbors were divided into g groups, and two consecutive convolutional layers, batch normalization, and ReLU activation layers were used to process the neighbor embeddings. The extracted context C within the neighbors is shown in the following equation:

[0044]

[0045] Where Conv1() and Conv2() represent functions with... Convolutional layers with 1×g kernels are used to dynamically aggregate contextual information;

[0046] Three parallel Self-Context Extraction (SCE) layers are used to process the three neighbor embeddings separately, obtaining three corresponding neighbor context features.

[0047] Steps 2-6: Utilize the cross-context interaction layer to fuse and adjust the interaction information between neighbors. Based on the self-attention mechanism, the cross-context interaction layer (CCI layer) has three parallel cross-attention branches. In each cross-attention branch, one type of neighbor context feature is used as ValuesV, while the other two types are used as QueriesQ and KeysK. The output I of the cross-attention branch is obtained through the self-attention mechanism, as shown in the following equation:

[0048] I = α(MLPs(A) w V))+C

[0049] Where α represents a learnable weight used to control the effect of cross attention. Let represent the attention weight matrix, MLP() represent the multilayer perceptron layer, and V represent a neighbor context feature, Values ​​V. This attention weight matrix is ​​calculated by matrix multiplication and a softmax function using the transpose of QueriesQ and KeysK. It measures the relevance of the matching and the enhanced Values ​​V. Furthermore, neighbor interaction features in three neighbor spaces are obtained. As the final output of the neighbor consistency block;

[0050] Figure 3 The structure diagram of the self-context extraction layer (SCE layer) and cross-context interaction layer (CCI layer) proposed in this invention is shown in the figure.

[0051] Step 2-7: Set up two consecutive pruning modules to execute the processing steps 2-1 to 2-6 twice in succession to obtain the inlier probability of feature matching after pruning;

[0052] Based on probability values, the N initial image feature matches are pruned into N² reliable candidate image feature match sets. and the corresponding interior point weights

[0053] To improve the reliability of feature learning, some existing network structures, such as sequentially obtainable blocks and globally consistent blocks, were added to each pruning module. At the same time, progressive neighbor refinement (i.e., using neighbor consistent blocks twice) was adopted to increase neighbor reliability and extract rich neighbor context information. Finally, accurate network logical values ​​were obtained for subsequent matching and pruning processes.

[0054] Step 3: Match the candidate image features based on the set obtained in Step 2 and the corresponding interior point weights The weighted eight-point algorithm is used to predict the corresponding essential matrix. As shown in the following formula:

[0055]

[0056] Where we() represents the weighted eight-point algorithm, This indicates that the predicted essential matrix is ​​used to recover the camera's pose, i.e., the corresponding rotation and translation;

[0057] Step 4: Based on the essential matrix obtained in Step 3 The inlier probabilities w of the initial image feature matching for all inputs are obtained using a full-size validation operation, as shown in the following equation:

[0058]

[0059] Where v() represents the full-size validation operation;

[0060] Finally, estimate the essential matrix of the two matching images and output the probability of correctly matching interior points.

[0061] As shown in Table 1, the performance of the method of the present invention on a large public outdoor dataset is shown. The experimental results show that the present invention achieves the best performance among all the technologies compared.

[0062] Table 1

[0063]

[0064] This dataset consists of 100 million travel images from the internet. Matching image pairs presents significant challenges due to issues such as viewpoint changes, lighting variations, occlusion, and blurring. In this table, RANSAC, OANet++, and CLNet were selected for relative pose estimation experiments, using mean average accuracy (mAP) as the evaluation metric. Error thresholds of 5° and 20° were chosen. The performance results are shown in the comparison table, with our proposed method achieving the best performance. Practical application demonstrates that this invention can be widely used in remote sensing image stitching and 3D reconstruction.

[0065] This invention can accurately recover the camera pose and identify correct matches. For example... Figure 4 As shown, some visualizations of the output results of the method of the present invention are presented, where correct matches are represented by green lines and incorrect matches are represented by red lines. It can be seen that the method of the present invention can obtain highly reliable matching results even in challenging outdoor scenarios.

[0066] In summary, the neighbor consistency mining network designed in this invention can learn more comprehensively about initial feature matching and can extract intra-neighbor context and explore inter-neighbor interactions for three different types of neighbors. This information is crucial for distinguishing whether an image match is correct. Addressing the limitation of the previous two types of neighbors finding that always contain some unwanted outliers, and based on the strong consistency constraint of inliers at the global level, this invention proposes a new global graph space to search for consistent neighbors for each match. Furthermore, based on the three types of neighbors, this invention proposes a self-context extraction layer to dynamically capture intra-neighbor relationships and aggregate their context, and a cross-context interaction layer to fuse and adjust the interaction information between neighbors. Finally, this invention utilizes an iterative pruning strategy to obtain a more reliable candidate matching set and estimate more accurate image feature matching results and camera pose.

[0067] It should be noted that although the present invention has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art should understand that the present invention is not limited to the above embodiments, and all modifications to the present invention fall within the scope of protection of the present invention.

Claims

1. A method for image matching based on progressive neighbor consistency mining, characterized in that, The method comprises the following steps: Step 1: Feature extraction algorithm is applied to the given pair of matching images and Step 2: Feature extraction is performed on the given pair of matching images using a feature extraction algorithm, each image feature is represented by a keypoint coordinate and a corresponding descriptor, all features are constructed as an initial image feature matching set : ; wherein, denotes the to match, denotes the total number of matches, and denote the to the image pair and normalized coordinates; Step 2: Refine the initial image feature match set based on the learned inlier probabilities Prune the inliers to obtain a candidate image feature match set and the corresponding inlier weights Step 2 further comprises the following procedure: Step 2-1: obtaining an initial image feature matching set obtaining an intermediate feature mapping set through 3 residual block processing wherein represents a channel dimension, represents a total number of matches, obtaining three types of neighbor spaces; wherein, the coordinate space is an initial feature matching , the feature space is an intermediate feature mapping , the global graph space is used to reflect the consistency of inliers at a global level; Step 2-2: Calculate the global graph space, the process is as follows: calculating preliminary inlier weights according to the intermediate feature map F wherein denotes a multi-layer perceptron layer, and denotes an activation function ; constructing a weighted global graph wherein, denotes all image feature match nodes, denotes undirected edges, with corresponding weights implementing each two image feature match connection, wherein , denote the first and j-th image feature match preliminary inlier weight, respectively, denotes the weight of its connecting edge, further resulting in an adjacency matrix ; obtaining a global map space as shown in the following equation: ; wherein, denotes a turd Laplacian matrix, F denotes an intermediate feature map, denotes an adjacency matrix with self-connections, denotes a diagonal identity matrix, denotes an adjacency matrix, denotes a diagonal degree matrix, denotes a trainable weight; Step 2-3: Based on the three neighbor spaces obtained in step 2-1, use the k nearest neighbor search strategy to find the space k neighbors, feature space k neighbors and global graph k neighbors for each image feature matching; Step 2-4: Constructing spatial, feature, and global graph neighbor embeddings for the three different types of neighbors obtained in Step 2-3 ;​​ Step 2-5: Capture the relationship within the neighbors and aggregate the context information using the self-context extraction layer, and the specific steps are described as follows: Spatial neighbor embeddings constructed based on steps 2-4 Feature space neighbor embeddings And global graph neighbor embeddings Neighbor consistency information mining is performed, each neighbor embedding is processed by using a self-context extraction layer, and context features between neighbors are dynamically extracted along the graph nodes by using a grouping convolution manner; k neighbors are divided into g groups, and two continuous convolution layers, batch normalization and an activation layer ReLU are used to process the neighbor embedding to obtain the context within the neighbors As shown in the following formula: ; wherein, and denote convolutional layers with and nuclear de-dynamically aggregate contextual information; Three parallel self-contextual extraction layers are utilized to process the three kinds of neighbor embeddings separately to obtain three corresponding neighbor context features ; Step 2-6: Fuse and adjust the interaction information between neighbors by cross- contextual interaction layer, i.e. in each cross-attention branch, one neighbor context feature is taken as Values , and the other two neighbor context features are taken as Queries and Keys ; the output of the cross-attention branch is obtained by self-attention mechanism operation , as shown in the following formula: ; wherein, represents a learnable weight, represents an attention weight matrix, represents a multi-layer perceptron layer, represents a kind of neighbor context feature Values ; Further, three neighbor space neighbor interaction features are obtained as an output of the neighbor consistency block; Steps 2-7: Set up two consecutive pruning modules to execute the processing steps 2-1 to 2-6 twice consecutively to obtain the inlier probabilities of the pruned image feature matching; based on the obtained inlier probability values, Initial image feature matching pruning Candidate image feature matching set and the corresponding interior point weights ; Step 3: Matching set of candidate image features obtained from step 2 and corresponding inlier weights , predict a corresponding essential matrix as shown in the following equation: ; wherein, denotes a weighted eight-point algorithm, denotes an essential matrix according to the prediction; Step 4: The essential matrix obtained from step 3 The probability of inliers for all inputted initial image feature matches is obtained As shown in the following formula: ; wherein, represents a full-size verification operation; Finally, the essential matrix of the two matched images is estimated, and the probability of outputting the correct matching inner point is outputted.

2. The image matching method based on progressive neighbor consistency mining according to claim 1, further comprising adding sequential obtainable blocks and global consistency blocks for each pruning module, while adopting progressive neighbor refinement processing, increasing neighbor reliability and extracting rich neighbor context information, to obtain accurate network logic values for subsequent matching pruning process.

Images