A method for image feature matching based on global-local consistency network

Abstract

The present invention relates to an image feature matching method based on a global-local consistency network, the method comprising obtaining a set of assumed matching points between a reference image and a perceived image; calculating the vector length and angle between the assumed matching point sets, and generating a three-dimensional motion vector distribution map; using a global transformation consistency module to roughly classify the motion vector distribution map to obtain a pure inlier set; using the inlier set as the neighborhood of the assumed matching point set to construct a neighborhood map for each feature point; using a local neighborhood consistency module to perform similarity evaluation on the neighborhood map to obtain an optimized correspondence relationship. The present invention can process a variety of transformation modes (rigid and non-rigid) without relying on any predefined transformation model, effectively filter out erroneous matches, and fully retain the correct correspondence relationship.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, in particular to an image feature matching method based on a global-local consistency network. Background Art

[0002] Feature matching is one of the basic tasks in remote sensing and photogrammetry, which focuses on identifying the dependent correspondence between two scenes with overlapping areas. As an important image matching tool, feature matching plays a vital role in image registration and fusion, 3D reconstruction, content-based image retrieval, panoramic image stitching, target recognition and tracking, and simultaneous localization and mapping (SLAM).

[0003] The matching problem is usually solved in a two-step manner, i.e., first constructing a set of hypothetical matches and then removing false matches from it. Often, the set of hypothetical matches is formed by simply picking out point pairs with sufficiently similar feature descriptors (e.g., Scale-Invariant Feature Transform, SIFT). However, due to the ambiguity of local descriptors (especially when the image is of low quality, occluded, and repeated patterns), the set of hypothetical points includes some false matches (redundant points) in addition to most of the true matches (inliers). Therefore, it is crucial to design a robust method to eliminate redundant points to improve the reliability of matching.

[0004] Although the problem of false match removal has made great progress in the past few years, an accurate, robust and efficient algorithm is still needed in practical applications. The challenges mainly lie in the following three aspects. First, the geometric transformation between an image pair is usually unknown in advance, so there is an urgent need for a general algorithm that can handle various transformations. Second, the feature matching problem in actual computer vision tasks is usually affected by factors such as object deformation, low-quality imaging, and repeated structures, resulting in a large number of false matches, which increases the difficulty of establishing accurate correspondences. Third, image registration in practical applications is usually not a simple parameterized transformation model, but a complex non-rigid model, which can easily cause excessive computational burden and poor matching effect.

[0005] In summary, how to design a general algorithm that can handle various transformations (rigid and non-rigid) with high efficiency and can be applied in real-time scenarios under the premise of a large number of mismatches is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of the shortcomings of existing image feature matching methods, the present invention proposes an image feature matching method based on a global-local consistency network. The algorithm can remove false matches from the assumed feature correspondence set and does not rely on any predefined transformation model.

[0007] The technical solution of the present invention provides an image feature matching method based on a global-local consistency network, comprising the following steps:

[0008] Step 1, using feature point extraction algorithm to obtain a set of assumed matching points between images;

[0009] Step 2, for the assumed matching point set generated by SIFT, calculate the vector length and angle between the point sets, and generate a three-dimensional motion vector distribution map;

[0010] Step 3: Use the global transformation consistency module to roughly classify the motion vector distribution map in step 2, not only to remove redundant points, but also to remove the internal points that are attached to the redundant points, so as to obtain a pure internal point set;

[0011] Step 4, using the internal point set obtained in step 3 as the neighborhood of the assumed matching point set, construct a neighborhood graph for each feature point;

[0012] Step 5, using the local neighborhood consistency module to evaluate the similarity of the neighborhood graph in step 4;

[0013] Step 6, using the similarity value distribution of the feature points obtained in step 5, obtain the optimized corresponding relationship.

[0014] Moreover, in step 1, considering an image pair, we can use the existing feature detection and description algorithms SIFT and nearest neighbor matching NNDR to obtain a set of hypothesized matching points.

[0015] Moreover, in step 2, generating a three-dimensional motion vector distribution map includes the following steps: for n pairs of assumed corresponding point sets in Respectively represent the coordinates of two corresponding feature points, we let represents the angle of the motion vector between the point pairs, where is the range of the point set S, max(·) and min(·) represent the maximum and minimum values ​​in the set respectively, and Represents the length of the motion vector between the point pairs. Subsequently, we can transform the set of assumed matching points into a three-dimensional motion vector distribution map D = {(θ, l, p)} with noise, where p = {1, 2, ..., n}.

[0016] Moreover, in step 3, the motion vector distribution map in step 2 is roughly classified using the global transformation consistency module to obtain a pure set of internal points. i , use KNN to search for the k nearest neighbors of each point Get a directed graph Represents the local topological structure of each point. Next, we use a shared multi-layer perceptron (MLP) to encode the graph feature f, and aggregate (select the maximum value) the graph feature of each vertex to obtain

[0017]

[0018] in, is the weight of the filter, obtained by MLP training, M represents the number of filters, m ranges from 1 to M; ReLU is the activation function. The entire MLP has 5 convolutional layers. We recalculate the nearest neighbor in the feature space output by each layer to update the directed graph g (g is g i The receptive field is as large as the diameter of the three-dimensional motion vector distribution map D.

[0019] Moreover, in step 4, the internal point set obtained in step 3 is used as the neighborhood of the assumed matching point set to construct a neighborhood graph for each feature point. The specific operation is as follows: for the assumed matching point set Z, we first normalize the spatial coordinates of Z to obtain the normalized point set Z':

[0020]

[0021] Then, the inner point set obtained in step 3 is used to find k nearest neighbors for Z', and the neighborhood graph of the final feature point is expressed as:

[0022]

[0023] Among them, t is the neighboring point of Z'.

[0024] Moreover, in step 5, the local neighborhood consistency module is used to evaluate the similarity of the neighborhood graph in step 4. The local neighborhood consistency module includes an encoder module to encode the neighborhood graph to obtain the neighborhood representation of the feature point:

[0025] Γ i =Encoder(G i )

[0026] Among them, i is the serial number of the feature point. Then, the self-attention and cross-attention in the attention mechanism are used to evaluate the similarity of the neighborhood representation:

[0027]

[0028] Among them, in the case of self-attention mechanism, Q i ,K i ,V i The values ​​of In the case of the cross-attention mechanism, Q i for K i ,V i The value of The self-attention mechanism and the cross-attention mechanism are iterated 4 times respectively. Represents the scale factor. Finally, the Euclidean distance between the two feature points is calculated:

[0029]

[0030] Finally, the minimum distance among the k edges is selected as the similarity value:

[0031]

[0032] Furthermore, the specific implementation of step 6 is: using the method of steps 1-5 to evaluate the similarity of the assumed matching point set, and then eliminating the point set with low similarity according to the distribution of similarity values ​​to obtain the final correct matching point set.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The present invention can effectively filter out false matches and fully preserve the correct correspondences, even in extreme cases of image distortion and non-rigid transformation.

[0035] (2) The present invention can handle a variety of transformation modes (rigid and non-rigid), does not rely on any predefined transformation model, and has much higher accuracy than existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of an embodiment of the present invention;

[0037] Figure 2 is a flow chart of the global transformation consistency module of the present invention;

[0038] Figure 3 It is a flow chart of the local neighborhood consistency module of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further explained below with reference to examples, and the specific steps of the image feature matching method based on the global-local consistency network are described in detail with reference to the accompanying drawings.

[0040] Step 1: Use feature point extraction algorithm to obtain a set of assumed matching points between images.

[0041] Considering an image pair, a reference image and a perceived image, we can use existing feature detection and description algorithms SIFT and nearest neighbor matching NNDR to obtain a set of hypothesized matching points.

[0042] Step 2: For the assumed matching point set generated by SIFT, the vector length and angle between the point sets are calculated, and a three-dimensional motion vector distribution map is generated.

[0043] We use the global transformation continuity and consistency between images to obtain a 3D motion vector distribution map. Generating a 3D motion vector distribution map includes the following steps: in Respectively represent the coordinates of two corresponding feature points, we let represents the angle of the motion vector between the point pairs, where is the range of the point set S, max(·) and min(·) represent the maximum and minimum values ​​in the set respectively, and Represents the length of the motion vector between the point pairs. Subsequently, we can transform the set of assumed matching points into a three-dimensional motion vector distribution map D = {(θ, l, p)} with noise, where p = {1, 2, ..., n}.

[0044] Step 3: Use the global transformation consistency module to roughly classify the motion vector distribution map in step 2, not only to remove redundant points, but also to remove the internal points that are attached to the redundant points, so as to obtain a pure set of internal points.

[0045] like Figure 2 As shown in FIG. 1 , the motion vector distribution map in step 2 is roughly classified using the global transformation consistency module to obtain a pure set of internal points. Specifically, for each point D in the motion vector map i , use KNN to search for the k nearest neighbors of each point Get a directed graph Represents the local topological structure of each point. Next, we use a shared multi-layer perceptron (MLP) to encode the graph feature f, and aggregate (select the maximum value) the graph feature of each vertex to obtain

[0046]

[0047] in, is the weight of M different filters, obtained by MLP training, m ranges from 1 to M; ReLU is the activation function. The entire MLP has 5 convolutional layers. We recalculate the nearest neighbor in the feature space output by each layer to update the directed graph g, so that the receptive field is as large as the diameter of the 3D motion vector distribution map D.

[0048] Step 4: Use the internal point set obtained in step 3 as the neighborhood of the assumed matching point set to construct a neighborhood graph for each feature point.

[0049] Use the internal point set obtained in step 3 as the neighborhood of the assumed matching point set to construct a neighborhood graph for each feature point. The specific operation is as follows: For the assumed matching point set Z, we first normalize the spatial coordinates of Z to obtain the normalized point set Z':

[0050]

[0051] Then, the inner point set obtained in step 3 is used to find k nearest neighbors for Z', and the neighborhood graph of the final feature point is expressed as:

[0052]

[0053] Among them, t is the neighboring point of Z'.

[0054] Step 5: Use the local neighborhood consistency module to evaluate the similarity of the neighborhood graph in step 4.

[0055] like Figure 3 As shown, the local neighborhood consistency module is used to evaluate the similarity of the neighborhood graph in step 4. The local neighborhood consistency module includes an encoder module to encode the neighborhood graph to obtain the neighborhood representation of the feature point:

[0056] Γ i =Encoder(G i )

[0057] Among them, i is the serial number of the feature point. Then, the self-attention and cross-attention in the attention mechanism are used to evaluate the similarity of the neighborhood representation:

[0058]

[0059] Among them, in the case of self-attention mechanism, Q i ,K i ,V i The values ​​of In the case of the cross-attention mechanism, Q i for K i ,V i The value of The self-attention mechanism and the cross-attention mechanism are iterated 4 times respectively. Represents the scale factor. Finally, the Euclidean distance between the two feature points is calculated:

[0060]

[0061] Finally, the minimum distance among the k edges is selected as the similarity value:

[0062]

[0063] Step 6, using the similarity value distribution of the feature points obtained in step 5, obtain the optimized corresponding relationship.

[0064] The similarity of the assumed matching point set is evaluated using the method of steps 1-5, and then the point set with low similarity is eliminated according to the distribution of similarity values ​​to obtain the final correct matching point set.

[0065] The above is only a preferred embodiment of the present invention, but the present invention is not limited to the described embodiments. For those skilled in the art, changes, modifications, substitutions and variations of the embodiments without departing from the principles and spirit of the present invention should all fall within the scope of protection of the present invention.

Claims

1. An image feature matching method based on a global-local consistency network, characterized in that: The following steps are involved: Step 1, using feature point extraction algorithm to obtain a set of assumed matching points between images; Step 2, for the generated assumed matching point set, calculate the vector length and angle between the point sets, and generate a three-dimensional motion vector distribution map D; Step 3: Based on the principle of global consistency, the motion vector distribution map in step 2 is roughly classified, and not only redundant points but also the internal points that are attached to the redundant points are eliminated to obtain a pure internal point set. Step 4, using the internal point set obtained in step 3 as the neighborhood of the assumed matching point set, construct a neighborhood graph G for each feature point; Step 5, evaluating the similarity of feature points of the neighborhood graph in step 4 based on the principle of local neighborhood consistency; In step 5, the neighborhood graph G in step 4 is evaluated for similarity using the local neighborhood consistency module. The local neighborhood consistency module includes an encoder module to encode the neighborhood graph to obtain the neighborhood representation of the feature point: G i =Encoder(G i ) Among them, i is the serial number of the feature point. Then, the self-attention and cross-attention in the attention mechanism are used to evaluate the similarity of the neighborhood representation: Among them, in the case of self-attention mechanism, Q i ,K i ,V i The values ​​of In the case of the cross-attention mechanism, Q i for K i ,V i The value of Represents the scale factor; the self-attention mechanism and the cross-attention mechanism are iterated L times respectively to calculate the Euclidean distance represented by the two feature points: Finally, the minimum distance among the k edges is selected as the similarity value: Step 6, using the similarity value distribution of the feature points obtained in step 5, obtain the optimized corresponding relationship.

2. The image feature matching method based on global-local consistency network according to claim 1, characterized in that: In step 1, consider an image pair, a reference image and a perceived image, and use existing feature detection and description algorithms SIFT and nearest neighbor matching NNDR to obtain a set of hypothesized matching points.

3. The image feature matching method based on global-local consistency network according to claim 1, characterized in that: In step 2, the global transformation continuity and consistency between images are used to obtain a three-dimensional motion vector distribution map.

4. The image feature matching method based on global-local consistency network according to claim 3 is characterized in that: Generate a 3D motion vector distribution map in step 2 The following steps are included: For n pairs of assumed matching point sets in Respectively represent the coordinates of two corresponding feature points, let represents the angle of the motion vector between the point pairs, where is the range of the point set S, max(·) and min(·) represent the maximum and minimum values ​​in the set respectively, and Represents the length of the motion vector between the point pairs; then, the assumed matching point set is converted into a three-dimensional motion vector distribution map D = {(θ, l, p)} with noise, where p = {1, 2, ..., n}.

5. The image feature matching method based on global-local consistency network according to claim 1, characterized in that: In step 3, the motion vector distribution map in step 2 is roughly classified using the global transformation consistency module to obtain a pure set of internal points. The specific details of the global transformation consistency module are as follows: For each point D in the three-dimensional motion vector distribution map D obtained in step 2 i , use KNN to search for the k nearest neighbors of each point Get a directed graph represents the local topological structure of each point; next, a shared multi-layer perceptron MLP is used to encode the graph feature f, and the graph features of each vertex are aggregated to obtain in, is the weight of the filter, obtained by MLP training, M represents the number of filters, m ranges from 1 to M; ReLU is the activation function, the entire MLP has 5 convolutional layers, and the nearest neighbor is recalculated in the feature space output by each layer to update the directed graph g, g is g i A set of , so that the receptive field is as large as the diameter of the motion vector distribution map D.

6. The image feature matching method based on global-local consistency network according to claim 1, characterized in that: The specific implementation method in step 4 is as follows: For the assumed matching point set Z, the spatial coordinates of Z are first normalized to obtain the normalized point set Z': Then, the inner point set obtained in step 3 is used to find k nearest neighbors for Z', and the neighborhood graph of the final feature point is expressed as: Among them, t is the neighboring point of Z'.

7. The image feature matching method based on global-local consistency network according to claim 1, characterized in that: The specific implementation method of step 6 is: use steps 1-5 to evaluate the similarity of the assumed matching point set, and then eliminate the point set with low similarity according to the distribution of similarity values ​​to obtain the final correct matching point set.

Images