A feature matching method based on motion filtering and adjustment

Abstract

The application relates to a feature matching method based on motion filtering and adjustment, and belongs to the technical field of computer vision. A motion filtering and adjustment network is proposed to reduce the influence of outliers when capturing context. Specifically, first, a motion filtering block is designed to remove outliers and iteratively capture context. Then, a regularization and adjustment block is designed to estimate and adjust the accurate motion field with additional position information. The method can effectively capture context and estimate accurate motion field; the method can be applied to pose estimation, point cloud registration, image stitching and motion segmentation and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a feature matching method based on motion filtering and adjustment. Background Technology

[0002] Feature matching aims to establish reliable point-to-point correspondences from image pairs of the same or similar scenes. It is fundamental to many tasks in computer vision, such as pose estimation, point cloud registration, image stitching, feature matching, and motion segmentation. The quality of the matching has a significant impact on the performance of these downstream tasks.

[0003] Given a pair of images from the same or similar scenes, a typical feature matching process includes feature extraction, feature matching, and outlier removal. Feature extraction is to extract key points and descriptors from the image using feature extractors (such as Scale Invariant Feature Transform (SIFT) [1], SuperPoint [2], etc.); feature matching is to establish an initial matching set based on the similarity of descriptors; outlier removal aims to identify and remove incorrect matches (i.e., outliers) and retain correct matches (i.e., inliers) as much as possible. Due to the complexity and variability of real-world scenes, the quality of the initial matching set is seriously affected, and there are often a large number of outliers inside, which poses a challenge to outlier removal. Therefore, many feature matching models based on deep learning are committed to exploring effective outlier removal methods to cope with complex and variable real-world scenes. LFGC [3] is the first method to use deep learning technology for outlier removal. It designs a deep network based on a multilayer perceptron to identify inliers and outliers and trains it in an end-to-end manner. This method uses context normalization to obtain context information to guide the classification of matches. However, by calculating the mean and variance of the feature map to capture the global context, the diversity and potential relationships between different matches are ignored. Many subsequent studies have been dedicated to addressing this problem. For example, OANet[4] designed differentiable pooling and depooling layers to capture local context, and a sequence-aware filtering module to capture global context. NM-Net[5] searches for reliable local context neighbors based on the compatibility score of the match. Although these subsequent studies have made impressive improvements, they have neglected the potential motion information of the image. Therefore, in order to utilize motion information, ConvMatch[6] computes a motion vector for each match and estimates a noisy motion field, and uses a convolutional neural network to extract the context and filter out the noise. Experiments on multiple indoor and outdoor scenes have demonstrated the effectiveness of this method. However, convolutional neural networks tend to estimate a globally smooth motion field, which is difficult to handle in cases where the motion field is segmented and smooth in complex scenes. In addition, the fixed-size receptive field of convolutional neural networks limits their ability to capture global context, and convolutional operations lose complex motion information, which is not conducive to further improvement of model performance.

[0004] References:

[0005] [1] D.G. Lowe, “Distinctive image features from scale - invariant keypoints,” International Journal of Computer Vision, vol. 60, pp. 91–110, 2004.

[0006] [2] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self - supervised interest point detection and description,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224–236, 2018.

[0007] [3] K.M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua, “Learning to find good correspondences,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2666–2674, 2018.

[0008] [4] J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, Y. Chen, L. Quan, and H. Liao, “Learning two - view correspondences and geometry using order - aware network,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 5845–5854, 2019.

[0009] [5] C. Zhao, Z. Cao, C. Li,

[0010] [6] S. Zhang and J. Ma, “Convmatch: Rethinking network design for two viewcorrespondence learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 2023, pp. 3472–3479. Summary of the Invention

[0011] The purpose of this invention is to alleviate the problems of existing technologies and provide a feature matching method based on motion filtering and adjustment, which can effectively capture context and estimate the motion field accurately.

[0012] To achieve the above objectives, the technical solution of the present invention is: a feature matching method based on motion filtering and adjustment, comprising:

[0013] Design a motion filter block to remove outliers and iteratively capture context;

[0014] Design a regularization and adjustment block to estimate and adjust the precise motion field with additional location information.

[0015] In one embodiment of the present invention, the method includes the following steps:

[0016] A. Given a pair of images consisting of two images, extract key points and corresponding descriptors from the two images in the pair, and construct an initial matching set containing mismatches based on the descriptor similarity.

[0017] B. Calculate the initial unordered motion vector set from the initial matching set and map it to a high-dimensional space through a multilayer perceptron;

[0018] C. Utilize a motion filtering module containing multiple attention pooling layers to progressively remove erroneous motion vectors and effectively capture contextual information;

[0019] D. Use grid embedding and graph attention networks to transform disordered motion vectors into ordered vectors to represent the motion field;

[0020] E. Use finer mesh embeddings to adjust the ordered motion vectors in the motion field to better represent local motion;

[0021] F. Use a graph attention network to correct the initial unordered set of motion vectors based on the motion field. Based on the difference between the unordered motion vectors before and after correction, use a multilayer perceptron to predict the correct / incorrect matching classification result.

[0022] G. Based on the classification results, estimate the essential matrix using the weighted eight-point algorithm.

[0023] In one embodiment of the present invention, steps C to G are iterated 6 times. In each iteration, the cross-entropy loss is calculated using the predicted classification results and the true category results, and the regression loss is calculated using the predicted essential matrix and the true essential matrix to guide network training, ultimately obtaining a high-performance feature matching model.

[0024] In one embodiment of the present invention, in step A, key points and corresponding descriptors are extracted from the two images of the image pair using scale-invariant feature transformation.

[0025] In one embodiment of the present invention, in step B, an initial set of unordered motion vectors M is calculated from the initial matching set and mapped to a C-dimensional space through a multilayer perceptron to obtain a C-dimensional vector set F.

[0026] In one embodiment of the present invention, step C is specifically implemented as follows:

[0027] Let F be the input to the p-th attention pooling layer. p The initial input to the attention pooling layer is F0 = F, based on a downsampling rate r. p Remove F p Potential erroneous motion vectors in the middle:

[0028] s p =sigmoid(Γ(PointCN(F)) p ))

[0029]

[0030] Where PointCN(·) is the context normalization module, Γ is a fully connected layer, sigmoid is the activation function, and s p This represents the set of confidence scores for motion vectors. This represents the subset of motion vectors filtered by the top-k function;

[0031] Context aggregation is used to fuse contextual information contained in the input set into a subset:

[0032]

[0033] Where Cgg represents the context aggregation operation; the resulting subset F p+1 It will be used as the input for the next attention pooling layer.

[0034] In one embodiment of the present invention, in step D, a grid of shape K×K×C is used to embed X. grid By regularizing the unordered motion vectors into ordered vectors, we obtain an ordered vector set F to represent the motion field. grid .

[0035] In one embodiment of the present invention, in step E, a finer mesh embedding is used to adjust the ordered motion vectors in the motion field to better represent local motion, specifically as follows:

[0036]

[0037] in, It is a shape of K ' ×K ' Mesh embedding, K ' >K, G represent graph attention networks.

[0038] The present invention also provides a feature matching system based on motion filtering and adjustment, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0039] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0040] Compared to existing technologies, this invention offers the following advantages: The method proposes a motion filtering module and an adjustment module. Before estimating the motion field, the motion filtering module uses multiple attention pooling layers to capture the context while removing most erroneous motion vectors from the disordered set of motion vectors, thus estimating a motion field with less noise. The adjustment module utilizes additional mesh embeddings on the estimated motion field to fuse more refined positional information, enabling the motion field to more accurately represent local motion at different locations, addressing the problem of smooth segmentation of motion fields in complex scenes. Using these modules, the model can obtain a higher-quality motion field, thereby more accurately classifying matches and estimating the essential matrix. This invention effectively captures the context and estimates an accurate motion field; it can be applied to multiple fields such as pose estimation, point cloud registration, image stitching, and motion segmentation. Attached Figure Description

[0041] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] This invention provides a feature matching method based on motion filtering and adjustment, comprising:

[0046] Design a motion filter block to remove outliers and iteratively capture context;

[0047] Design a regularization and adjustment block to estimate and adjust the precise motion field with additional location information.

[0048] The following is a detailed implementation process of the present invention.

[0049] like Figure 1 As shown, the present invention provides a feature matching method based on motion filtering and adjustment, which specifically includes the following steps:

[0050] A. Given a pair of images, use SIFT (Scale Invariant Feature Transform) to extract key points and corresponding descriptors from the two images of the image pair, and construct an initial matching set containing mismatches based on the descriptor similarity.

[0051] B. Calculate the initial unordered motion vector set M from the initial matching set, and map it to the C-dimensional space through a multilayer perceptron (Embedding) to obtain the C-dimensional vector set F.

[0052] C. Design a motion filtering module containing multiple attention pooling layers to progressively remove erroneous motion vectors and effectively capture contextual information. Let F be the input of the p-th attention pooling layer. p (F0=F), firstly based on a downsampling rate r p Remove F p Potentially erroneous motion vectors.

[0053] s p =sigmoid(Γ(PointCN(F)) p ))

[0054]

[0055] Where PointCN(·) is the context normalization module, Γ is a fully connected layer, and s p This represents the set of confidence scores for motion vectors. This represents the subset of motion vectors after being filtered by the top-k function.

[0056] Then, context aggregation is used to fuse the context information contained in the input set into the subset.

[0057]

[0058] Where Cgg represents the context aggregation operation, resulting in the subset F p+1 It will be used as the input for the next attention pooling layer.

[0059] D. After filtering and capturing the context using multiple attention pooling layers, use a grid embedding of shape K×K×C. grid By regularization, disordered motion vectors are transformed into ordered vectors, resulting in an ordered vector set F used to represent the motion field. grid .

[0060] E. Use finer mesh embeddings to adjust the ordered motion vectors in the motion field to better represent local motion.

[0061]

[0062] in It is a shape of K ' ×K ' Mesh embedding (K ' >K), G represents a graph attention network.

[0063] F. Use a graph attention network to correct (recover) the initial unordered set of motion vectors based on the motion field, to obtain the corrected unordered set of motion vectors. Based on the difference between the disordered motion vectors before and after correction, the classification results of correct / incorrect matches are obtained through a multilayer perceptron-based prediction process.

[0064] G. Based on the classification results, estimate the essential matrix using the weighted eight-point algorithm.

[0065] Steps C through G will be executed iteratively 6 times, each time resulting in a set of unordered motion vectors. All of these are used as input for the next iteration. In addition, each iteration uses the predicted classification results and the true category results to calculate the cross-entropy loss, and uses the predicted essence matrix and the true essence matrix to calculate the regression loss to guide the network training, ultimately obtaining a high-performance feature matching model.

[0066] The present invention also provides a feature matching system based on motion filtering and adjustment, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0067] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method of feature matching based on motion filtering and adjustment, characterized in that, The method comprises the following steps: A motion filtering block is designed to remove outliers and iteratively capture context; A regularization and adjustment block is designed to estimate and adjust the accurate motion field with additional position information; The method comprises the following steps: A, given a set of image pairs composed of two images, extract key points and corresponding descriptors from the two images of the image pair, and construct an initial matching set containing false matches according to the similarity of the descriptors; B, calculate the initial unordered motion vector set from the initial matching set, and map it to a high-dimensional space through a multi-layer perceptron; C, use a motion filtering module containing multiple attention pooling layers to gradually remove false motion vectors and effectively capture context information; D, use grid embedding and graph attention network to convert unordered motion vectors into ordered vectors to represent the motion field; E, adjust the ordered motion vectors in the motion field using a finer grid embedding to better represent local motion; F, use a graph attention network to correct the initial unordered motion vector set according to the motion field, and use a multi-layer perceptron to predict the correct / incorrect matching classification result according to the difference between the corrected and uncorrected unordered motion vectors; G, according to the classification result, estimate the essential matrix using the weighted eight-point algorithm; In step B, the initial unordered motion vector set M is calculated from the initial matching set, and is mapped to a C-dimensional space through a multi-layer perceptron to obtain a C-dimensional vector set F. Step C is implemented as follows: Let the input of the p-th attention pooling layer be F p , the input of the initial attention pooling layer F0=F, and according to one down-sampling rate r p , remove the potential error motion vector in F p : s p = sigmoid(Γ(PointCN(F p ))) where PointCN(·) is a context normalization module, Γ is a fully connected layer, sigmoid is an activation function, s p denotes a set of motion vector confidences, denotes a subset of motion vectors after top-k function screening; Context aggregation is used to fuse the context information contained in the input set into the subset: where Cgg denotes a contextual aggregation operation; the resulting subset F p+1 will be the input to the next attention pooling layer.

2. The method of claim 1, wherein, Steps C to G are iterated 6 times, and in each iteration, the cross-entropy loss is calculated using the predicted classification result and the true class result, and the regression loss is calculated using the predicted essential matrix and the true essential matrix to guide network training, and finally a high-performance feature matching model is obtained.

3. The method of claim 1, wherein, In step A, scale-invariant feature transform is used to extract key points and corresponding descriptors from the two images of the image pair.

4. The method of claim 1, wherein, In step D, X is embedded using a grid of shape K x K x C grid By regularizing the unordered motion vectors into ordered vectors, a set of ordered vectors F is obtained for representing the motion field grid .

5. The method of claim 4, wherein, In step E, a finer grid embedding is used to adjust the ordered motion vectors in the motion field to better represent local motion, which is specifically represented as follows: wherein, is a grid embedding of shape K' x K', K' > K, G denotes a graph attention network.

6. A system for feature matching based on motion filtering and adjustment, comprising: The computer program instructions stored on the memory and capable of being executed by the processor when the processor executes the computer program instructions can implement the method steps of any one of claims 1-5.

7. A computer readable storage medium having stored thereon computer program instructions capable of being executed by a processor, which when the processor executes the computer program instructions, can implement the method steps of any one of claims 1-5.

Images