Feature matching method based on motion filtering and adjustment

Abstract

The invention relates to a feature matching method based on motion filtering and adjustment, and belongs to the technical field of computer vision. A motion filtering and adjustment network is presented to mitigate the effects of outliers when capturing the context. Specifically, first, a motion filter block is designed to remove outliers and iteratively capture contexts. A regularization and adjustment block is then designed to estimate and adjust the precise motion field with additional position information. According to the method, the context can be effectively captured, and the accurate motion field can be estimated; the method can be applied to multiple fields such as attitude estimation, point cloud registration, image splicing and motion segmentation.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a feature matching method based on motion filtering and adjustment. Background Art

[0002] Feature matching aims to establish reliable point-to-point correspondences from pairs of images of the same or similar scenes, which is the basis of many tasks in computer vision, such as pose estimation, point cloud registration, image stitching, feature matching, and motion segmentation. The quality of matching has a great impact on the performance of these downstream tasks.

[0003] Given a pair of images from the same or similar scenes, a typical feature matching process includes feature extraction, feature matching, and outlier removal. Feature extraction is to extract key points and descriptors from images using feature extractors (such as scale-invariant feature transform (SIFT) [1], SuperPoint [2], etc.); feature matching is to establish an initial matching set based on the similarity of descriptors; outlier removal aims to identify and remove incorrect matches (i.e., outliers) and retain correct matches (i.e., inliers) as much as possible. Since real scenes are complex and changeable, the quality of the initial matching set is seriously affected. There are often a large number of outliers inside them, which poses a challenge to outlier removal. Therefore, many deep learning-based feature matching models are committed to exploring effective outlier removal methods to cope with complex and changeable real scenes. LFGC [3] is the first method to use deep learning technology for outlier removal. It designs a deep network based on multi-layer perceptron to identify inliers and outliers and trains them in an end-to-end manner. This method applies context normalization to obtain context information to guide the classification of matches. However, the global context is captured by calculating the mean and variance of the feature map, ignoring the diversity and potential relationships between different matches. Many subsequent studies have been devoted to addressing this problem. For example, OANet[4] designed differentiable pooling and unpooling layers to capture local context, and a sequence-aware filtering module to capture global context. NM-Net[5] searches for reliable local context neighbors based on the compatibility scores of the matches. Although these subsequent studies have achieved impressive improvements, they ignore the potential motion information of the image pairs. Therefore, in order to utilize motion information, ConvMatch[6] calculates motion vectors for each match and estimates a noisy motion field, and uses a convolutional neural network to extract context and filter out noise. Experiments on multiple indoor and outdoor scenes demonstrate the effectiveness of this approach. However, convolutional neural networks tend to estimate a globally smooth motion field, which makes it difficult to cope with the situation where the motion field is piecewise smooth in complex scenes. In addition, the fixed-size receptive field of convolutional neural networks limits their ability to capture global context, and the convolution operation loses complex motion information, which is not conducive to further improving the model performance.

[0004] References:

[0005] [1] D.G. Lowe, “Distinctive image features from scale - invariant keypoints,” International Journal of Computer Vision, vol. 60, pp. 91–110, 2004.

[0006] [2] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self - supervised interest point detection and description,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224–236, 2018.

[0007] [3] K.M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua, “Learning to find good correspondences,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2666–2674, 2018.

[0008] [4] J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, Y. Chen, L. Quan, and H. Liao, “Learning two - view correspondences and geometry using order - aware network,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 5845–5854, 2019.

[0009] [5] C. Zhao, Z. Cao, C. Li,

[0010] [6] S. Zhang and J. Ma, “Convmatch: Rethinking network design for two viewcorrespondence learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 2023, pp. 3472–3479. Summary of the invention

[0011] The purpose of the present invention is to alleviate the problems existing in the prior art and provide a feature matching method based on motion filtering and adjustment, which can effectively capture context and estimate accurate motion field.

[0012] To achieve the above object, the technical solution of the present invention is: a feature matching method based on motion filtering and adjustment, comprising:

[0013] Design a motion filtering block to remove outliers and iteratively capture context;

[0014] A regularization and adjustment block is designed to estimate and adjust the precise motion field with additional position information.

[0015] In one embodiment of the present invention, the method comprises the following steps:

[0016] A. Given a set of image pairs consisting of two images, extract key points and corresponding descriptors from the two images of the image pair, and construct an initial matching set containing false matches based on the descriptor similarity;

[0017] B. Calculate the initial unordered motion vector set from the initial matching set and map it to a high-dimensional space through a multi-layer perceptron;

[0018] C. Use a motion filtering module consisting of multiple attention pooling layers to gradually remove erroneous motion vectors and effectively capture contextual information;

[0019] D. Use grid embedding and graph attention network to transform the unordered motion vectors into ordered vectors to represent the motion field;

[0020] E. Use a finer grid embedding to adjust the ordered motion vectors in the motion field to better represent local motion;

[0021] F. Use the graph attention network to correct the initial set of unordered motion vectors according to the motion field. Use a multi-layer perceptron to predict the correct / wrong match classification results based on the difference between the unordered motion vectors before and after correction;

[0022] G. Based on the classification results, the essential matrix is ​​estimated using the weighted eight-point algorithm.

[0023] In one embodiment of the present invention, steps C to G are iterated 6 times, and each iteration uses the predicted classification results and the actual category results to calculate the cross entropy loss, and uses the predicted essential matrix and the actual essential matrix to calculate the regression loss to guide network training, and finally obtain a high-performance feature matching model.

[0024] In one embodiment of the present invention, in step A, key points and corresponding descriptors are extracted from two images of the image pair using scale-invariant feature transformation.

[0025] In one embodiment of the present invention, in step B, an initial unordered motion vector set M is calculated from the initial matching set, and mapped to a C-dimensional space through a multi-layer perceptron to obtain a C-dimensional vector set F.

[0026] In one embodiment of the present invention, step C is specifically implemented as follows:

[0027] Let the input of the p-th attention pooling layer be F p , the input of the initial attention pooling layer F0 = F, according to a downsampling rate r p , remove F p Potentially erroneous motion vectors in:

[0028] s p =sigmoid(Γ(PointCN(F p ))

[0029]

[0030] Among them, PointCN(·) is the context normalization module, Γ is a fully connected layer, sigmoid is the activation function, and s p represents the motion vector confidence set, Represents the motion vector subset after being filtered by the top-k function;

[0031] Use context aggregation to fuse the context information contained in the input set into a subset:

[0032]

[0033] Among them, Cgg represents the context aggregation operation; the obtained subset F p+1 It will be used as the input of the next attention pooling layer.

[0034] In one embodiment of the present invention, in step D, a grid with a shape of K×K×C is used to embed X grid The disordered motion vectors are converted into ordered vectors by regularization, and the ordered vector set F used to represent the motion field is obtained. grid .

[0035] In one embodiment of the present invention, in step E, a finer grid is used to embed and adjust the ordered motion vectors in the motion field to better represent the local motion, which is specifically expressed as follows:

[0036]

[0037] in, is a grid embedding of shape K'×K', K'>K, and G represents the graph attention network.

[0038] The present invention also provides a feature matching system based on motion filtering and adjustment, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0039] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention proposes a motion filtering module and an adjustment module; before estimating the motion field, the motion filtering module uses multiple attention pooling layers to capture context and removes most of the erroneous motion vectors in the disordered motion vector set to estimate a motion field containing less noise; the adjustment module uses additional grid embedding on the estimated motion field to fuse more refined position information, so that the motion field can more accurately represent the local motion of different positions to cope with the problem of piecewise smoothing of the motion field in complex scenes; using the above modules, the model can obtain a higher quality motion field, so as to more accurately classify the matches and estimate the essential matrix. The method of the present invention can effectively capture the context and estimate the accurate motion field; the method of the present invention can be applied to multiple fields such as posture estimation, point cloud registration, image stitching and motion segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 FIG. 1 is a flow chart of an embodiment of the present invention.

[0042] Figure 2 This is a diagram showing the effect of an embodiment of the present invention on the YFCC100M dataset.

[0043] Figure 3 This is a diagram showing the effect of an embodiment of the present invention on the SUN3D dataset. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0045] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] The present invention provides a feature matching method based on motion filtering and adjustment, comprising:

[0048] Design a motion filtering block to remove outliers and iteratively capture context;

[0049] A regularization and adjustment block is designed to estimate and adjust the precise motion field with additional position information.

[0050] The following is the specific implementation process of the present invention.

[0051] like Figure 1 As shown, the present invention provides a feature matching method based on motion filtering and adjustment, which specifically includes the following steps:

[0052] A. Given a pair of images, use SIFT (Scale Invariant Feature Transform) to extract key points and corresponding descriptors from the two images of this image pair, and construct an initial matching set including false matches based on the descriptor similarity.

[0053] B. Calculate the initial unordered motion vector set M from the initial matching set, and map (embedding) it to the C-dimensional space through a multi-layer perceptron to obtain a C-dimensional vector set F.

[0054] C. Design a motion filtering module (MotionFiltering) containing multiple attention pooling layers (Pooling) to gradually remove erroneous motion vectors and effectively capture context information. Let the input of the pth attention pooling layer be F p (F0=F), first according to a downsampling rate r p , remove F p Potentially erroneous motion vectors in .

[0055] s p =sigmoid(Γ(PointCN(F p ))

[0056]

[0057] Where PointCN(·) is a context normalization module, Γ is a fully connected layer, and s p represents the motion vector confidence set, Represents the motion vector subset after being filtered by the top-k function.

[0058] Context aggregation is then used to fuse the context information contained in the input set into the subset.

[0059]

[0060] Where Cgg represents the context aggregation operation, the resulting subset F p+1 It will be used as the input of the next attention pooling layer.

[0061] D. After filtering and capturing context using multiple attention pooling layers, use a grid embedding X with a shape of K×K×C grid The disordered motion vectors are converted into ordered vectors by regularization, and the ordered vector set F used to represent the motion field is obtained. grid .

[0062] E. Use a finer grid to embed the ordered motion vectors in the adjustment motion field to better represent the local motion.

[0063]

[0064] in is a grid embedding of shape K'×K' (K'>K), and G represents the graph attention network.

[0065] F. Use the graph attention network to correct the initial set of disordered motion vectors according to the motion field (Recovery) to obtain the corrected set of disordered motion vectors According to the difference before and after the disordered motion vector correction, the correct / wrong match classification results are obtained through the prediction process based on the multi-layer perceptron (Inlierprediction)

[0066] G. Based on the classification results, the essential matrix is ​​estimated using the weighted eight-point algorithm.

[0067] Steps C to G will be iterated 6 times, and the unordered motion vector set obtained each time is All are used as input for the next iteration. In addition, each iteration uses the predicted classification results and the actual category results to calculate the cross entropy loss, and uses the predicted essential matrix and the actual essential matrix to calculate the regression loss to guide network training, and finally obtain a high-performance feature matching model.

[0068] Figure 2 , Figure 3 These are the effect diagrams of this embodiment on the YFCC100M dataset and the SUN3D dataset respectively.

[0069] The present invention also provides a feature matching system based on motion filtering and adjustment, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0070] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A feature matching method based on motion filtering and adjustment, characterized in that: include: Design a motion filtering block to remove outliers and iteratively capture context; A regularization and adjustment block is designed to estimate and adjust the precise motion field with additional position information.

2. A feature matching method based on motion filtering and adjustment according to claim 1, characterized in that: The method comprises the following steps: A. Given a set of image pairs consisting of two images, extract key points and corresponding descriptors from the two images of the image pair, and construct an initial matching set containing false matches based on the descriptor similarity; B. Calculate the initial unordered motion vector set from the initial matching set and map it to a high-dimensional space through a multi-layer perceptron; C. Use a motion filtering module consisting of multiple attention pooling layers to gradually remove erroneous motion vectors and effectively capture contextual information; D. Use grid embedding and graph attention network to transform the unordered motion vectors into ordered vectors to represent the motion field; E. Use a finer grid embedding to adjust the ordered motion vectors in the motion field to better represent local motion; F. Use the graph attention network to correct the initial set of unordered motion vectors according to the motion field. Use a multi-layer perceptron to predict the correct / wrong match classification results based on the difference between the unordered motion vectors before and after correction; G. Based on the classification results, the essential matrix is ​​estimated using the weighted eight-point algorithm.

3. A feature matching method based on motion filtering and adjustment according to claim 2, characterized in that: Steps C to G are iterated 6 times. In each iteration, the cross entropy loss is calculated using the predicted classification results and the actual category results, and the regression loss is calculated using the predicted essential matrix and the actual essential matrix to guide network training, and finally a high-performance feature matching model is obtained.

4. The feature matching method based on motion filtering and adjustment according to claim 2, characterized in that: In step A, the key points and corresponding descriptors are extracted from the two images of the image pair using scale-invariant feature transformation.

5. The feature matching method based on motion filtering and adjustment according to claim 2, characterized in that: In step B, an initial unordered motion vector set M is calculated from the initial matching set, and mapped to a C-dimensional space through a multi-layer perceptron to obtain a C-dimensional vector set F.

6. The feature matching method based on motion filtering and adjustment according to claim 5, characterized in that: Step C is specifically implemented as follows: Let the input of the p-th attention pooling layer be F p , the input of the initial attention pooling layer F0 = F, according to a downsampling rate r p , remove F p Potentially erroneous motion vectors in: s p =sigmoid(Γ(PointCN(F p )) Among them, PointCN(·) is the context normalization module, Γ is a fully connected layer, sigmoid is the activation function, and s p represents the motion vector confidence set, Represents the motion vector subset after being filtered by the top-k function; Use context aggregation to fuse the context information contained in the input set into a subset: Among them, Cgg represents the context aggregation operation; the obtained subset F p+1 It will be used as the input of the next attention pooling layer.

7. The feature matching method based on motion filtering and adjustment according to claim 2, characterized in that: In step D, embed X using a grid of shape K×K×C grid The disordered motion vectors are converted into ordered vectors by regularization, and the ordered vector set F used to represent the motion field is obtained. grid .

8. The feature matching method based on motion filtering and adjustment according to claim 7, characterized in that: In step E, a finer grid embedding is used to adjust the ordered motion vectors in the motion field to better represent the local motion, as shown below: in, is a grid embedding of shape K'×K', K'>K, and G represents the graph attention network.

9. A feature matching system based on motion filtering and adjustment, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images