Characteristic matching method based on grouped motion enhancement

Abstract

The invention relates to a feature matching method based on grouped motion enhancement, and belongs to the technical field of computer vision. According to the method, a grouping motion enhancement module GME is provided, motion fields are grouped along the channel dimension so as to learn multiple motion modes, and the network can accurately recognize inner layers in different motion modes. According to the invention, different motion modes in a scene can be effectively perceived and an accurate motion field can be estimated. The method can be applied to multiple fields such as attitude estimation, point cloud registration, image splicing, stereo matching and motion segmentation.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a feature matching method based on group motion enhancement. Background Art

[0002] Given a pair of images from the same or similar scenes, the goal of feature matching is to establish reliable point-to-point correspondences between the two images. Many tasks in computer vision, such as pose estimation, point cloud registration, image stitching, stereo matching, and motion segmentation, rely on good matching relationships between images.

[0003] The general feature matching process first detects key points and constructs local descriptors, and then establishes an initial set of matches based on the similarity of the descriptors. Common feature extractors, such as the scale-invariant feature transform (SIFT) [1], have been widely used for decades. In recent years, learning-based descriptors, such as SuperPoint [2], have also been widely studied. However, in some complex scenes, such as large viewpoint changes and repeated structures, descriptors show weak discrimination ability, resulting in a large number of false matches (outliers) in the initial match set. Therefore, it is necessary to apply outlier removal methods to identify true matches (inliers). Outlier removal has been studied for many years. Some traditional methods, such as RANSAC [3], obtain the optimal parameter model through repeated sampling to guide the identification of inliers. However, a large number of outliers will seriously affect the effectiveness of sampling, resulting in unsatisfactory performance. Therefore, learning-based methods have begun to be widely studied. A common strategy of learning-based methods is to learn robust feature representations based on the consistency prior of the matches. The pioneering method LFGC [4] regards outlier removal as a classification problem and learns effective feature representations through the proposed context normalization. Since then, many subsequent studies have proposed powerful feature embedding modules to enhance the model's ability to capture context. These learning-based methods have made good progress in outlier removal. However, they only focus on the interactions between correspondences and ignore the valuable information in the underlying motion field. Therefore, in order to exploit the coherence prior of the motion field, ConvMatch[5] uses the matched motion information to estimate the motion field and smoothes the motion field using a convolutional neural network. However, in some scenes with large depth or viewpoint changes, the motion in the motion field may only be locally consistent, resulting in multiple motion patterns contained in the motion field. In this case, the convolutional neural network will over-smooth the motion field, resulting in the inability to preserve the discontinuity of the motion field. This defect in ConvMatch limits its performance in complex scenes.

[0004] References:

[0005] [1]D.G.Lowe,“Distinctive image features from scaleinvariantkeypoints,”International Journal ofComputerVision,vol.60,pp.91–110,2004.

[0006] [2]D.DeTone,T.Malisiewicz,and A.Rabinovich,“Superpoint:Self-supervised interest point detection and description,”in Proceedings oftheIEEE Conference on Computer Vision and Pattern RecognitionWorkshops,pp.224–236,2018.

[0007] [3]M.A.Fischler and R.C.Bolles,“Random sample consensus:A paradigmfor model fitting with applications to image analysis andautomatedcartography,”Communications oftheACM,vol.24,no.6,pp.381–395,1981.

[0008] [4]K.M.Yi,E.Trulls,Y.Ono,V.Lepetit,M.Salzmann,and P.Fua,“Learning tofind good correspondences,”in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition,pp.2666–2674,2018.

[0009] [5] S. Zhang and J. Ma, “Convmatch: Rethinking network design for two viewcorrespondence learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 2023, pp. 3472–3479. Summary of the invention

[0010] The purpose of the present invention is to alleviate the problems existing in the background technology and provide a feature matching method based on group motion enhancement, which can effectively perceive different motion modes in a scene and estimate an accurate motion field.

[0011] To achieve the above-mentioned purpose, the technical solution of the present invention is: a feature matching method based on group motion enhancement, a group motion enhancement module GME is proposed, the motion field is grouped along the channel dimension to learn multiple motion modes, so that the network can accurately identify the inner layer under different motion modes.

[0012] In one embodiment of the present invention, the method comprises the following steps:

[0013] A. Given a pair of images from the same or similar scenes, extract key points and descriptors from the two images respectively, and construct an initial matching set based on the similarity of the descriptors;

[0014] B. Calculate the initial unordered motion vector set from the initial matching set and map it to a high-dimensional space using a multi-layer perceptron;

[0015] C. Interpolate the unordered motion vectors into the grid and convert them into an ordered vector set F used to represent the motion field grid ;

[0016] D. Design a group motion enhancement module GME, which uses two parallel branches to sort the ordered vector set F grid Processing, adding the outputs of the two branches to obtain a new ordered motion vector set to indicate an enhanced playing field;

[0017] E. Use the motion field to correct the initial unordered motion vector set through the graph attention network, and use the multi-layer perceptron to predict the classification result based on the difference before and after correction;

[0018] F. Estimate the essential matrix from the predicted classification results using the weighted eight-point algorithm.

[0019] In one embodiment of the present invention, steps C to F are iterated 6 times, and each iteration calculates a cross entropy loss for the predicted classification result and a regression loss for the estimated essential matrix to guide network training, and finally obtains a high-performance feature matching model.

[0020] In one embodiment of the present invention, in step A, a key point detector is used to extract key points and descriptors from the two images respectively.

[0021] In one embodiment of the present invention, in step C, a graph attention network is used to interpolate the unordered motion vectors into a grid.

[0022] In one embodiment of the present invention, in step D, the first branch uses the grouped motion to enhance the attention to obtain the enhanced ordered vector set F e ; The second branch uses a convolutional neural network to train the ordered vector set F grid Smoothing and denoising.

[0023] In one embodiment of the present invention, the first branch uses the grouped motion to enhance the attention to obtain the enhanced ordered vector set F e , the specific implementation is as follows:

[0024] The ordered vector set F grid Group along the channel dimension, each group is a sub-vector set F' = {f' i}, for each group, use average pooling to calculate a global feature g:

[0025]

[0026] Where m is the number of sub-vectors in the sub-vector set, and AvgPool(·) is the average pooling operation;

[0027] Then, the affinity c of each sub-vector to the global feature is calculated i , and use the affinity enhancer vector:

[0028] c i = g·f′ i

[0029]

[0030] Among them, σ(·) is the sigmoid activation function, norm(·) is the normalization operation, is the enhanced sub-vector; each grouped sub-vector is reorganized to obtain an enhanced ordered vector set F e :

[0031]

[0032] where ∥ denotes the operation of connecting along the channel, Γ(·) denotes the operation of reshaping the input into a K×K×C shape, Represents the set of subvectors of the i-th group.

[0033] In one embodiment of the present invention, the outputs of the two branches are added to obtain a new ordered motion vector set: To represent the enhanced motion field, the specific calculation formula is as follows:

[0034]

[0035] Among them, CNN(·) represents convolutional neural network.

[0036] The present invention also provides a feature matching system based on group motion enhancement, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0037] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention proposes to combine a grouped motion enhancement module with a convolutional neural network. The grouped motion enhancement module groups the ordered motion vectors representing the motion field along the channel dimension, allowing each group to learn a specific motion pattern during the network training process, so that the motion vector belonging to the motion pattern can be strongly responded to by the sub-field of the group. The combination of the grouped motion enhancement module and the convolutional neural network can make the obtained motion field better adapt to the situation containing multiple motion patterns, thereby enabling the model to more accurately classify matches and estimate the essential matrix. The method of the present invention can effectively perceive different motion patterns in the scene and estimate an accurate motion field. The method of the present invention can be applied to multiple fields such as posture estimation, point cloud registration, image stitching, stereo matching and motion segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of an embodiment of the present invention.

[0040] Figure 2 A group motion enhancement module ( Figure 1 Flowchart of the GME module in .

[0041] Figure 3 This is a diagram showing the effect of an embodiment of the present invention on the YFCC100M dataset.

[0042] Figure 4 This is a diagram showing the effect of an embodiment of the present invention on the SUN3D dataset. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0044] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0046] The present invention provides a feature matching method based on grouped motion enhancement and proposes a grouped motion enhancement module GME, which groups motion fields along the channel dimension to learn multiple motion modes so that the network can accurately identify inner layers under different motion modes.

[0047] The following is the specific implementation process of the present invention.

[0048] like Figure 1 As shown, the present invention provides a feature matching method based on group motion enhancement, which specifically includes the following steps:

[0049] A. Given a pair of images from the same or similar scenes, use a key point detector to extract key points and descriptors from the two images respectively, and construct an initial matching set based on the descriptor similarity.

[0050] B. Calculate the initial unordered motion vector set M from the initial matching set, and use a multi-layer perceptron (Projection) to map it to a C-dimensional space to obtain a C-dimensional vector set F.

[0051] C. Use the Graph Attention Network (GAT) to interpolate the unordered motion vectors into a grid X of shape K×K×C grid , converted into an ordered vector set F used to represent the motion field grid .

[0052] D. Figure 2 As shown in the figure, a group motion enhancement module (GME block) is designed to use two parallel branches to sort the ordered vector set F gridto be processed.

[0053] In the first branch, the ordered vectors are grouped along the channel dimension, and each group is a subvector set F' = {f' i}, for each group, a global feature g is calculated using average pooling.

[0054]

[0055] Where m is the number of sub-vectors and AvgPool(·) is the average pooling operation. Then, the affinity c of each sub-vector with the global feature is calculated. i , and use the affinity enhancer vector.

[0056] c i = g·f′ i

[0057]

[0058] Among them, σ(·) is the sigmoid activation function, norm(·) is the normalization operation, is the enhanced sub-vector. After each group of sub-vectors are reorganized, the enhanced ordered vector set F is obtained. e .

[0059]

[0060] where ∥ denotes the operation of connecting along the channel, Γ(·) denotes the operation of reshaping the input into a K×K×C shape, Represents the set of subvectors of the i-th group.

[0061] In the second branch, a convolutional neural network is used to train the ordered vector set F grid Smoothing and denoising are performed. The outputs of the two branches are added together to obtain a new set of ordered motion vectors. to indicate an enhanced playing field.

[0062]

[0063] Among them, CNN(·) represents convolutional neural network.

[0064] E. Use the motion field to correct the initial set of unordered motion vectors through GAT According to the difference before and after correction, the classification result is obtained through the prediction process based on multi-layer perceptron (Inlierprediction)

[0065] F. Use the weighted eight-point algorithm to classify the results Estimate the essential matrix in .

[0066] Steps C to F will be iterated 6 times, and the unordered motion vector set obtained in each iteration is As the input of the next iteration. In addition, each iteration calculates a cross entropy loss for the predicted classification result and a regression loss for the predicted essential matrix to guide network training, and finally obtain a high-performance feature matching model.

[0067] Figure 3 , Figure 4 These are the effect diagrams of this embodiment on the YFCC100M dataset and the SUN3D dataset respectively.

[0068] The present invention also provides a feature matching system based on group motion enhancement, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0069] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A feature matching method based on group motion enhancement, It is characterized in that A grouped motion enhancement module (GME) is proposed to group motion fields along the channel dimension to learn multiple motion modes, so that the network can accurately identify inner layers under different motion modes.

2. A feature matching method based on group motion enhancement according to claim 1, It is characterized in that The method comprises the following steps: A. Given a pair of images from the same or similar scenes, extract key points and descriptors from the two images respectively, and construct an initial matching set based on the similarity of the descriptors; B. Calculate the initial unordered motion vector set from the initial matching set and map it to a high-dimensional space using a multi-layer perceptron; C. Interpolate the unordered motion vectors into the grid and convert them into an ordered vector set F used to represent the motion field grid ; D. Design a group motion enhancement module GME, which uses two parallel branches to sort the ordered vector set F grid Processing, adding the outputs of the two branches to obtain a new ordered motion vector set to indicate an enhanced playing field; E. Use the motion field to correct the initial unordered motion vector set through the graph attention network, and use the multi-layer perceptron to predict the classification result based on the difference before and after correction; F. Estimate the essential matrix from the predicted classification results using the weighted eight-point algorithm.

3. The feature matching method based on group motion enhancement according to claim 2, It is characterized in that Steps C to F are iterated 6 times. In each iteration, a cross entropy loss is calculated for the predicted classification result, and a regression loss is calculated for the estimated essential matrix to guide network training, and finally a high-performance feature matching model is obtained.

4. The feature matching method based on group motion enhancement according to claim 2, It is characterized in that In step A, a key point detector is used to extract key points and descriptors from the two images respectively.

5. The feature matching method based on group motion enhancement according to claim 2, It is characterized in that In step C, the unordered motion vectors are interpolated into the grid using a graph attention network.

6. The feature matching method based on group motion enhancement according to claim 2, It is characterized in that In step D, the first branch uses grouped motion to enhance attention and obtains an enhanced ordered vector set F e ; The second branch uses a convolutional neural network to train the ordered vector set F grid Smoothing and denoising.

7. The feature matching method based on group motion enhancement according to claim 6, It is characterized in that The first branch uses grouped motion to enhance attention and obtains an enhanced ordered vector set F e , the specific implementation is as follows: The ordered vector set F grid Group along the channel dimension, each group is a sub-vector set F' = {f' i }, for each group, use average pooling to calculate a global feature g: Where m is the number of sub-vectors in the sub-vector set, and AvgPool(·) is the average pooling operation; Then, the affinity c of each sub-vector to the global feature is calculated i , and use the affinity enhancer vector: c i =g·f′ i Among them, σ(·) is the sigmoid activation function, norm(·) is the normalization operation, is the enhanced sub-vector; each grouped sub-vector is reorganized to obtain an enhanced ordered vector set F e : where ∥ denotes the operation of connecting along the channel, Γ(·) denotes the operation of reshaping the input into a K×K×C shape, Represents the set of subvectors of the i-th group.

8. The feature matching method based on group motion enhancement according to claim 7, It is characterized in that Add the outputs of the two branches to get a new set of ordered motion vectors To represent the enhanced motion field, the specific calculation formula is as follows: Among them, CNN(·) represents convolutional neural network.

9. A feature matching system based on group motion enhancement, It is characterized in that The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images