Image matching pair filtering method and device based on multi-view cyclic consistency
By constructing a multi-view cycle-consistent image matching method, and using the triangle hypergraph and MVCEM algorithm to prune mismatched edges, the problem of insufficient accuracy and stability in multi-view image matching is solved, and higher precision image matching results are achieved.
Patent Information
- Application Number
- CN202310344970.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-31
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2043-03-31
AI Technical Summary
Existing image matching methods are insufficient in terms of accuracy and stability, especially in the case of multiple views, where the effect of filtering out mismatches is poor.
A multi-view cycle consistency-based image matching pair filtering method is adopted. By acquiring image sets from different angles in the same scene, feature matching is performed using the dual-view matching method to construct a triangular hypergraph. The MVCEM algorithm is then used to perform cycle consistency constraints under the constraint of any two-hop chain. The matching matrix is refined to prune mismatched edges and obtain accurate matching pairs.
It improves the accuracy and stability of image matching, is suitable for large-scale image matching problems, and is independent of local feature types and dual-view matching methods, making it widely applicable.
Smart Images
Figure CN116403013B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to the field of image matching, in particular to an image matching pair filtering method and device based on multi-view cyclic consistency, a computer device and a storage medium. BACKGROUND
[0002] Estimating the correspondence between two images is a key module in many computer vision tasks, including structure from motion (SfM), multi-view stereo (MVS), and image-based relocalization. The standard correspondence finding pipeline is implemented in two steps. First, a set of putative correspondences is obtained by matching local features through (approximate) nearest neighbor search. Second, to prevent affecting downstream vision tasks, post-processing measures are adopted to extract correct matches (also known as inliers) and reject false matches (outliers) for matching pair filtering. In recent years, with the advent of deep learning, people have been pushing to solve the false match filtering problem in a differentiable way.
[0003] However, the current traditional false match filtering work mostly focuses on the two-view information, and has problems of low accuracy, low stability, complexity, etc. SUMMARY
[0004] Therefore, it is necessary to provide an image matching pair filtering method and device based on multi-view cyclic consistency, a computer device and a storage medium, which can improve the image matching accuracy.
[0005] An image matching pair filtering method based on multi-view cyclic consistency, the method comprises:
[0006] Obtaining an image set to be filtered; the image set is a set of original views of different shooting angles under the same scene;
[0007] Performing image retrieval and feature matching on the image set according to a two-view matching method to obtain a scene graph; the scene graph comprises nodes and edges; the nodes represent local features; the edges are provided with paired matching information; the matching information comprises a matching matrix;
[0008] Constructing a triangle-based hypergraph according to a triangle enumeration technique, using the hypergraph to record all double-hop chains connecting the scene graphs, and refining the matching matrix according to the cyclic consistency constraint under the restriction of any double-hop chain by using an MVCEM algorithm to obtain filtered matching pairs.
[0009] In one of the embodiments, the triangle-based hypergraph constructed according to the triangle enumeration technique comprises:
[0010] The triangle-based hypergraph constructed according to the triangle enumeration technique is G H ={V H ,EH}, wherein V H represents the same node set as the original view graph, E H represents an edge set containing all possible paths brought by the triangular pattern.
[0011] In one embodiment, the cycle consistency constraint includes a first constraint, a second constraint and a third constraint; the first constraint is X a，a = I M ; the second constraint is X a，b = (X b，a ) T ; and the third constraint is X a，b X k，b ≤ X a，b ; wherein x a，b represents a matching matrix of view I a and view I b , I M represents a unit matrix of size M, and x a，a represents a self-matching matrix.
[0012] In one embodiment, the matching matrix is refined according to the cycle consistency constraint under the restriction of any double-hop chain by using the MVCEM algorithm to obtain filtered matching pairs, including:
[0013] The matching matrix is refined according to the third constraint under the restriction of any double-hop chain by using the MVCEM algorithm to obtain filtered matching pairs.
[0014] In one embodiment, the matching matrix is refined according to the third constraint under the restriction of any double-hop chain by using the MVCEM algorithm to obtain filtered matching pairs, including:
[0015] The matching matrix is locally refined according to the third constraint under the restriction of any double-hop chain to obtain locally evolved matching pairs;
[0016] The locally evolved matching pairs are
[0017]
[0018] wherein i represents the serial number of the path, and k i represents the i-th path.
[0019] In one embodiment, the locally evolved matching pairs are globally evolved according to the third constraint under the restriction of any double-hop chain to obtain filtered matching pairs;
[0020] The filtered matching pairs are
[0021]
[0022] wherein N c represents all possible paths in the hypergraph.
[0023] An image matching pair filtering device based on multi-view cyclic consistency, the device comprising:
[0024] An image set acquisition module, configured to acquire an image set to be filtered; the image set being a collection of original views at different shooting angles under the same scene;
[0025] A view matching module, configured to perform image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprising nodes and edges; the nodes representing local features; the edges carrying paired matching information; the matching information comprising a matching matrix;
[0026] A matching pair filtering module, configured to construct a triangle-based hypergraph according to a triangle enumeration technique, utilize the hypergraph to record all double-hop chains connecting the scene graphs, and refine the matching matrix according to cyclic consistency constraints under the restriction of any double-hop chain by using an MVCEM algorithm to obtain filtered matching pairs.
[0027] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:
[0028] Acquiring an image set to be filtered; the image set being a collection of original views at different shooting angles under the same scene;
[0029] Performing image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprising nodes and edges; the nodes representing local features; the edges carrying paired matching information; the matching information comprising a matching matrix;
[0030] Constructing a triangle-based hypergraph according to a triangle enumeration technique, utilizing the hypergraph to record all double-hop chains connecting the scene graphs, and refining the matching matrix according to cyclic consistency constraints under the restriction of any double-hop chain by using an MVCEM algorithm to obtain filtered matching pairs.
[0031] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:
[0032] Acquiring an image set to be filtered; the image set being a collection of original views at different shooting angles under the same scene;
[0033] Performing image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprising nodes and edges; the nodes representing local features; the edges carrying paired matching information; the matching information comprising a matching matrix;
[0034] According to the triangle enumeration technique, a triangle-based hypergraph is constructed, all double-hop chains between the scene graphs are recorded by using the hypergraph, and the MVCEM algorithm is used to refine the matching matrix according to the loop consistency constraint under the restriction of any double-hop chain, so that the filtered matching pairs are obtained.
[0035] The image matching pair filtering method, device, computer device and storage medium based on multi-view loop consistency can first perform image retrieval and feature matching on the image set according to the double-view matching method, obtain the scene graph, strengthen the edge information of the original view, improve the accuracy of the scene graph after matching, refine the matching matrix according to the loop consistency constraint under the restriction of any double-hop chain, prune all edges in the scene graph, remove the mis-matched edges, obtain a more accurate view, make the pruned view have the triple consistency, introduce the internal constraint of the multi-view to prune the noise feature matching with low computational complexity, and further improve the accuracy of the subsequent matching pair filtering. The method is also applicable to large-scale image matching problems. According to the hypergraph, all double-hop chains between the scene graphs are recorded, the matching matrix is refined according to the loop consistency constraint under the restriction of any double-hop chain, the reachability of all available paths in the view is checked to retain the trusted matching pairs, and the filtered matching pairs are obtained. Meanwhile, the method proposed in the application is independent of the type of local feature and the way of double-view matching, can be easily integrated into any general application program depending on local feature matching, and has a wider applicability range. BRIEF DESCRIPTION OF DRAWINGS
[0036] Figure 1 The application scenario diagram of the image matching pair filtering method based on multi-view loop consistency in one embodiment;
[0037] Figure 2 The structural block diagram of the image matching pair filtering device based on multi-view loop consistency in one embodiment;
[0038] Figure 3 The internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.
[0040] In one embodiment, as shown in Figure 1 a method for filtering image matching pairs based on multi-view loop consistency is provided, including the following steps:
[0041] Step 102, obtaining an image set to be filtered; the image set is a collection of original views of different shooting angles under the same scene.
[0042] Step 104, performing image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph includes nodes and edges; the nodes represent local features; the edges have paired matching information; the matching information includes a matching matrix.
[0043] Performing image retrieval and feature matching on the image set according to the double-view matching method can enhance the edge information of the original view and improve the accuracy of the scene graph after matching.
[0044] Step 106, constructing a triangle-based hypergraph according to a triangle enumeration technique, using the hypergraph to record all double-hop chains connecting the scene graphs, and using an MVCEM algorithm to refine the matching matrix according to the loop consistency constraint under the restriction of any double-hop chain to obtain filtered matching pairs.
[0045] Constructing a triangle-based hypergraph according to the triangle enumeration technique can explore multi-view information beyond the limitation of double views, record all possible double-hop chains connecting the views according to the hypergraph, expand the search range of the matching pairs, reduce errors during subsequent filtering, and obtain more accurate matching pairs; using the MVCEM algorithm to refine the matching matrix according to the loop consistency constraint under the restriction of any double-hop chain can prune all edges in the scene graph, remove mis-matched edges, obtain more accurate views, make the pruned views have triplet consistency, introduce the inherent constraint of multi-views to prune noise feature matching with low computational complexity, and further improve the accuracy of subsequent matching pair filtering, which is also applicable to large-scale image matching problems; under the restriction of any double-hop chain, the loop consistency constraint is used to refine the matching matrix, check the reachability of all available paths in the view to retain reliable matching pairs, and obtain filtered matching pairs.
[0046] In the image matching pair filtering method based on multi-view cyclic consistency, firstly, image retrieval and feature matching are performed on the image set according to the double-view matching method to obtain a scene graph, the edge information of the original view can be enhanced, and the accuracy of the scene graph after matching can be improved, the MVCEM algorithm is used to refine the matching matrix according to the cyclic consistency constraint under the restriction of any double-hop chain, all edges in the scene graph are pruned, the mis-matching edges are removed, and a more accurate view is obtained, so that the pruned view has a triple consistency, the internal constraint of the multi-view is introduced to prune the noise feature matching with low computational complexity, and then the accuracy of subsequent matching pair filtering is improved, and the method is also applicable to large-scale image matching problems, all double-hop chains between the scene graphs are connected according to the hypergraph record, the matching matrix is refined according to the cyclic consistency constraint under the restriction of any double-hop chain, the reachability of all available paths in the view is checked to retain the reliable matching pair, and the filtered matching pair is obtained. Meanwhile, the method provided in the application is irrelevant to the type of local feature and the way of double-view matching, can be easily integrated into any general application program depending on local feature matching, and has a wider applicability range.
[0047] In one embodiment, a triangle-based hypergraph is constructed according to a triangle enumeration technique, including:
[0048] The triangle-based hypergraph constructed according to the triangle enumeration technique is G H ={V H ,E H},wherein V H represents a same node set as the original view graph, and E H represents an edge set containing all possible paths brought by the triangle pattern.
[0049] In a specific embodiment, wherein a and b are in V H , [k1, k2, …, k n ] represents a transfer node connecting the node a and the node b.
[0050] In one embodiment, the cyclic consistency constraint includes a first constraint, a second constraint and a third constraint; the first constraint is X a，a =I M ; the second constraint is X a，b =(X b，a ) T ; and the third constraint is X a，k X k，b ≤X a，b ; wherein X a，b represents a matching matrix of a view I a and a view I b , I M represents a unit matrix with a size of M, and Xa，a represents a matching matrix.
[0051] In specific embodiments, the matching matrix satisfies the following constraints
[0052] X a，b 1N≤1 M and(X a，b )T 1M ≤1 N
[0053] In one of the embodiments, the MVCEM algorithm is used to refine the matching matrix under the constraint of arbitrary double-hop chain according to the loop consistency constraint to obtain the filtered matching pairs, including:
[0054] The MVCEM algorithm is used to refine the matching matrix under the constraint of arbitrary double-hop chain according to the third constraint to obtain the filtered matching pairs.
[0055] In one of the embodiments, the MVCEM algorithm is used to refine the matching matrix under the constraint of arbitrary double-hop chain according to the third constraint to obtain the filtered matching pairs, including:
[0056] The matching matrix is locally refined under the constraint of arbitrary double-hop chain according to the third constraint to obtain the locally evolved matching pairs;
[0057] The locally evolved matching pairs are
[0058]
[0059] Wherein, i represents the serial number of the path, k i represents the i-th path.
[0060] In one of the embodiments, the locally evolved matching pairs are globally evolved under the constraint of arbitrary double-hop chain according to the third constraint to obtain the filtered matching pairs;
[0061] The filtered matching pairs are
[0062]
[0063] Wherein, N c represents all possible paths in the hypergraph.
[0064] In specific embodiments, the MVCEM algorithm is constructed to modify all edges E in the view graph to enforce the triple consistency. For each edge in the view graph First, obtain its two-hop realizable chain Based on the hypergraph, part of the matching matrix is refined from X a，b to wherein
[0065] updated Also consistent with the cycle-consistency constraint:
[0066]
[0067] The specific refinement includes two stages: in the first stage, local evolution is performed under the constraint of any path (a, k i , b), only the corresponding relationship that can be verified from the specified path (a, k i , b) is retained, and the locally evolved matching pair is obtained as
[0068]
[0069] Where & denotes an element-wise AND operator.
[0070] In the second stage, all double-hop paths (a, k i , b) are considered globally, and the trusted matching pair is retained by checking the reachability of all available paths (a, k i , b), and the filtered most accurate matching pair is obtained, and the final result is as follows:
[0071]
[0072] Where || denotes an element-wise OR operator.
[0073] By pruning all edges in the view graph using the MVCEM, a consistent view graph G = (V, E) can be obtained, which is conducive to subsequent computer vision work.
[0074] It should be understood that, although the steps in the flowchart of Figure 1 are shown in order according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.
[0075] Figure 2 In one embodiment, asAs shown, an image matching pair filtering device based on multi-view cycle consistency is provided, comprising: an image set acquisition module 202, a view matching module 204, and a matching pair filtering module 206, wherein:
[0076] The image set acquisition module 202 is configured to acquire an image set to be filtered; the image set is a collection of original views at different shooting angles under the same scene;
[0077] The view matching module 204 is configured to perform image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprises nodes and edges; the nodes represent local features; the edges carry paired matching information; the matching information comprises a matching matrix;
[0078] The matching pair filtering module 206 is configured to construct a triangle-based hypergraph according to a triangle enumeration technique, record all double-hop chains connecting the scene graphs using the hypergraph, and refine the matching matrix according to a cycle consistency constraint under the restriction of any double-hop chain using an MVCEM algorithm to obtain a filtered matching pair.
[0079] In one embodiment, the matching pair filtering module 206 is further configured to construct a triangle-based hypergraph according to a triangle enumeration technique, comprising:
[0080] The triangle-based hypergraph constructed according to the triangle enumeration technique is G H ={V H , E H}, wherein V H represents a node set identical to that of the original view graph, and E H represents an edge set containing all possible paths brought by the triangle pattern.
[0081] In one embodiment, the matching pair filtering module 206 is further configured to refine the matching matrix according to a cycle consistency constraint under the restriction of any double-hop chain using an MVCEM algorithm to obtain a filtered matching pair, comprising:
[0082] The matching matrix is refined according to the third constraint under the restriction of any double-hop chain using the MVCEM algorithm to obtain a filtered matching pair.
[0083] In one embodiment, the matching pair filtering module 208 is further configured to refine the matching matrix according to the third constraint under the restriction of any double-hop chain using the MVCEM algorithm to obtain a filtered matching pair, comprising:
[0084] The matching matrix is locally refined according to the third constraint under the restriction of any double-hop chain to obtain a locally evolved matching pair;
[0085] The locally evolved matching pair is
[0086]
[0087] wherein i represents the serial number of the path, k i represents the i-th path.
[0088] In one of the embodiments, the matching pair filtering module 208 is further configured to globally evolve the locally evolved matching pairs according to the third constraint under the restriction of any two-hop chain, to obtain filtered matching pairs.
[0089] The filtered matching pairs are
[0090]
[0091] wherein N c represents all possible paths in the hypergraph.
[0092] The specific definitions of the image matching pair filtering device based on multi-view cyclic consistency can refer to the definitions of the image matching pair filtering method based on multi-view cyclic consistency in the foregoing, and will not be described here. Each module in the image matching pair filtering device based on multi-view cyclic consistency can be realized by software, hardware, and a combination thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above-mentioned modules.
[0093] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an image matching pair filtering method based on multi-view cyclic consistency. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad, or mouse, etc.
[0094] Those skilled in the art can understand that, Figure 3The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0096] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.
[0097] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.
Claims
1. A method for filtering image matching pairs based on multi-view cyclic consistency, characterized in that, The method comprises: acquiring an image set to be filtered; the image set is a set of original views of different shooting angles under the same scene; performing image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprises nodes and edges; the nodes represent local features; the edges are provided with matching information in pairs; the matching information comprises a matching matrix; constructing a triangle-based hypergraph according to a triangle enumeration technique, using the hypergraph to record all double-hop chains connecting the scene graphs, and refining the matching matrix under the restriction of any double-hop chain according to the cyclic consistency constraint by using an MVCEM algorithm to obtain filtered matching pairs.
2. The method of claim 1, wherein, constructing a triangle-based hypergraph according to a triangle enumeration technique, comprising: According to the triangle enumeration technique, a triangle-based hypergraph is constructed as G H = {V H , E H}, where V H represents the same node set as the original view graph, and E H represents an edge set containing all possible paths brought by the triangle pattern.
3. The method of claim 1, wherein, The cycle consistency constraint comprises a first constraint, a second constraint and a third constraint; the first constraint is X a，a = I M ; the second constraint is X a，b = (X b，a ) T ; and the third constraint is X a，k X k，b ≤ X a，b ; wherein x a，a represents a matching matrix of view I a and view I b , I M represents a unit matrix of size M, and x a，a represents a self-matching matrix.
4. The method of claim 3, wherein, refining the matching matrix under the restriction of any double-hop chain according to the cyclic consistency constraint by using an MVCEM algorithm to obtain filtered matching pairs, comprising: refining the matching matrix under the restriction of any double-hop chain according to the third constraint by using an MVCEM algorithm to obtain filtered matching pairs.
5. The method of claim 4, wherein, refining the matching matrix under the restriction of any double-hop chain according to the third constraint by using an MVCEM algorithm to obtain filtered matching pairs, comprising: locally refining the matching matrix under the restriction of any double-hop chain according to the third constraint to obtain locally evolved matching pairs; the locally evolved matching pairs are wherein i represents the serial number of the path, k i represents the i-th path.
6. The method of claim 4, wherein, the method further comprises: globally evolving the locally evolved matching pairs under the restriction of any double-hop chain according to the third constraint to obtain filtered matching pairs; the filtered matching pairs are where N c represents all possible paths in the hypergraph.
7. An image matching pair filtering apparatus based on multi-view cyclic consistency, characterized by, the device comprises: an image set acquisition module configured to acquire an image set to be filtered; the image set is a set of original views of different shooting angles under the same scene; a view matching module configured to perform image retrieval and feature matching on the image set according to a double-view matching method to obtain a scene graph; the scene graph comprises nodes and edges; the nodes represent local features; the edges are provided with matching information in pairs; the matching information comprises a matching matrix; a matching pair filtering module configured to construct a triangle-based hypergraph according to a triangle enumeration technique, use the hypergraph to record all double-hop chains connecting the scene graphs, and refine the matching matrix under the restriction of any double-hop chain according to the cyclic consistency constraint by using an MVCEM algorithm to obtain filtered matching pairs.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.