Systems and methods for virtual reality and augmented reality

Abstract

This specification relates to feature matching. Our method establishes point-to-point correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attention map neural network to solve an assignment optimization problem. A deep mid-level matcher acts as a mid-level and gracefully handles partial point visibility and occlusions, yielding a partial assignment matrix.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 935,597, filed November 14, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to connected mobile computing systems, methods, and configurations, and more particularly to mobile computing systems, methods, and configurations characterized by at least one wearable component that can be used for virtual and / or augmented reality operations. Background Technology

[0004] The desired features are lightweight, low-cost, small-sized, wide virtual image field of view, and as transparent as possible for mixed reality or augmented reality near-eye displays. Furthermore, it is desirable to have configurations that present virtual image information in multiple focal planes (e.g., two or more) to suit a wide range of use cases without exceeding acceptable tolerances for convergence-accommodation mismatch. (Reference) Figure 8 An augmented reality system is illustrated, characterized by a head-mounted viewing component (2), a handheld controller component (4), and interconnected auxiliary computing or controller components (6), which can be configured to be worn by the user as a waist bag, etc. Each of these components can be operatively coupled to each other (10, 12, 14, 16, 17, 18) and to other connected resources (8), such as cloud computing or cloud storage resources, via wired or wireless communication configurations (such as those specified by IEEE 802.11, Bluetooth (RTM), and other connectivity standards and configurations). For example, in the United States, patent application serial numbers 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 627,155, 62 / 518,539, 16 / 229,532, 16 / 155,564, 15 / 413,284, 16 / 020,541, 62,702,322, 62 / 206,765, 15,597,694, 16 As described in / 221,065, 15 / 968,673, 62 / 682,788 and 62 / 899,678, each of which is incorporated herein by reference in its entirety, various aspects of such components, such as various embodiments of the two depicted optical elements (20), through which a user can see the world around them in conjunction with visual components that can be generated by associated system components for an augmented reality experience. Figure 8As shown, such a system may also include various sensors configured to provide information about the user's surroundings, including but not limited to sensors of various camera types (such as monochrome, color / RGB, and / or thermal imaging components) (22, 24, 26), depth camera sensors (28), and / or sound sensors such as microphones (30). Compact and persistently connected wearable computing systems and components, such as those described herein, are needed to provide users with a rich augmented reality experience. Summary of the Invention

[0005] This document describes some aspects of what may be called a "deep intermediate matcher," where a neural network is configured to match two sets of local features by jointly finding correspondences and rejecting unmatched points. This neural network configuration can be used with, for example, Figure 8 The spatial computing resources shown are used in association, including but not limited to cameras and processing resources that include such spatial computing systems. Within a deep intermediate-end matcher type configuration, assignments can be estimated by solving an optimal transfer problem, the cost of which is predicted by a graph neural network. We describe an attention-based, flexible context aggregation mechanism that enables deep intermediate-end matcher configurations to jointly infer underlying 3D scenes and feature assignments. Compared to traditional hand-designed heuristics, our technique learns prior knowledge of geometric transformations and regularities in the 3D world through end-to-end training from images to correspondences. Deep intermediate-end matchers outperform other learning methods and set new state-of-the-art standards on pose estimation tasks in challenging real-world indoor and outdoor environments. These methods and configurations are matched in real-time on modern graphics processing units (“GPUs”) and can be easily integrated into modern motion recovery structures (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which can be incorporated into systems such as... Figure 8 The system shown.

[0006] This invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium, and a set of instructions on the computer-readable medium. The set of instructions may include a deep intermediate matcher architecture, which may include an attention graph neural network having a keypoint encoder for mapping keypoint locations p and their visual descriptors d to a single vector; alternating self-attention layers and cross-attention layers that create a representation f based on the vector being repeated L times; and an optimal matching layer that creates an M×N score matrix from the representation f and finds an optimal partial assignment based on the M×N score matrix.

[0007] The computer system may further include an initial representation of each keypoint i in the keypoint encoder. (0) xi Combining visual appearance and location, the corresponding keypoint locations are embedded into a high-dimensional vector with a multilayer perceptron (MLP), as follows:

[0008] (0) x i =d i +MLP(p i ).

[0009] The computer system may further include a keypoint encoder, which allows attention graph neural networks to jointly infer appearance and location.

[0010] The computer system may further include a multipath graph neural network with a single complete graph in the keypoint encoder, the complete graph having nodes that serve as keypoints for two images.

[0011] The computer system may further include a multipath graph, which has two types of undirected edges, namely, in-image edges (from edge; E) connecting a keypoint i to all other keypoints within the same image. self ) and image-to-image edges (crossing edges, E) that connect keypoint i to all keypoints in another image. cross The resulting multipath graph neural network propagates information along both types of edges using a message-passing formula, starting from the high-dimensional state of each node and computing the updated representation at each layer by simultaneously aggregating messages across all given edges across all nodes.

[0012] The computer system may further include if (l) x A i If the intermediate representation of element i in image A at layer l is given, then message m... E→i It is the result of the aggregation of all keypoints {j:(i,j)∈E}, where E∈{E self E cross}, and the remaining message passing for all i in A is updated as follows:

[0013]

[0014] Where [·||·] indicates cascading.

[0015] The computer system may further include a fixed number of layers L with different parameters linked together and alternately aggregated along their own edges and cross edges, such that starting from l = 1, if l is odd, then E = E self And if l is even, then E = E cross .

[0016] The computer system may further include alternating self-attention layers and cross-attention layers employing an attention mechanism for computation, where the computation message m E→i And perform aggregation, where self-edges are based on self-attention and cross edges are based on cross-attention, where for the representation of i, query q i Based on some element attributes (keys) j Get the values ​​v of these elements j And calculate the message as a weighted average of the values:

[0017]

[0018] The computer system may further include an attention mask α. ij It is a Softmax based on key query similarity:

[0019]

[0020] The computer system may further include computing the corresponding keys, queries, and values ​​as linear projections of deep features of a graph neural network, where the query keypoint i is in image Q and all source keypoints are in image S, (Q,S)∈{A,B} 2 In the equation:

[0021]

[0022]

[0023] The computer system may further include alternating self-attention layers and cross-attention layers. The final matching descriptor is a linear projection.

[0024]

[0025] The computer system may further include an optimal matching layer that expresses the pairwise scores of the set as similarity of matching descriptors:

[0026]

[0027] Where <·,·> is the inner product. Unlike the learned visual descriptors, the matching descriptors are not normalized, and their sizes can vary per feature as well as during training to reflect prediction confidence.

[0028] The computer system may further include an optimal matching layer for occlusion and visibility to suppress occluded keypoints, and augment each set of keypoints with dustbin scores so that mismatched keypoints are explicitly assigned to dustbin scores.

[0029] The computer system may further include expanding the fraction S to S by adding new rows and columns, point-to-box and box-to-box fractions populated with a single learnable parameter:

[0030]

[0031] The computer system can further include an optimal matching layer that uses the Sinkhorn algorithm for T iterations to find the optimal partial allocation based on an M×N fraction matrix.

[0032] The computer system can further include, after T iterations, the optimal matching layer discarding the trash bin scores and restoring P = P - 1:M,1:N ,in

[0033] P1 N ≤1 M and

[0034] It is the original allocation and

[0035] and

[0036] It is an allocation of expanded trash can points.

[0037] The present invention also provides a computer-implemented method system, which may include a keypoint encoder of an attention graph neural network employing a deep intermediate matcher architecture, mapping keypoint locations p and their visual descriptors d to a single vector; and alternating self-attention layers and cross-attention layers of the attention graph neural network employing a deep intermediate matcher architecture, performing repetition L times based on the vector to create a representation f; and performing an optimal matching layer of the attention graph neural network employing a deep intermediate matcher architecture to create an M×N score matrix from the representation f, and finding the optimal part assignment based on the M×N score matrix. Attached Figure Description

[0038] The invention is further described by way of example with reference to the accompanying drawings, in which:

[0039] Figure 1 This is a representative sketch illustrating feature matching using a deep intermediate-end matcher;

[0040] Figure 2 The diagram shows the estimated correspondence between two difficult indoor image pairs, calculated by a deep intermediate-end matcher.

[0041] Figure 3 This is a representative sketch illustrating how we devise a deep intermediate matcher to solve the optimization problem;

[0042] Figure 4It is an image that displays the mask as rays;

[0043] Figure 5 It is a graph showing the indoor and outdoor attitude estimates;

[0044] Figure 6 This demonstrates qualitative image matching;

[0045] Figure 7 Visual attention is shown in self-attention masks and cross-attention masks at different layers and heads; and

[0046] Figure 8 An augmented reality system is shown. Detailed Implementation

[0047] Finding correspondences between points in an image is a crucial step in computer vision tasks such as 3D reconstruction or visual localization (e.g., Simultaneous Localization and Mapping (SLAM) and Structure-of-Motion (SfM)). These correspondences are used to estimate 3D structure and camera pose after matching local features (a process called data association). Factors such as large viewpoint changes, occlusion, blurring, and lack of texture make 2D-to-2D data association particularly challenging.

[0048] In this description, we propose a novel way of thinking about the feature matching problem. Instead of learning better, task-independent local features and then performing simple matching heuristics and tricks, we propose using a novel neural architecture called Deep Intermediate Matcher (DMEM) to learn the matching process from pre-existing local features. In the context of SLAM, where the problem is typically decomposed into a visual feature detection front-end and a bundle conditioning or pose estimation back-end, our network sits directly in the middle—the Deep Intermediate Matcher is the learnable middle end. Figure 1 This paper illustrates feature matching using a deep intermediate-end matcher. Our method establishes point-to-point correspondences between challenging image pairs. It takes readily available local features as input and uses an attention-map neural network to solve the assignment optimization problem. The deep intermediate-end matcher acts as an intermediate and gracefully handles partial point visibility and occlusion, producing a partial assignment matrix.

[0049] In this work, learning feature matching is viewed as finding partial assignments between two sets of local features. We revisit the classic graph-based matching strategy by solving the linear assignment problem, which can be discriminatively solved when relaxed to the optimal transfer problem [see References 50, 9, 31 below]. The cost function for this optimization is predicted by a graph neural network (GNN). Inspired by the success of the Transformer [see Reference 48 below], it uses ego (intra-image) and cross (inter-image) attention to leverage the spatial relationships and visual appearance of keypoints. This formulation reinforces the predicted assignment structure while learning sophisticated priors, gracefully handling occluded and non-repeatable keypoints. Our approach is end-to-end training from image to correspondence—we learn priors for pose estimation from a large, annotated dataset, enabling a deep intermediate matcher to infer 3D scenes and assignments. Our work can be applied to a variety of multi-view geometry problems requiring high-quality feature correspondences.

[0050] We demonstrate the superiority of the deep intermediate-end matcher over both the handmade matcher and the learned interior point classifier. Figure 2 The diagram shows the estimated correspondences for two challenging indoor image pairs obtained by the deep mid-end matcher. The deep mid-end matcher successfully estimates the accurate pose, while other learned or hand-crafted methods fail (correct correspondences in green). When combined with the deep front-end SuperPoint [see Reference 14 below], the proposed method delivers the most substantial improvements, advancing state-of-the-art techniques for homography estimation and indoor / outdoor pose estimation tasks, and paving the way for deep SLAM.

[0051] 2. Related work

[0052] Local feature matching is typically performed by: i) detecting interest points, ii) computing visual descriptors, iii) matching these with a nearest neighbor (NN) search, iv) filtering out incorrect matches, and finally v) estimating the geometric transformation. Classical pipelines developed in the 2000s are typically based on SIFT [see reference 25 below], filtering matches with Lowe's ratio test [see reference 25 below], cross-checking, and heuristic algorithms such as neighborhood consensus [see references 46, 8, 5, 40 below], and employing robust solvers such as RANSAC to find the transformation [see references 17, 35 below].

[0053] Recent work on deep learning for matching has generally focused on using convolutional neural networks (CNNs) to learn better sparse detectors and local descriptors from data [see references 14, 15, 29, 37, 54 below]. To improve their discriminative power, some works have explicitly focused on broader context using region features [see reference 26 below] or log-polar coordinate patches [see reference 16 below]. Other methods learn to filter matches by classifying them as inliers and outliers [see references 27, 36, 6, 56 below]. These operate on the set of matches still estimated by the NN search and therefore ignore assignment structure and discard visual information. To date, practical work on learning matches has mainly focused on dense matching [see reference 38 below] or 3D point clouds [see reference 52 below] and still exhibits this limitation. In contrast, our learnable intermediate performs context aggregation, matching, and filtering simultaneously in a single end-to-end architecture.

[0054] Graph matching is often formulated as a quadratic assignment problem, which is NP-hard and requires expensive, complex, and therefore impractical solvers [see reference 24 below]. For local features, computer vision literature from the 2000s [see references 4, 21, 45 below] uses handcrafted costs with many heuristics, making it complex and fragile. Caetano et al. [see reference 7 below] learned optimized costs for simpler linear assignments, but only using shallow models, while our deep intermediate-end matcher uses neural networks to learn flexible costs. Related to graph matching is the optimal transport problem [see reference 50 below]—a generalized linear assignment with efficient and simple approximate solutions, namely the Sinkhorn algorithm [see references 43, 9, 31 below].

[0055] Deep learning of ensembles such as point clouds aims to design permutation-equivariant or invariant functions by aggregating information across elements. Some works treat all these problems equally through global pooling [see references 55, 32, 11 below] or instance normalization [see references 47, 27, 26 below], while others focus on local neighborhoods in coordinate or feature spaces [see references 33, 53 below]. Attention [see references 48, 51, 49, 20 below] can perform global and data-dependent local aggregations by focusing on specific elements and attributes, and is therefore more flexible. Our work takes advantage of the fact that it can be viewed as a specific instance of a message-passing graph neural network on a complete graph [see references 18, 3 below]. By applying attention to a multi-sided or multi-way graph, similar to [see references 22, 57 below], a deep intermediate matcher can learn complex reasoning about two sets of local features.

[0056] 3. Deep Intermediate Matcher Architecture

[0057] Motivation: In image matching problems, we can leverage certain regularities of the world: the 3D world is largely smooth and sometimes planar; if the scene is static, all correspondences for a given pair of images arise from a single epipolar transform; and some poses are more likely than others. Furthermore, 2D keypoints are often projections of salient 3D points, such as corner points or blobs, so correspondences across images must adhere to certain physical constraints: i) a keypoint can have at most a single correspondence in another image; and ii) some keypoints will not match due to detector occlusion and faults. An effective feature matching model should aim to find all correspondences between reprojections of the same 3D points and identify keypoints that do not match. Figure 3 This demonstrates how we formulate a deep intermediate-end matcher to solve an optimization problem, with its cost predicted by a deep neural network. The deep intermediate-end matcher consists of two main parts: an attention map neural network (Section 3a) and an optimal matching layer (Section 3b). The first component uses a keypoint encoder to map keypoint locations p and their visual descriptors d into a single vector, and then uses alternating self-attention layers and cross-attention layers (repeated L times) to create a more robust representation f. The optimal matching layer creates an M×N score matrix, augments it with a garbage bin, and then uses the Sinkhorn algorithm (for T iterations) to find the optimal part assignment. This alleviates the need for domain expertise and heuristics—we learn relevant priors directly from the data.

[0058] Formulation: Consider two images A and B, each with a set of keypoint locations p and associated visual descriptors d—we jointly call these local features (p, d). The keypoints consist of x and y image coordinates and detection confidence c, where p... i :=(x,y,c) i Visual descriptor d i ∈R D These can be visual descriptors extracted by CNNs such as SuperPoint or traditional descriptors such as SIFT. Images A and B have M and N local features, respectively, and their keypoint index sets are A:={1,...,M} and B:={1,...,N}.

[0059] Partial Assignment: Constraints i) and ii) imply that the correspondences originate from partial assignments between two sets of keypoints. For integration into downstream tasks and better interpretability, each possible correspondence should have a confidence value. Therefore, we define the partial soft assignment matrix P∈[0,1]. M×N Defined as:

[0060] P1N ≤1 M and

[0061] Our goal is to design a neural network that can predict assignment P from two sets of local features.

[0062] 3.1. Attention Graph Neural Network

[0063] The first major block of the deep intermediate matcher (see Section 3a) is an attention map neural network, which works as follows: given initial local features, it computes matching descriptors by allowing the features to communicate with each other. Long-range feature communication is crucial for robust matching and requires the aggregation of information from within the image and across image pairs.

[0064] Intuitively, unique information about a given keypoint depends on its visual appearance and location, as well as its spatial and visual relationships relative to other commonly visible keypoints (e.g., adjacent or prominent keypoints). On the other hand, knowledge of keypoints in a second image can help resolve ambiguity by comparing candidate matches or estimating relative luminosity or geometric transformations from global and explicit cues.

[0065] When asked to match given fuzzy keypoints, humans look back and forth between two images: they sift through tentative keypoint matches, examine each one, and look for contextual cues that help distinguish true matches from other self-similarity. This suggests that attention can be focused on an iterative process at specific locations.

[0066] Keypoint encoder: Initial representation of each keypoint i (0) x i This combines visual appearance and location. We employ a multilayer perceptron (MLP) to embed keypoint locations into high-dimensional vectors, as follows:

[0067] (0) x i =d i +MLP(p i (2)

[0068] The encoder allows the network to jointly reason about appearance and position (which is particularly powerful for attention mechanisms) and is an instance of the “position encoder” introduced in Transformer [see reference 48 below].

[0069] Multipath Graph Neural Network: We consider a single complete graph where nodes are key points between two images. This graph has two types of undirected edges—it is a multipath graph. Image-internal edges or edges from within the image. self Connect keypoint i to all other keypoints within the same image. Image-to-image edges or intersections E. crossKeypoint i is connected to all keypoints in another image. We use a message-passing formula [see references 18, 3 below] to propagate information along both types of edges. The resulting multipath graph neural network starts from the high-dimensional state of each node and computes the updated representation at each layer by simultaneously aggregating messages across all given edges of all nodes.

[0070] make (l) x A i This is the intermediate representation of element i in image A at layer l. Message m E→i It is the result of the aggregation of all keypoints {j:(i,j)∈E}, where E∈{E self E cross The remaining message passing updates for all i in A are as follows:

[0071]

[0072] Where [·||·] denotes cascading. Similar updates can be performed simultaneously on all keypoints in image B. A fixed number of layers L with different parameters are linked and alternately aggregated along their own edges and cross edges. Therefore, starting with L=1, if l is odd, then E=E self And if l is even, then E = E cross .

[0073] Attention aggregation: The attention mechanism computes message m E→i And perform aggregation. Self-edges are based on self-attention [see reference 48 below], while cross-edges are based on cross-attention. Similar to database retrieval, for the representation of i, query q. i Based on certain element attributes (keys) j Get the values ​​v of these elements j We calculate the message as a weighted average of the values:

[0074]

[0075] The attention mask α ij It is the Softmax of key query similarity:

[0076]

[0077] The keys, queries, and values ​​are computed as linear projections of the deep features of the graph neural network. Given that the query keypoint i is in image Q, and all source keypoints are in image S, (Q,S)∈{A,B} 2 We can write:

[0078]

[0079]

[0080] Each layer l has its own projection parameters, and they are shared for all keypoints in both images. In practice, we enhance expressiveness through multi-head attention [see reference 48 below].

[0081] Our formula offers maximum flexibility because the network can learn to focus on subsets of key points based on specific attributes. Figure 4 In the middle, mask α ij This is represented as a ray. Attention convergence builds a dynamic graph between keypoints. Self-attention (top) can focus on any location within the same image, such as unique locations, and is therefore not limited to nearby locations. Cross-attention (bottom) focuses on locations in another image, such as potential matches with similar local appearances. Deep intermediate matchers can take or participate based on both appearance and keypoint location, as they represent x... i Encoding in the middle. This involves focusing on nearby keypoints and obtaining the relative positions of similar or prominent keypoints. This enables geometric transformations and assigned representations. The final matching descriptor is a linear projection:

[0082]

[0083] The key points in B are similar.

[0084] 3.2. Optimal Matching Layer

[0085] The second main block of the deep intermediate matcher (see Section 3b) is the optimal matching layer, which produces a partial assignment matrix. Similar to the standard graph matching formula, the assignment P can be calculated by computeding the score matrix S∈R for all possible matches. M×N Maximize the total score under the constraints in Equation 1. This is equivalent to solving the linear assignment problem.

[0086] Score prediction: It would be prohibitive to build separate representations for all (M+1)×(N+1) potential matches. Instead, we express pairwise scores as the similarity of the matching descriptors:

[0087]

[0088] Where <·,·> is the inner product. Unlike the learned visual descriptors, the matching descriptors are not normalized, and their sizes can vary per feature as well as during training to reflect prediction confidence.

[0089] Occlusion and Visibility: To enable the network to suppress occluded keypoints, we augment each set with a trash bin, allowing unmatched keypoints to be explicitly assigned to it. This technique is common in graph matching, and SuperPoint [see Reference 14 below] also uses trash bins to handle potentially undetected image units. We augment the score S- to S- by adding new rows and columns, point-to-bin and bin-to-bin scores, and padding with a single learnable parameter:

[0090]

[0091] Although keypoints in A will be assigned to a single keypoint or trash can in B, each trash can has as many matches as keypoints in the other set: N and M are used for trash cans in A and B, respectively. We denote the expected number of matches for each keypoint and trash can in A and B as... and Expand full allocation Now subject to constraints:

[0092] and

[0093] Sinkhorn Algorithm: The solution to the above optimization problem corresponds to the optimal transfer between discrete distributions a and b with fractions S- [see reference 31 below]. It can be approximated using the Sinkhorn algorithm [see references 43, 9 below], a differentiable version of the Hungarian algorithm [see reference 28 below], classically used for bipartite matching. It solves the regularized transfer problem, naturally leading to soft assignment. This normalization is equivalent to iteratively performing alternating Softmax along rows and columns, and is therefore easily parallelized on a GPU. After T iterations, we discard the bins and restore P = P - 1:M,1:N .

[0094] 3.3. Losses

[0095] By design, both the graph neural network and the optimal matching layer are differentiable—making backpropagation from matching to visual descriptors possible. The deep intermediate matcher matches underlying facts in a supervised manner. Training is performed. These are estimated based on relative transformations of the underlying facts—using pose and depth maps or homography. This also allows us to extract some keypoints if there are no reprojections nearby. and Marked as a mismatch. Given labels, we minimize the assignment P. - Negative log-likelihood:

[0096]

[0097] This supervision aims to maximize both the accuracy and recall of the matching.

[0098] 3.4. Comparison with related work

[0099] Deep intermediate-end matchers compared to internal classifiers [see references 27, 56 below]: Deep intermediate-end matchers benefit from strong inductive bias by being equivariant about complete permutations of both image and local features. They also directly embed commonly used cross-checking constraints into training: probabilities P... i,j Any match with a value greater than 0.5 must be consistent with each other.

[0100] Deep intermediate matcher versus instance normalization [see reference 47 below]: The attention used by the deep intermediate matcher is a more flexible and powerful context aggregation mechanism than instance normalization. It treats all key points equally and has been used in previous feature matching work [see references 27, 56, 26 below].

[0101] Deep intermediate matchers compared to ContextDesc [see reference 26 below]: Deep intermediate matchers can jointly infer appearance and location, while ContextDesc processes them separately. Furthermore, ContextDesc is the front end, requiring an additional, larger region extractor and a loss for keypoint scoring. Deep intermediate matchers only need to learn or manually assign local features and can therefore easily replace existing matchers at any time.

[0102] Deep intermediate-end matcher vs. Transformer [see reference 48 below]: The deep intermediate-end matcher borrows the self-attention of the Transformer, but embeds it into a graph neural network and introduces additional symmetrical cross-attention. This simplifies the architecture and leads to better feature reuse across layers.

[0103] 4. Implementation details

[0104] Deep intermediate matchers can be combined with any local feature detector and descriptor, but they work particularly well with SuperPoint [see reference 14 below], which produces repeatable and sparse keypoints—achieving very efficient matching. Visual descriptors are bilinearly sampled from differentiable semi-dense feature maps. Both local feature extraction and the subsequent “glue” are performed directly on the GPU. At test time, to extract matches from the soft assignment, a confidence threshold can be used to retain some, or all matches and their confidences can simply be used in subsequent steps such as weighted pose estimation.

[0105] Architecture details: All intermediate representations (keys, query values, descriptors) have the same dimension D = 256 as the SuperPoint descriptor. We use L = 9 layers of alternating multi-head self-attention and cross-attention, each layer with 4 heads, and perform T = 100 Sinkhorn iterations in log space for numerical stability. The model is implemented in PyTorch [see reference 30 below] and runs in real time on a GPU: the forward pass takes an average of 150ms (7FPS).

[0106] Training Details: To allow for data augmentation, the SuperPoint detection and description steps are performed as batches on the fly during training. Numerous random keypoints are further added to achieve efficient batch processing and enhanced robustness. More details are provided in Appendix A.

[0107] 5. Experiment

[0108] 5.1. Homography Estimation

[0109] We employ robust (RANSAC) and non-robust (DLT) estimators to perform large-scale homography estimation experiments using real images and synthetic homography.

[0110] Dataset: We generate image pairs by sampling random homography and applying random photometric distortion to real images, following a formula similar to [see references 12, 14, 37, 36 below]. The underlying images are a set of 1 million perturbation images from the Oxford and Paris datasets [see reference 34 below], which are divided into training, validation, and test sets.

[0111] Baseline: We compare the deep intermediate-end matcher with several matchers applied to local features of SuperPoint—nearest neighbor (NN) matchers and various outlier rejectors: mutual checking (or cross-checking), PointCN [see reference 27 below], and Order-Aware Network (OANet) [see reference 56 below]. All learned methods, including the deep intermediate-end matcher, are trained on basic fact correspondences found by projecting keypoints from one image to another. We generate homography and photometric distortion on the fly—an image pair never appears twice during training.

[0112] Metrics: Matching precision (P) and recall (R) are calculated based on the correspondences of the underlying facts. Homography estimation is performed using both RANSAC and the direct linear transform [see reference 19 below] (DLT), the latter having a direct least squares solution. We calculate the average reprojection error at the four corners of the image and report the area under the cumulative error curve (AUC) for up to 10 pixel values.

[0113] Results: The Deep Intermediate Matcher (DIM) is expressive enough to master homography, achieving 98% recall and high precision. Table 1 shows the homography estimations of the DLM, DLT, and RANSAC. The DLM recovers almost all possible matches while suppressing most outliers. Since the DLM correspondence is a high-quality Direct Linear Transform (DLT), it is a least-squares-based solution that outperforms RANSAC without robustness mechanisms. The estimated correspondences are so good that a robust estimator is not required—the DLM works even better than RANSAC on DLT. Outlier rejection methods such as PointCN and OANet fail to predict more accurate matches than the NN matcher itself, relying excessively on the initial descriptor.

[0114]

[0115]

[0116] Table 1

[0117] 5.2. Indoor Attitude Estimation

[0118] Indoor image matching is extremely challenging due to the lack of texture, rich self-similarity, complex 3D geometry of scenes, and large viewpoint variations. As we show below, a deep intermediate matcher can effectively learn priors to overcome these challenges.

[0119] Dataset: We used ScanNet [see Reference 10 below], a large-scale indoor dataset consisting of monocular sequences with base fact pose and depth images, along with well-defined training, validation, and test segmentations corresponding to different scenes. Previous work selected training and evaluation pairs based on temporal difference [see References 29, 13 below] or SfM co-visibility [see References 27, 56, 6 below], typically computed using SIFT. We argue that this limits the difficulty of the pairs, instead selecting them based on the overlap score computed for all possible image pairs in a given sequence using only the base fact pose and depth. This results in a significantly wider baseline of pairs, corresponding to the current frontier in real-world indoor image matching. Discarding pairs with too little or too much overlap, we obtained 230 million training pairs and sampled 1500 test pairs. More details are provided in Appendix A.

[0120] Metrics: As with previous work [see References 27, 56, 6 below], we report the AUC of the pose error at thresholds (5°, 10°, 20°), where the pose error is the angular error of rotation and translation. Relative pose is obtained from the fundamental matrix estimation using RANSAC. We also report the matching accuracy and matching score [see References 14, 54 below], where a match is considered correct based on its epipolar distance.

[0121] Baselines: We evaluate both the deep intermediate matcher and various baseline matchers using root-normalized SIFT [see References 25.2 below] and SuperPoint [see Reference 14 below] features. The deep intermediate matcher is trained using correspondences and mismatched keypoints derived from the base fact pose and depth. All baselines are based on nearest neighbor (NN) matchers and possible outlier rejection methods. In the “Handmade” category, we consider simple cross-checking (mutual), ratio testing [see Reference 25 below], descriptor distance thresholding, and the more complex GMS [see Reference 5 below]. The methods in the “Learned” category are PointCN [see Reference 27 below], and its successors OANet [see Reference 56 below] and NG-RANSAC [see Reference 6 below]. We retrain PointCN and OANet on ScanNet for both SuperPoint and SIFT with classification losses using the correctness criteria defined above and their corresponding regression losses. For NG-RANSAC, we use the original training model. We do not include any graph matching methods because they are orders of magnitude slower for the number of keypoints we consider. We report other local features for reference: ORB [see reference 39 below] with GMS, D2-Net [see reference 15 below] and ContextDesc [see reference 26 below] using publicly available training models.

[0122] Results: The deep intermediate matcher achieves significantly higher pose accuracy compared to both hand-crafted and learned matchers. Table 2 shows the wide-baseline indoor pose estimation on ScanNet. We report the AUC, matching score (MS), and accuracy (P) for pose error, all expressed as a percentage of the pose estimation AUC. When applied to both SIFT and SuperPoint, the deep intermediate matcher outperforms all hand-crafted and learned matchers. These benefits are substantial when applied to both SIFT and SuperPoint. Figure 5Indoor and outdoor pose estimations are shown. The Deep Mid-End Matcher significantly improves pose accuracy compared to the state-of-the-art outlier rejection neural network OANet. It exhibits significantly higher accuracy than other learned matchers, demonstrating its superior representational power. It also produces a significantly larger number of correct matches—up to 10 times higher than ratio testing when applied to SIFT—because it operates on all possible matches rather than a limited set of nearest neighbors. SuperPoint and the Deep Mid-End Matcher together achieve state-of-the-art results in indoor pose estimation. They complement each other because repeatable keypoints can estimate a larger number of correct matches, even in very challenging conditions—see [link to documentation]. Figure 2 .

[0123]

[0124]

[0125] Table 2

[0126] exist Figure 6 The diagram illustrates qualitative image matching. We compare the deep intermediate matcher with the nearest neighbor (NN) matcher in three environments, incorporating both hand-crafted and learned outlier rejectors. The deep intermediate matcher consistently estimates more accurate matches (green lines) and fewer mismatches (red lines) to cope with repetitive textures, large viewpoints, and lighting variations.

[0127] 5.3. Outdoor Attitude Estimation

[0128] Since outdoor image sequences present their own set of challenges (e.g., illumination variations and occlusion), we train and evaluate a deep intermediate matcher for pose estimation in outdoor settings. We use the same evaluation metrics and baseline methods as for the indoor pose estimation task.

[0129] Datasets: We evaluate the PhotoTourism dataset, which is part of the CVPR'19 Image Matching Challenge [see Reference 1 below]. It is a subset of the YFCC100M dataset [see Reference 44 below] and has base-fact poses and sparse 3D models obtained from off-the-shelf SfM tools [see References 29, 41, 42 below]. For training, we use the MegaDepth dataset [see Reference 23 below], which also has clean depth maps computed using multi-view stereo. Scenes located in the PhotoTourism test set will be removed from the training set.

[0130] Results: Table 3 shows the outdoor pose estimation results for the PhotoTourism dataset. Matching SuperPoint and SIFT features with the deep intermediate matcher yields significantly higher pose accuracy (AUC), precision (P), and matching score (MS) compared to hand-crafted or other learning methods. When applied to both SuperPoint and SIFT, the deep intermediate matcher outperforms all baselines at all relative pose thresholds. Most notably, the final matching accuracy is very high (84.9%), reinforcing the analogy of the deep intermediate matcher “glue” local features.

[0131]

[0132] Table 3

[0133] 5.4. Understanding Deep Intermediate Matchers

[0134] Ablation Study: To evaluate our design decisions, we repeated the indoor ScanNet experiments, but this time focused on different deep intermediate matcher variants. Table 4 shows the ablation of deep intermediate matchers on ScanNet with Super-Point local features. All deep intermediate matcher blocks are useful and bring considerable performance improvements. The differences relative to the full model are shown. While the optimal matching layer improves the baseline nearest neighbor matcher individually, the GNN explains most of the gains from the deep intermediate matchers. Cross-attention and positional encoding are both crucial for strong cohesion, and deeper networks further improve accuracy.

[0135]

[0136] Table 4

[0137] Visualizing Attention: A complete understanding of the proposed technique would be incomplete without attempting to visualize the attention patterns of the deep intermediate matcher throughout the matching process. The extensive diversity of self-attention and cross-attention patterns in… Figure 7 This is shown in the figure, and reflects the complexity of learning behavior. Figure 7 Visualizing attention: Self-attention and cross-attention mask α at various layers and the head. ij Deep intermediate matchers learn multiple patterns and can focus on global or local context, self-similarity, unique features, and match candidates.

[0138] 6. Conclusion

[0139] In this disclosure, we describe what we call a “Deep Intermediate Matcher”—an attention-mapped neural network inspired by the success of Transformers in NLP—for local feature matching. We argue that the data association component of the 3D reconstruction pipeline has not received sufficient attention from the research community, and a powerful learning-based intermediate is our solution. The Deep Intermediate Matcher enhances the receptive field of local features and downplays features lacking correspondence, effectively performing both the roles of ContextDesc and internal classification. Importantly, the internal workings of the Deep Intermediate Matcher are entirely learned from real-world data. Our results on 2D-to-2D feature matching demonstrate a significant improvement over state-of-the-art techniques.

[0140] Our description here provides a strong case study of using learnable intermediates in feature matching pipelines as an alternative to modern, deep learning-based, hand-designed heuristics. Some of our future work will focus on evaluating deep intermediate matchers in a complete 3D reconstruction pipeline.

[0141] Various exemplary embodiments of the invention are described herein. Reference is made to these examples in a non-limiting manner. They are provided to illustrate aspects of the invention's broader applicability. Various changes may be made to the described invention and equivalents may be substituted without departing from the true spirit and scope of the invention. Furthermore, numerous modifications can be made to adapt particular circumstances, materials, composition, processes, process actions, or steps to the objectives, spirit, or scope of the invention. Moreover, as those skilled in the art will understand, each individual variant described and illustrated herein has discrete components and features that can be readily separated from or combined with features of any of the other embodiments without departing from the scope or spirit of the invention. All such modifications are intended to fall within the scope of the claims associated with this disclosure.

[0142] This invention includes methods that can be performed using the subject device. The method may include actions to provide such a suitable device. This provision can be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on, or otherwise act to provide the necessary device in the subject method. The methods listed herein can be performed in any logically possible order of the listed events, as well as in the order of the listed events.

[0143] The exemplary aspects of the invention, along with details regarding material selection and manufacturing, have been set forth above. As for other details of the invention, these can be understood in conjunction with the patents and publications cited above, as well as what is generally known or understood by those skilled in the art. The same applies to the method-based aspects of the invention, and to the additional actions typically or logically employed.

[0144] Furthermore, while the invention has been described with reference to several examples that optionally incorporate various features, the invention is not limited to what is described or indicated when considering each variation of the invention. Various changes may be made to the described invention and equivalents may be substituted (whether listed herein or not included for brevity) without departing from the true spirit and scope of the invention. Moreover, where numerical ranges are provided, it should be understood that every intermediate value between the upper and lower limits of the range and any other specified or intermediate value within the range is covered by the invention.

[0145] Furthermore, any optional features of the variants of the invention contemplated herein may be set forth and claimed independently, or in combination with any one or more features described herein. References to singular items include the possibility that multiple identical items may exist. More specifically, as used in and related claims, the singular forms “a,” “an,” “the,” and “the” include plural indicators unless expressly stated otherwise. In other words, the use of this document allows for the inclusion of “at least one” subject matter items in the foregoing description and the claims associated with this disclosure. It should be further noted that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended as a preliminary basis for the use of exclusive terms such as “unique,” ​​“only,” etc., when referencing claim elements or using the “negative” limitation.

[0146] Without using such exclusive terminology, the term "comprising" in claims associated with this disclosure shall allow for the inclusion of any additional elements—whether or not a given number of elements are enumerated in such claim, or the addition of a feature may be considered to alter the nature of an element set forth in such claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given as broadly as possible their generally understood meaning while maintaining the validity of the claims.

[0147] The scope of this invention is not limited to the examples and / or subject matter provided, but is limited only to the scope of the language of the claims associated with this disclosure.

[0148] 7. Appendix A - Further Experimental Details

[0149] Homography estimation:

[0150] The test set contains 1024 pairs of 640×480 images. Homography is generated by applying random perspective, scaling, rotation, and translation to the original full-size images to avoid boundary artifacts. We use the 512 highest-scoring keypoints detected by SuperPoint for evaluation, with a non-maximum suppression (NMS) radius of 4 pixels. A correspondence is considered correct if the reprojection error is less than 3 pixels. When estimating homography using RANSAC, we use an OpenCV function with 3000 iterations and an internal threshold of 3 pixels to find homography.

[0151] Indoor attitude estimation:

[0152] After taking into account missing depth values ​​and occlusion (checking for depth consistency using relative error), the overlap score between two images A and B is the average ratio of pixels in A to those in B that are visible (and vice versa). We train and evaluate with an overlap range of 0.4 to 0.8. For training, we sample 200 pairs per scene per epoch, similar to reference

[15] . The test set is generated by sampling sequences 15 times and then sampling 15 pairs for each of 300 sequences. We resize all ScanNet images and depth maps to VGA 640×480. We detect up to 1024 SuperPoint keypoints (using a publicly available training model with an NMS radius of 4) and 2048 SIFT keypoints (using an OpenCV implementation). When calculating precision and matching scores, we use an epipolar threshold of 5.10e-4. Pose is calculated by first estimating the fundamental matrix using OpenCV's findEssentialMat and RANSAC, where the inlier threshold is 1 pixel divided by the average focal length, followed by pose recovery. Compared to previous work [28, 59, 6], we use explicit integration instead of coarse histograms to compute a more accurate AUC.

[0153] Outdoor attitude estimation:

[0154] For training on Megadepth, the overlap score is the ratio of the number of triangulated keypoints visible in two images, as shown in

[15] . We sampled pairs with overlap scores of [0.1, 0.7] at each epoch. For evaluation on the PhotoTourism dataset, we used overlap scores calculated for all 11 scenes and Ono

[30] , selected in the range of [0.1, 0.4]. The images were resized so that their longest side was less than 1600 pixels. We detected 2048 keypoints (NMS radius of 3) for both SIFT and SuperPoint. Other evaluation parameters were the same as those used in the indoor evaluation.

[0155] Training of deep intermediate matchers:

[0156] For training on homography / indoor / outdoor data, we used the Adam optimizer with an initial constant learning rate of 10e-4 for the first 200,000 / 100,000 / 50,000 iterations, then exponentially decaying by 0.999998 / 0.999992 / 0.999992 until iteration 900,000. When using SuperPoint features, we used batches of 512 / 400 / 1024 keypoints per image with 32 / 64 / 16 image pairs. When using SIFT features, we used 1024 keypoints and 24 pairs. Due to the limited number of training scenarios, the outdoor model was initialized using a homography model. Before keypoint encoding, keypoints were normalized using the maximum edge of the image.

[0157] The underlying fact correspondences M and unmatched sets I and J are generated by first calculating the M×N reprojection matrix between all detected keypoints using underlying fact homography or pose and depth maps. A correspondence is a cell with the minimum reprojection error along both rows and columns, below a given threshold: 3 pixels for homography, 5 pixels for indoor, and 3 pixels for outdoor matching. For homography, unmatched keypoints are simply those that do not appear in M. For indoor and outdoor matching, due to depth and pose errors, unmatched keypoints must additionally have a minimum reprojection error greater than 15 pixels and 5 pixels for outdoor matching, respectively. This allows the labels of keypoints with ambiguous correspondences to be ignored, while still providing some supervision through Sinkhorn normalization.

[0158] 8. References

[0159] Each of the following references is incorporated herein by reference in its entirety and cited in the description above:

[0160] [1] Phototourism Challenge, CVPR 2019 Image Matching Workshop. https: / / image matching-workshop.github.io. Accessed November 8, 2019. 7

[0161] [2]Relja and Andrew Zisserman.Three things everyone should know to improve object retrieval.In CVPR,2012.6

[0162] [3]Peter W Battaglia,Jessica B Hamrick,Victor Bapst,Alvaro Sanchez-Gonzalez,Vinicius Zambaldi,Mateusz Malinowski,Andrea Tacchetti,David Raposo,Adam Santoro,Ryan Faulkner,et al.Relational inductive biases,deep learning,and graph networks.arXiv:1806.01261,2018.2,3

[0163] [4]Alexander C Berg,Tamara L Berg,and Jitendra Malik.Shape matchingand object recognition using low distortion correspondences.In CVPR,2005.2

[0164] [5]JiaWang Bian,Wen-Yan Lin,Yasuyuki Matsushita,Sai-Kit Yeung,Tan-DatNguyen,and Ming-Ming Cheng.GMS:Grid-based motion statistics for fast,ultra-robust feature correspondence.In CVPR,2017.2,6

[0165] [6]Eric Brachmann and Carsten Rother.Neural-Guided RANSAC:LearningWhere to Sample Model Hypotheses.In ICCV,2019.2,6

[0166] [7]Tibério S Caetano,Julian J McAuley,Li Cheng,Quoc V Le,and Alex JSmola.Learning graph matching.IEEE TPAMI,31(6):1048–1058,2009.2

[0167] [8]Jan Cech,Jiri Matas,and Michal Perdoch.Efficient sequentialcorrespondence selection by cosegmentation.IEEE TPAMI,32(9):1568–1581,2010.2

[0168] [9]Marco Cuturi.Sinkhorn distances:Lightspeed computation of optimaltransport.In NIPS,2013.1,2,4

[0169]

[10] Angela Dai,Angel X Chang,Manolis Savva,Maciej Halber,ThomasFunkhouser,and Matthias Nieβner.Scannet:Richly-annotated 3d reconstructionsof indoor scenes.In CVPR,2017.6

[0170]

[11] Haowen Deng,Tolga Birdal,and Slobodan Ilic.Ppfnet:Global contextaware local features for robust 3d point matching.In CVPR,2018.2

[0171]

[12] Daniel DeTone,Tomasz Malisiewicz,and Andrew Rabinovich.Deep imagehomography estimation.In RSS Work-shop:Limits and Potentials of Deep Learningin Robotics,2016.5

[0172]

[13] Daniel DeTone,Tomasz Malisiewicz,and Andrew Rabinovich.Self-improving visual odometry.arXiv:1812.03245,2018.6

[0173]

[14] Daniel DeTone,Tomasz Malisiewicz,and AndrewRabinovich.SuperPoint:Self-supervised interest point detection anddescription.In CVPR Workshop on Deep Learning for Visual SLAM,2018.2,4,5,6

[0174]

[15] Mihai Dusmanu,Ignacio Rocco,Tomas Pajdla,Marc Pollefeys,JosefSivic,Akihiko Torii,and Torsten Sattler.D2-net:A trainable cnn for jointdetection and description of local features.In CVPR,2019.2,6

[0175]

[16] Patrick Ebel,Anastasiia Mishchuk,Kwang Moo Yi,Pascal Fua,andEduard Trulls.Beyond cartesian representations for local descriptors.In ICCV,2019.2

[0176]

[17] Martin A Fischler and Robert C Bolles.Random sample consensus:aparadigm for model fitting with applications to image analysis and automatedcartography.Communications of the ACM,24(6):381–395,1981.2

[0177]

[18] Justin Gilmer,Samuel S Schoenholz,Patrick F Riley,Oriol Vinyals,and George E Dahl.Neural message passing for quantum chemistry.In ICML,2017.2,3

[0178]

[19] Richard Hartley and Andrew Zisserman.Multiple view geometry incomputer vision.Cambridge university press,2003.6

[0179]

[20] Juho Lee,Yoonho Lee,Jungtaek Kim,Adam Kosiorek,Seungjin Choi,andYee Whye Teh.Set transformer:A frame-work for attention-based permutation-invariant neural networks.In ICML,2019.2

[0180]

[21] Marius Leordeanu and Martial Hebert.A spectral technique forcorrespondence problems using pairwise constraints.In ICCV,2005.2

[0181]

[22] Yujia Li,Chenjie Gu,Thomas Dullien,Oriol Vinyals,and PushmeetKohli.Graph matching networks for learning the similarity of graph structuredobjects.In ICML,2019.2

[0182]

[23] Zhengqi Li and Noah Snavely.Megadepth:Learning single-view depthprediction from internet photos.In CVPR,2018.7

[0183]

[24] Eliane Maria Loiola,Nair Maria Maia de Abreu,Paulo OswaldoBoaventura-Netto,Peter Hahn,and Tania Querido.A survey for the quadraticassignment problem.European journal of operational research,176(2):657–690,2007.2

[0184]

[25] David G Lowe.Distinctive image features from scale-invariantkeypoints.International Journal of Computer Vision,60(2):91–110,2004.2,6

[0185]

[26] Zixin Luo,Tianwei Shen,Lei Zhou,Jiahui Zhang,Yao Yao,Shiwei Li,Tian Fang,and Long Quan.Contextdesc:Local descriptor augmentation with cross-modality context.In CVPR,2019.2,5,6

[0186]

[27] Kwang Moo Yi,Eduard Trulls,Yuki Ono,Vincent Lepetit,MathieuSalzmann,and Pascal Fua.Learning to find good correspondences.In CVPR,2018.2,5,6

[0187]

[28] James Munkres.Algorithms for the assignment and transportationproblems.Journal of the society for industrial and applied mathematics,5(1):32–38,1957.4

[0188]

[29] Yuki Ono,Eduard Trulls,Pascal Fua,and Kwang Moo Yi.LF-Net:Learning local features from images.In NeurIPS,2018.2,6,7

[0189]

[30] Adam Paszke,Sam Gross,Soumith Chintala,Gregory Chanan,EdwardYang,Zachary DeVito,Zeming Lin,Al-ban Desmaison,Luca Antiga,and AdamLerer.Automatic differentiation in pytorch.In NIPS Workshops,2017.5

[0190]

[31] Gabriel Peyré,Marco Cuturi,et al.Computational optimaltransport.Foundations and Trends in Machine Learning,11(5-6):355–607,2019.1,2,4

[0191]

[32] Charles R Qi,Hao Su,Kaichun Mo,and Leonidas J Guibas.Pointnet:Deep learning on point sets for 3d classification and segmentation.In CVPR,2017.2

[0192]

[33] Charles Ruizhongtai Qi,Li Yi,Hao Su,and Leonidas JGuibas.Pointnet++:Deep hierarchical feature learning on point sets in ametric space.In NIPS,2017.2

[0193]

[34] Filip Ahmet Iscen,Giorgos Tolias,Yannis Avrithis,and Chum.Revisiting Oxford and Paris:Large-scale image retrieval benchmarking.In CVPR,2018.5,9

[0194]

[35] Rahul Raguram,Jan-Michael Frahm,and Marc Pollefeys.A ComparativeAnalysis of Ransac Techniques Leading to Adaptive Real-Time Random SampleConsensus.In ECCV,2008.2

[0195]

[36] René Ranftl and Vladlen Koltun.Deep fundamental matrix estimation.In ECCV,2018.2,5

[0196]

[37] Jerome Revaud,Philippe Weinzaepfel,Ceesar DeSouza,Noah Pion,Gabriela Csurka,Yohann Cabon,and Martin Humenberger.R2D2:Repeatable andReliable Detector and Descriptor.In NeurIPS,2019.2,5

[0197]

[38] Ignatius Rocco,Mircea Cimpoi,Relja Neighborhood consensus networks.In NeurIPS,2018.2

[0198]

[39] Ethan Rublee,Vincent Rabaud,Kurt Konolige,and Gary R Bradski.Orb:An Efficient Alternative to Sifting or Surfing.In ICCV,2011.6

[0199]

[40] Torsten Sattler,Bastian Leibe,and Leif Kobbelt.Scramsac:Improvingransac’s efficiency with a spatial consistency filter.In ICCV,2009.2

[0200]

[41] Johannes Lutz Schonberger and Jan-Michael Frahm.Structure-from-motion revisited.In CVPR,2016.7

[0201]

[42] Johannes Lutz Schonberger,Enliang Zheng,Marc Pollefeys,and Jan-Michael Frahm.Pixelwise view selection for un-structured multi-view stereo.InECCV,2016.7

[0202]

[43] Richard Sinkhorn and Paul Knopp.Concerning nonnegative matricesand doubly stochastic matrices.Pacific Journal of Mathematics,1967.2,4

[0203]

[44] Bart Thomee,David A Shamma,Gerald Friedland,Benjamin Elizalde,Karl Ni,Douglas Poland,Damian Borth,and Li-Jia Li.Yfcc100m:The new data inmultimedia research.Communications of the ACM,59(2):64–73,2016.7

[0204]

[45] Lorenzo Torresani,Vladimir Kolmogorov,and Carsten Rother.Featurecorrespondence via graph matching:Models and global optimization.In ECCV,2008.2

[0205]

[46] Tinne Tuytelaars and Luc J Van Gool.Wide baseline stereo matchingbased on local,affinely invariant regions.In BMVC,2000.2

[0206]

[47] Dmitry Ulyanov,Andrea Vedaldi,and Victor Lempitsky.Instance Normalization:The Missing Ingredient for Fixed Stylization.arXiv:1607.08022,2016.2,5

[0207]

[48] Ashish Vaswani,Noam Shazeer,Niki Parmar,Jacob Uszkoreit,LlionJones,Aidan N Gomez, Kaiser,and Illia Polosukhin.In NIPS,2017.1,2,3,4,5

[0208]

[49] Petar William Cucurull,First Casanova,AdrianaRomero,Pietro and Joshua Bengio.Graph Attention Networks.In ICLR,2018.2

[0209]

[50] Cédric Villani.Optimal Transportation:oldandnew,volume338.SpringerScience&Business Media,2008.1,2

[0210]

[51] Xiaolong Wang,Ross Girshick,Abhinav Gupta,and Kaiming He.Non-local neural networks.In CVPR,2018.2

[0211]

[52] Yue Wang and Justin M Solomon.Deep Closest Point:Learningrepresentations for point cloud registration.In ICCV,2019.2

[0212]

[53] Yue Wang,Yongbin Sun,Ziwei Liu,Sanjay E.Sarma,MichaelM.Bronstein,and Justin M.Solomon.Dynamic Graph CNN for learning on pointclouds.ACM Transactions on Graphics,2019.2

[0213]

[54] Kwang Moo Yi,Eduard Trulls,Vincent Lepetit,and Pascal Fua.Lift:Learned invariant feature transform.In ECCV,2016.2,6

[0214]

[55] Manzil Zaheer,Satwik Kottur,Siamak Ravanbakhsh,Barnabas Poczos,Ruslan R Salakhutdinov,and Alexander J Smola.Deep sets.In NIPS,2017.2

[0215]

[56] Jiahui Zhang,Dawei Sun,Zixin Luo,Anbang Yao,Lei Zhou,TianweiShen,Yurong Chen,Long Quan,and Hon-gen Liao.Learning two-view correspondencesand geometry using order-aware network.In ICCV,2019.2,5,6

[0216]

[57] Li Zhang,Xiangtai Li,Anurag Arnab,Kuiyuan Yang,Yunhai Tong,andPhilip HS Torr.Dual graph convolutional network for semantic segmentation.InBMVC,2019.

Claims

1. A computer system, comprising: Computer-readable media; A processor connected to the computer-readable medium; as well as A set of instructions on the computer-readable medium includes: A deep intermediate matcher architecture, comprising an attention graph neural network and an optimal matching layer, wherein: The attention map neural network has the following characteristics: A keypoint encoder maps keypoint locations p and their visual descriptors d to a single vector, where each of images A and B has a set of keypoint locations p and associated visual descriptors d, and images A and B have M and N local features (p, d), respectively; and Alternating self-attention layers and cross-attention layers, which are repeated L times based on the vector to create a representation f; and The optimal matching layer creates an M×N score matrix from the representation f and finds the optimal part allocation based on the M×N score matrix.

2. The computer system according to claim 1, wherein, In the keypoint encoder, the initial representation of each keypoint i (0) x i Combining visual appearance and location, the corresponding keypoint locations are embedded into a high-dimensional vector with a multilayer perceptron (MLP), as follows: 。 3. The computer system according to claim 2, wherein, The keypoint encoder allows the attention graph neural network to jointly infer appearance and location.

4. The computer system according to claim 1, wherein, The keypoint encoder includes a multipath graph neural network with a single complete graph having nodes that serve as keypoints for two images.

5. The computer system according to claim 4, wherein, The complete graph is a multi-way graph, which has two types of undirected edges: in-image edges E connecting keypoint i to all other keypoints within the same image. self And the inter-image edge E connecting keypoint i to all keypoints in another image. cross The message-passing formula is used to propagate information along both types of edges, so that the resulting multipath graph neural network starts from the high-dimensional state of each node and computes the updated representation at each layer by simultaneously aggregating messages across all given edges of all nodes.

6. The computer system according to claim 5, wherein, if (l) x A i If the keypoint i in image A at layer l is the intermediate representation, then message m E→i It is the result of the aggregation from all keypoints {j : (i, j ) ∈ E}, where E ∈ {E self E cross }, and the remaining message passing for all i in A is updated as follows: ， Where [· || ·] indicates cascading.

7. The computer system according to claim 6, wherein, A fixed number L of layers l with different parameters are linked and alternately aggregated along the inner and inter-image edges, such that starting from l = 1, if l is odd, then E = E self And if l is even, then E = E cross .

8. The computer system according to claim 6, wherein, Alternating self-attention layers and cross-attention layers employ an attention mechanism for computation, which computes the message m. E→i And perform the aggregation, wherein the intra-image edges are based on self-attention and the inter-image edges are based on cross-attention, wherein for representation i, query q i Based on the key k of some elements j The property that retrieves the value v of these elements j The message is then calculated as a weighted average of the values: ， in For attention masking.

9. The computer system according to claim 8, wherein, Attention mask α ij It is a Softmax based on key query similarity: 。 10. The computer system according to claim 8, wherein, The corresponding keys, queries, and values ​​are computed as linear projections of the deep features of the attention map neural network, where the query keypoint i is in image Q and all source keypoints are in image S, (Q, S) ∈ {A, B} 2 In the equation: 。 11. The computer system according to claim 8, wherein, The matching descriptors for the alternating self-attention layer and the cross-attention layer are linear projections: as well as .

12. The computer system according to claim 11, wherein, The optimal matching layer expresses the pairwise scores S of the set as the similarity of matching descriptors: ， Where <·, ·> are inner products, and unlike the learned visual descriptors, the matching descriptors are not normalized, and their sizes can be changed feature-by-feature and during training to reflect prediction confidence.

13. The computer system according to claim 12, wherein, The optimal matching layer for occlusion and visibility suppresses occluded keypoints and augments each set of keypoints with trash bin scores, so that mismatched keypoints are explicitly assigned to trash bin scores.

14. The computer system according to claim 13, wherein, The paired fraction S is expanded by adding new rows and columns. The point-to-bin and box-to-box scores filled with a single learnable parameter are: 。 15. The computer system according to claim 13, wherein, The optimal matching layer uses the Sinkhorn algorithm for T iterations to find the optimal partial allocation based on the M×N score matrix.

16. The computer system according to claim 15, wherein, After T iterations, the optimal matching layer discards the trash can score and restores the probability P = 1:M,1:N ,in It is the original allocation and It is an allocation of the expanded trash can scores.

17. A computer-implemented method system, comprising: A keypoint encoder using an attention graph neural network with a deep intermediate matchinger architecture maps keypoint locations p and their visual descriptors d to a single vector, where each of images A and B has a set of keypoint locations p and associated visual descriptors d, and images A and B have M and N local features (p, d), respectively. as well as The attention graph neural network employing the deep intermediate matcher architecture uses alternating self-attention layers and cross-attention layers, and performs repetition L times based on the vector to create a representation f; as well as The optimal matching layer of the deep intermediate matcher architecture is executed to create an M×N score matrix from the representation f, and the optimal partial assignment is found based on the M×N score matrix.

Images