System and method for virtual and augmented reality

Abstract

To provide a system and a method for virtual and augmented reality.SOLUTION: This invention is related to connected mobile computing systems, methods, and configurations, and more specifically to mobile computing systems, methods, and configurations featuring at least one wearable component which may be utilized for virtual and / or augmented reality operations. The description relates to feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem. The deep middle-end matcher acts as a middle-end and handles partial point visibility and occlusion elegantly, producing a partial assignment matrix.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 935,597, filed November 14, 2019, the entirety of which is incorporated herein by reference in its entirety.

[0002] The present invention relates to connected mobile computing systems, methods, and configurations, and more particularly to mobile computing systems, methods, and configurations featuring at least one wearable component that may be utilized for virtual reality and / or augmented reality operations. [Background technology]

[0003] It is desirable for a mixed reality or augmented reality eyepiece display to be lightweight, low cost, have a small form factor, have a wide virtual image field of view, and be as transparent as possible. In addition, it is desirable to have a configuration that presents virtual image information in multiple focal planes (e.g., two or more) to be practical for a wide variety of use cases without exceeding acceptable tolerances for vergence-divergence and accommodation misalignment. Referring to FIG. 8, an augmented reality system is illustrated that features a head-mounted viewing component (2), a handheld controller component (4), and an interconnected auxiliary computing or controller component (6) that may be configured to be worn as a beltpack or equivalent on a user. Each of these components may be operatively coupled (10, 12, 14, 16, 17, 18) to each other and to other connected resources (8), such as cloud computing or cloud storage resources, via wired or wireless communication configurations, such as those defined by IEEE 802.11, Bluetooth (RTM), and other connectivity standards and configurations. See, for example, U.S. Patent Application Nos. 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 627,155, 62 / 518,539, 16 / 229,532, 16 / 155,564, 15 / 413,284, 16 / 020,541, 62,702,322, 62 / 206,765, 15,597,694, 16 / 221,065, 15 / 968,673, As described in US Pat. Nos. 62 / 682,788, 62 / 899,678, and 62 / 682,788 (each of which is incorporated herein by reference in its entirety), various aspects of such components are described, such as various embodiments of two depicted optical elements (20) through which a user may view the world around them, along with visual components that may be produced by an associated system component, for an augmented reality experience.As illustrated in Figure 8, such systems may also comprise a variety of sensors configured to provide information about the user's surrounding environment, including, but not limited to, various camera-type sensors (22, 24, 26) (such as monochrome, color / RGB, and / or thermal imaging components), a depth camera sensor (28), and / or a sound sensor (30) such as a microphone. There is a need for compact, persistently connected wearable computing systems and assemblies such as those described herein that can be utilized to provide a user with the perception of a rich augmented reality experience. Summary of the Invention [Means for solving the problem]

[0004] This paper describes certain aspects of what may be referred to as a "deep mid-end matcher," a neural network configured to match two sets of local features, both by finding correspondences and eliminating unmatchable points. Such neural network configurations may be utilized in conjunction with spatial computing resources, such as those illustrated in FIG. 8, including, but not limited to, cameras and processing resources that constitute such spatial computing systems. In deep mid-end matcher type configurations, assignments may be estimated by solving an optimal transportation problem, the cost of which is predicted by a graph neural network. We describe an attention-based, flexible context aggregation mechanism that enables the deep mid-end matcher configuration to jointly infer about the underlying 3D scene and feature assignments. Compared to traditional hand-designed heuristics, our technique learns initial values ​​over geometric transformations and regularities of the 3D world through end-to-end training from images to correspondences. The deep mid-end matcher outperforms other learned approaches and brings a new state-of-the-art to the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations can be scaled in real time to modern graphical processing units ("GPUs") and easily integrated into modern structure from motion ("SfM") or simultaneous localization and mapping ("SLAM") systems, all of which may be incorporated into a system such as that illustrated in FIG. 8.

[0005] The present invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium, and a set of instructions on the computer-readable medium. The set of instructions may include a deep mid-end matcher architecture that may include a feature point encoder for mapping feature point locations p and their visual descriptors d into a single vector, an attention graph neural network with alternating self- and cross-attention layers that are iterated L times based on the vector to create a representation f, and an optimal matching layer that creates an M×N score matrix from the representation f and finds an optimal partial assignment based on the M×N score matrix.

[0006] The computer system further includes, in the feature point encoder, generating an initial representation for each feature point i. [ka] However, it may involve combining visual appearance and location, where individual feature point locations are embedded into a high-dimensional vector with a multi-layer perceptron (MLP), as follows: [ka]

[0007] The computer system may further include a feature point encoder that enables the attention graph neural network to infer jointly about appearance and location.

[0008] The computer system may further include, where the feature point encoder includes a multi-graph neural network having a single complete graph with nodes that are feature points of the two images.

[0009] The computer system further determines whether the graph contains two types of undirected edges: intra-image edges (self-edges; E self) and the inter-image edges (cross edges, E cross ), where the resulting multi-graph neural network is a multi-graph that propagates information along both types of edges using a message passing formula such that, for each node, it starts from a high-dimensional state and, at each layer, computes an updated representation by aggregating messages simultaneously across all given edges for all nodes.

[0010] The computer system further comprises: [ka] If is the intermediate representation for element i in image A at layer l, then message m E→i is the result of aggregation from all feature points {j:(i,j)∈E}, where E∈{E self ,E cross}, and the remaining message passing updates for all i in A are: [ka] In the formula, [·||·] may be included to indicate a linkage.

[0011] The computer system further comprises: a fixed number of layers L with different parameters are chained together, starting with l=1, such that, for l odd, E=E self If l is even, then E=E cross , which may include alternating aggregation along self and cross edges, such that

[0012] The computer system further comprises: an alternating self- and cross-attention layer that receives a message m E→i and performing aggregation, the self-edges are calculated based on self-attention and the cross-edges are calculated based on cross-attention. With respect to the representation of i, the query qi is its attribute, key k j Based on the value of some element v j and the message is calculated as a weighted average of the values ​​as follows: [ka]

[0013] The computer system further comprises an attention mask α ij may include being a softmax over the key-query similarity, as follows: [ka]

[0014] The computer system further includes a method for computing the individual keys, queries, and values ​​as linear projections of deep features of a graph neural network, where query feature point i is in image Q and all source feature points are in image S, such that (Q,S)∈{A,B} in the following equation: 2 It may include that. [ka]

[0015] The computer system may further include that the final matching descriptor of the alternating self- and cross-attention layer is a linear projection of: [ka]

[0016] The computer system further comprises: the optimal matching layer expressing the pairwise scores for the sets as similarities of the matching descriptors, as follows: [ka] where <·,·> is the dot product, and in contrast to the learned visual descriptors, the matching descriptors are not normalized and their magnitude may vary per feature, reflecting the prediction confidence during training.

[0017] The computer system may further include augmenting each set of feature points with a dust bin score such that the optimal matching layer suppresses occluded feature points and non-matched feature points are explicitly assigned a dust bin score for occlusion and visibility.

[0018] The computer system further calculates score S by appending points / bins and bin / bin scores that are filled with a single learnable parameter in new rows and columns as follows: - It may include being extended to: [ka]

[0019] The computer system may further include an optimal matching layer that finds an optimal partial assignment based on the M×N score matrix using a Sinkhorn algorithm over T iterations.

[0020] The computer system further determines that after T iterations, the optimal matching layer drops the dustbin score, P=P - 1:M,1:N Restore the [ka] is the original allocation, [ka] be an allocation with dustbin scored extension.

[0021] The present invention also provides a computer-implemented method that may include using a feature point encoder of an attention graph neural network of a deep mid-end matcher architecture to map the feature point locations p and their visual descriptors d into a single vector; running an alternating self- and cross-attention layer of the attention graph neural network of the deep mid-end matcher architecture for L iterations to create a representation f based on the vector; running an optimal matching layer of the attention graph neural network of the deep mid-end matcher architecture to create an M×N score matrix from the representation f, and finding an optimal partial assignment based on the M×N score matrix. The present invention provides, for example, the following items. (Item 1) 1. A computer system comprising: A computer readable medium; a processor coupled to the computer readable medium; A set of instructions on a computer readable medium; Equipped with The set of instructions on the computer readable medium comprises: A deep mid-end matcher architecture, comprising: An attention graph neural network, comprising: a feature encoder for mapping feature locations p and their visual descriptors d into a single vector; an alternating self- and cross-attention layer that is repeated L times to create a representation f based on the vectors; An attention graph neural network having an optimal matching layer, the optimal matching layer creating an M×N score matrix from the representation f and finding an optimal partial assignment based on the M×N score matrix; Deep mid-end matcher architecture, including 2. A computer system comprising: (Item 2) In the feature point encoder, an initial representation for each feature point i is

number

number

number

number

number

number

number

number

number

number

number

number

[0022] The invention will further be described, by way of example only, with reference to the accompanying drawings in which:

[0023] [Figure 1] FIG. 1 is a representative sketch illustrating feature matching using a deep mid-end matcher.

[0024] [Diagram 2] Figure 2 shows the correspondences estimated by a deep mid-end matcher for two challenging indoor image pairs.

[0025] [Diagram 3] FIG. 3 is a representative sketch showing how to formulate a deep mid-end matcher and solve an optimization problem.

[0026] [Figure 4] FIG. 4 is an image showing the mask as a ray.

[0027] [Diagram 5] FIG. 5 is a graph showing indoor and outdoor pose estimation.

[0028] [Figure 6] FIG. 6 illustrates qualitative image matching.

[0029] [Figure 7]FIG. 7 shows the steps for visualizing attention in self- and cross-attention masks at different layers and heads.

[0030] [Figure 8] FIG. 8 shows an augmented reality system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Detailed Description Finding correspondences between points in images is an essential step for computer vision tasks addressing 3D reconstruction or visual localization, such as simultaneous localization and mapping (SLAM) and structure from motion (SfM), which estimate 3D structure and camera pose from such correspondences after matching local features, a process known as data association. Factors such as large viewpoint changes, occlusion, blurring, and lack of texture make 2D / 2D data association particularly challenging.

[0032] In this paper, we present a new way to consider the feature matching problem. Instead of learning better task-independent local features followed by simple matching heuristics and tricks, we propose to learn the matching process from existing local features using a novel neural architecture called the deep mid-end matcher (DMEM). In the context of SLAM, where the problem is typically split into a visual feature detection front-end and a bundle adjustment or pose estimation back-end, our network is directly in the middle, i.e., the deep mid-end matcher is a learnable mid-end. Figure 1 illustrates feature matching using the deep mid-end matcher. Our approach establishes point-wise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attention graph neural network to solve the assignment optimization problem. The deep mid-end matcher acts as a middle-end, gracefully handling partial point visibility and occlusions and producing a partial assignment matrix.

[0033] In this work, learning feature matching is considered as finding partial assignments between two sets of local features. We revisit classical graph-based strategies for matching by solving a linear assignment problem, which can be solved discriminatively when reduced to an optimal transportation problem [see Refs. 50, 9, 31 below]. Our optimization cost function is predicted by a graph neural network (GNN). Inspired by the success of the Transformer [see Ref. 48 below], both the spatial relations of feature points and their visual appearance are exploited, using self (intra-image) and cross (inter-image) attention. Our formulation enforces the assignment structure of the prediction while enabling the cost to learn complex initial values ​​and to gracefully handle occlusions and non-repeatable feature points. Our method is trained end-to-end, from images to correspondences, i.e., learning initial values ​​for pose estimation from a large annotated dataset, enabling a deep mid-end matcher to infer about the 3D scene and the assignments. Our work can be applied to a variety of multi-view geometry problems that require high-quality feature correspondences.

[0034] The superiority of the deep mid-end matcher compared to both the manual matcher and the learned correct correspondence classifier is shown. Figure 2 shows the correspondences estimated by the deep mid-end matcher for two challenging indoor image pairs. The deep mid-end matcher successfully estimates the correct pose, while other trained or manual methods fail (correct correspondences are in green). The proposed method provides the most substantial improvement when combined with a deep front-end, SuperPoint [see Reference 14 below], thereby advancing the state-of-the-art on the tasks of homography estimation and indoor and outdoor pose estimation and paving the way for deep SLAM.

[0035] 2. Related Research

[0036] Local feature matching is typically performed by i) detecting interest points, ii) computing visual descriptors, iii) matching these with a nearest neighbor (NN) search, iv) filtering incorrect matches, and finally, v) estimating the geometric transformation. Classical pipelines developed in the 2000s are often based on SIFT [see Ref. 25 below], filtering matches using heuristics such as Lowe's proportion test [see Ref. 25 below], cross-checks, and neighborhood consensus [see Refs. 46, 8, 5, 40 below], and finding the transformation using robust solvers such as RANSAC [see Refs. 17, 35 below].

[0037] Recent work on deep learning for matching often focuses on learning better sparse detectors and local descriptors [see Refs. 14, 15, 29, 37, 54 below] from data, often using convolutional neural networks (CNNs). To improve their discriminability, some work explicitly looks to the broader context, using region features [see Ref. 26 below] or log-polar patches [see Ref. 16 below]. Other approaches learn to filter matches by classifying them into correct and incorrect correspondences [see Refs. 27, 36, 6, 56 below]. These still operate on the set of matches, estimated by the NN search, thus ignoring the assignment structure and discarding visual information. Learning to actually match, work has so far focused on dense matching [see Ref. 38 below] or 3D point clouds [see Ref. 52 below] and still exhibits such limitations. In contrast, our learnable middle-end simultaneously performs context aggregation, matching, and filtering within a single end-to-end architecture.

[0038] Graph matching problems are usually formulated as quadratic assignment problems, which are NP-hard and require expensive, complex, and therefore impractical solvers [see Reference 24 below]. For local features, computer vision literature from the 2000s [see References 4, 21, 45 below] uses hand-crafted costs with many heuristics, making it complex and brittle. Our deep mid-end matcher uses neural networks to learn flexible costs, while Caetano et al. [see Reference 7 below] learns optimal costs for a simpler linear assignment, but using a shallow model. Related to graph matching is the optimal transportation problem [see Reference 50 below], a generalized linear assignment problem with an efficient but simple approximate solution, namely the sinkhorn algorithm [see References 43, 9, 31 below].

[0039] Deep learning for sets such as point clouds aims to design permutation-equivalent or invariant functions by aggregating information across elements. Some works treat all of them equally through global pooling [see Refs. 55, 32, 11 below] or instance normalization [see Refs. 47, 27, 26 below], while others focus on local neighborhoods in coordinate or feature space [see Refs. 33, 53 below]. Attention [see Refs. 48, 51, 49, 20 below] can implement both global and data-dependent local aggregation by focusing on concrete elements and attributes, and is therefore more flexible. Our work uses the fact that specific instances of message-passing graph neural networks [see Refs. 18, 3 below] can be recognized on complete graphs. By applying attention to multi-edge, i.e., multigraphs, similar to [see Refs. 22, 57 below], a deep mid-end matcher can learn to make complex inferences about two sets of local features.

[0040] 3. Deep Mid-end Matcher Architecture

[0041] Motivation: In image matching problems, some regularities of the world can be exploited. That is, the 3D world is mostly smooth and sometimes planar, all correspondences for a given image pair are derived from a single epipolar transformation if the scene is static, and some poses are more likely than others. In addition, 2D features are usually projections of salient 3D points such as corners or blobs, and therefore correspondences across images must conform to certain physical constraints. That is, i) a feature point can have at most a single correspondence in the other image, and ii) some feature points will not be matched due to occlusions and detector failures. An effective model for feature matching should aim to find all correspondences between reprojections of the same 3D point and identify feature points that do not have a match. Figure 3 shows how the deep mid-end matcher is formulated as a step to solve an optimization problem, the cost of which is predicted by a deep neural network. The deep mid-end matcher includes two main components: an attention graph neural network (Section 3a) and an optimal matching layer (Section 3b). The first component uses a feature point encoder to map feature point locations p and their visual descriptors d into a single vector, then uses an alternating self- and cross-attention layer (repeated L times) to create a more effective representation f. The optimal matching layer creates an M×N score matrix, augments it with dust bins, and then uses a sinkhorn algorithm (over T iterations) to find optimal partial assignments. This alleviates the need for domain expertise and heuristics, i.e., learning relevant initial values ​​directly from the data.

[0042] Formulation: Consider two images A and B, each with a set of visual descriptors d associated with feature point locations p. We call them (p,d) together local features. A feature point is a set of x and y image coordinates and a detection confidence c, i.e., p i :=(x,y,c) iIt consists of visual descriptor d i ∈R D can be extracted by a CNN such as SuperPoint or a traditional descriptor such as SIFT. Images A and B have M and N local features, whose sets of feature point indices are A:={1,...,M} and B:={1,...,N}, respectively.

[0043] Partial assignment: Constraints i) and ii) imply that the correspondences are derived from a partial assignment between the two sets of feature points. For integration into downstream tasks and better interpretability, each possible correspondence should have a certain confidence value. As a result, we define a partial soft assignment matrix P∈[0,1] as follows: M×N Define. [ka]

[0044] Our goal is to design a neural network that predicts the following: the assignment P from two sets of local features.

[0045] 3.1. Attention Graph Neural Network

[0046] The first main block of the deep mid-end matcher (see Section 3a) is an attention graph neural network whose task is to find the following: given initial local features, we find a matching descriptor, f, by connecting the features with each other. i ∈R D Long-range feature connections are essential for robust matching and require the aggregation of information from within images and across image pairs.

[0047] Intuitively, distinct information about a given feature point depends not only on its visual appearance and its location, but also on its spatial and visual relations to other simultaneously visible feature points, e.g., nearby or salient ones. On the other hand, knowledge of the feature point in the second image can help resolve ambiguities by comparing candidate matches or estimating relative photometric or geometric transformations from global and unambiguous cues.

[0048] When asked to match given ambiguous features, humans repeatedly compare both images, selecting tentative matching features, examining each of them, and looking for contextual cues that help to disambiguate the true match from other self-similarities. This suggests an iterative process that can focus attention on specific locations.

[0049] Feature point encoder: the initial representation for each feature point i [ka] combines visual appearance and location. We embed the feature point locations into a high-dimensional vector with a multi-layer perceptron (MLP) as follows: [ka]

[0050] The encoder is an instance of a "position encoder" introduced in Transformers [see Reference 48 below] that allows the network to jointly infer both appearance and position (particularly powerful when used with attention mechanisms).

[0051] Multigraph Neural Network: We consider a single complete graph whose nodes are the feature points of both images. The graph has two types of undirected edges, i.e., it is a multigraph: intra-image edges, i.e., self-edges, E selfconnects feature point i to all other feature points in the same image. Inter-image edges, i.e., cross edges, E cross connects feature point i to all feature points in the other image. We propagate information along both types of edges using a message passing formula [see References 18, 3 below]. The resulting multigraph neural network starts from a high-dimensional state for each node and computes an updated representation at each layer by simultaneously aggregating messages across all given edges for all nodes.

[0052] [ka] Let m be the intermediate representation for element i in image A at layer l. E→i is the result of aggregation from all feature points {j:(i,j)∈E}, where E∈{E self ,E cross}. The remaining message passing updates for all i in A are [ka] where [||] denotes connectivity. Similar updates can be performed for all feature points in image B simultaneously. A fixed number of layers L with different parameters are chained and aggregated alternately along self and cross edges. Thus, starting from l=1, if l is odd, then E=E self If l is even, then E=E cross It is.

[0053] Attention aggregation: The attention mechanism aggregates messages m E→i We compute,and perform aggregation.,Self-edges are based on self-attention [see,Reference 48 below], and cross-edges are based on cross-attention.,Similar to database retrieval, for a representation of,i,, we compute,q,. i is its attribute, key k j Based on the value of some element vj Calculate the message as a weighted average of the values ​​as follows: [ka] In the formula, attention mask α ij is the softmax over the key-query similarity, as follows: [ka]

[0054] The keys, queries, and values ​​are computed as linear projections of the deep features of the graph neural network. Let query feature point i be in image Q, and all source feature points be in image S, such that (Q,S)∈{A,B}. 2 Then, it can be written as follows: [ka]

[0055] Each layer,l,has its own projection parameters, which are shared for all,feature points in both images.,In practice, we use multi-head attention to,improve the representation [see Reference 48 below].

[0056] Our formulation offers maximum flexibility since the network can learn to focus on a subset of feature points based on their specific attributes. In Fig. 4, the masks αij are shown as rays. Attention aggregation builds a dynamic graph between feature points. Self-attention (top) can direct attention to any location in the same image, e.g., a distinctly different location, and is therefore not restricted to nearby locations. Cross-attention (bottom) directs attention to locations in other images that have similar local appearance, such as potential matches. Deep mid-end matchers use a representation x where both appearance and feature point locations are represented. iSince the feature points are encoded in the , they can be retrieved or attended to based on them. This involves attending to nearby feature points and retrieving the relative positions of similar or salient feature points. This enables the representation of geometric transformations and assignments. The final matching descriptor is a linear projection such that [ka] The same is true for feature points in B.

[0057] 3.2. Optimal Matching Layer

[0058] The second main block of the deep mid-end matcher (see Section 3b) is the optimal matching layer, which produces a partial assignment matrix. As in the standard graph matching formulation, the assignment P is the score matrix S ∈ R for all possible matchings. M×N Calculate the total score under the constraints in Equation 1. [ka] This is comparable to solving the linear assignment problem.

[0059] Score Prediction: Building a separate representation for all (M+1)×(N+1) potential matchings would be prohibitive. Instead, we express the pairwise scores as the similarity of the matching descriptors as follows: [ka] where <·,·> is the dot product. In contrast to the learned visual descriptors, the matching descriptors are not normalized and their magnitude may vary per feature, reflecting the prediction confidence during training.

[0060] Occlusion and visibility: To have the network suppress occluded feature points, we extend each set with a dust bin to which unmatched feature points are explicitly assigned. This technique is common in graph matching, and dust bins are also used by SuperPoint [see Reference 14 below] to account for image cells that cannot be detected. We extend the score S to S by appending new rows and columns with point / bins and bin / bin scores filled with a single learnable parameter as follows: - Expand to. [ka]

[0061] A feature point in A will be assigned to a single feature point or dust bin in B, but each dust bin has as many matches as N, M with respect to feature points present in the other set, i.e., the dust bins in A and B, respectively. [ka] Let us denote the number of expected matches for each feature point and dust bin in A and B. The extended allocation P - Here, we have the following constraints: [ka]

[0062] Sinkhorn Algorithm: The solution to the above optimization problem has a score S -This corresponds to the optimal transportation between discrete distributions a and b [see Ref. 31 below] with P = P. This can be approximately solved using the Sinkhorn algorithm [see Refs. 43, 9 below], which is a discriminative version of the Hungarian algorithm [see Ref. 28 below] classically used for bipartite matching. This solves a regularized transportation problem, which inevitably leads to soft assignments. This regularization corresponds to iteratively performing an alternating softmax along the rows and columns, and is therefore easily parallelized on a GPU. After T iterations, we drop the dust bin and fix P = P - 1:M,1:N Restore.

[0063] 3.3.Loss

[0064] By design, both the graph neural network and the optimal matching layer are discriminative, i.e., this enables backpropagation from the matching to the visual descriptors. The deep mid-end matcher performs ground truth matching in a supervised fashion. [ka] These are estimated from ground truth relative transformations, i.e. using pose and depth maps or homographies. This also includes some feature points [ka] Let,P,be labeled as not matched if they do not have any reprojections in their,neighborhood. Given the labels, we assign,P, - Minimize the negative log-likelihood of [ka]

[0065] The present teacher aims to simultaneously maximize the precision and recall of matching.

[0066] 3.4. Comparison with related studies

[0067] Deep mid-end matchers versus positive correspondence classifiers [see refs. 27, 56 below]: The deep mid-end matcher benefits from the strong induced bias by being globally permutation equivariant to both images and local features. It additionally embeds the commonly used cross-check constraint directly into the training, i.e., the probability P i,j Any matches involving are necessarily mutually consistent.

[0068] Deep mid-end matcher vs. instance normalization [see Ref. 47 below]: Attention, as used by deep mid-end matchers, treats all feature points equally and is a more flexible and effective context aggregation mechanism than instance normalization [see Refs. 27, 56, 26 below] used by previous work on feature matching.

[0069] Deep mid-end matchers vs. ContextDesc [see Reference 26 below]: Deep mid-end matchers can infer both appearance and location, while ContextDesc handles them separately. In addition, ContextDesc is a front-end that additionally requires a larger region extractor and loss for feature point scoring. Deep mid-end matchers only require local features to be learned or hand-crafted, and therefore can be a simple drop-in replacement for existing matchers.

[0070] Deep mid-end matcher vs. transformer [see reference 48 below]: The deep mid-end matcher borrows self-attention from the transformer but embeds it into a graph neural network, and additionally introduces cross-attention, which is symmetric. This simplifies the architecture and leads to better feature reuse across layers.

[0071] 4. Implementation details

[0072] The deep mid-end matcher can be combined with any local feature detector and descriptor, but works particularly well with SuperPoint [see Reference 14 below], which produces reproducible and sparse feature points, thus enabling very efficient matching. The visual descriptors are bilinearly sampled from a semi-dense feature map, which is discriminative. Both the local feature extraction and the subsequent "glue" are performed directly on the GPU. At test time, to extract matches from the soft assignments, we can reserve some, using a confidence threshold, or simply use all of them and their confidences in a subsequent step such as weighted pose estimation.

[0073] Architecture details: All intermediate representations (keys, query values, descriptors) have the same dimension D=256 as the SuperPoint descriptor. We perform T=100 SyncHorn iterations in log-space for numerical stability, using L=9 layers of alternating multi-head self- and cross-attention, each with 4 heads. The model is implemented in PyTorch [see Reference 30 below] and run on the GPU in real-time. A forward pass takes, on average, 150ms (7 FPS).

[0074] Training Details: To enable data augmentation, the SuperPoint detection and description steps are performed on the fly as batches during training. A few random feature points are also added for efficient batching and increased robustness. Further details are provided in Appendix A.

[0075] 5. Experiment

[0076] 5.1. Homography Estimation

[0077] We conduct large-scale homography estimation experiments using real images and synthetic homographies with both robust (RANSAC) and non-robust (DLT) estimators.

[0078] Dataset: Following a recipe similar to [see References 12, 14, 37, 36 below], we generate image pairs by sampling random homographies and applying random photometric distortions to real images. The basis images come from a set of 1 million disturbed images in the Oxford and Paris dataset [see Reference 34 below], which are split into training, validation, and test sets.

[0079] Baselines: We compare the deep mid-end matcher against several matchers applied to SuperPoint local features, namely a nearest neighbor (NN) matcher and various miscorrespondence rejecters, namely mutual check (or crosscheck), PointCN [see ref. 27 below], and order-aware network (OANet) [see ref. 56 below]. All learned methods, including the deep mid-end matcher, are trained on ground truth correspondences, which are found by projecting feature points from one image to the other. We generate homographies and photometric distortions on the fly, i.e., an image pair is never seen twice during training.

[0080] Metrics: Matching precision (P) and recall (R) are calculated from the ground truth correspondences. Homography estimation is performed using both RANSAC and Direct Linear Transformation (DLT) [see Reference 19 below], which has a direct least squares solution. We calculate the average reprojection error of the four corners of the image and report the area under the cumulative error curve (AUC) for the maximum 10 pixel values.

[0081] Results: The deep mid-end matcher is expressive enough to master the homography, achieving 98% recall and precision. Table 1 shows the homography estimation for the deep mid-end matcher, DLT, and RANSAC. The deep mid-end matcher recovers almost all possible matchings while suppressing most false correspondences. Direct linear transformation (DLT), a least-squares based solution without robustness mechanisms, outperforms RANSAC because the deep mid-end matcher correspondences are of high quality. The estimated correspondences are good enough that a robust estimator is not required, i.e., the deep mid-end matcher works even better with DLT than with RANSAC. False correspondence rejection methods such as PointCN and OANet cannot predict the correct matching better than the NN matcher itself and rely too heavily on the initial descriptor. [Table 1]

[0082] 5.2. Indoor Pose Estimation

[0083] Indoor image matching is very challenging due to the lack of texture, numerous self-similarities, complex 3D geometric scenes, and large viewpoint changes. As shown below, a deep mid-end matcher can effectively learn initialization and overcome these challenges.

[0084] Dataset: We use ScanNet [see Reference 10 below], a large-scale indoor dataset consisting of monocular sequences with ground truth pose and depth images, and a well-defined training, validation, and test split corresponding to different scenes. Previous works select training and evaluation pairs based on time difference [see References 29, 13 below] or SfM co-visibility [see References 27, 56, 6 below], usually calculated using SIFT. We argue that this limits the difficulty of the pairs, and instead select them based on overlap scores calculated for every possible image pair in a given sequence, using only ground truth poses and depths. This results in a significantly broader range of baseline pairs, which corresponds to the current state-of-the-art for real-world indoor image matching. We obtain 230 million training pairs by discarding pairs with too small or too large overlaps, and sample 1,500 test pairs. Further details are provided in Appendix A.

[0085] Metrics: As in previous work [27, 56, 6 below], we report the AUC of the pose error at thresholds (5·, 10·, 20·), where the pose error is the maximum of the angular errors in rotation and translation. The relative pose is obtained from the fundamental matrix estimation using RANSAC. We also report the matching accuracy and matching score [14, 54 below], where a match is considered correct based on its epipolar distance.

[0086] Baselines: We evaluate the deep mid-end matcher and various baseline matchers using both square-root normalized SIFT [see Ref. 25, below, 2] and SuperPoint [see Ref. 14, below] features. The deep mid-end matcher is trained on corresponding and unmatched feature points derived from ground truth pose and depth. All baselines are based on nearest neighbor (NN) matchers and potential mismatch rejection methods. In the "manual" category, we consider simple cross-checks, ratio tests [see Ref. 25, below], descriptor distance thresholds, and the more complex GMS [see Ref. 5, below]. Methods in the "trained" category are PointCN [see Ref. 27, below] and its follow-ups OANet [see Ref. 56, below] and NG-RANSAC [see Ref. 6, below]. Using the accuracy criteria defined above and their respective regression losses, we retrain PointCN and OANet with classification losses on ScanNet for both SuperPoint and SIFT. For NG-RANSAC, we use the original trained models. We do not include any graph matching methods, as they slow down by several orders of magnitude for the number of feature points considered. For reference, other local feature methods using publicly available trained models are also reported: ORB with GMS [see Reference 39 below], D2-Net [see Reference 15 below], and ContextDesc [see Reference 26 below].

[0087] Results: The deep mid-end matcher enables significantly higher pose accuracy compared to both the hand-crafted and trained matchers. Table 2 shows the wide-baseline indoor pose estimation on ScanNet. The AUC of pose error, matching score (MS), and precision (P) are all reported in pose estimation AUC percentage. The deep mid-end matcher outperforms all hand-crafted and trained matchers when applied to both SIFT and SuperPoint. These advantages are substantial when applied to both SIFT and SuperPoint. Figure 5 shows indoor and outdoor pose estimation. The deep mid-end matcher significantly improves pose accuracy over OANet, a state-of-the-art false-match rejection neural network. It has significantly higher accuracy than other trained matchers, demonstrating its higher expressive power. It also produces up to 10 times more correct matches than the ratio test when applied to SIFT because it operates on the full set of possible matches rather than a limited set of nearest neighbors. Both SuperPoint and the deep mid-end matcher achieve state-of-the-art results in indoor pose estimation. They complement each other well because repeatable feature points allow us to estimate a larger number of correct matches even in very challenging situations (see Figure 2). [Table 2]

[0088] Figure 6 shows qualitative image matching. We compare the deep mid-end matcher with a nearest neighbor (NN) matcher with two mismatch rejecters, one hand-crafted and one trained, in three environments. The deep mid-end matcher consistently estimates more correct matches (green line) and fewer mismatches (red line), surviving repeated texture, large viewpoint, and lighting changes.

[0089] 5.3. Outdoor Pose Estimation

[0090] Because outdoor image sequences present their own set of challenges (e.g., lighting variations and occlusions), we train and evaluate a deep mid-end matcher for pose estimation in outdoor settings, using the same evaluation metric and baseline method as in the indoor pose estimation task.

[0091] Dataset: We evaluate on the PhotoTourism dataset, which is part of the CVPR'19 Image Matching Challenge [see Reference 1 below]. It is a subset of the YFCC100M dataset [see Reference 44 below], with ground truth poses and sparse 3D models obtained from off-the-shelf SfM tools [see References 29, 41, 42 below]. For training, we use the MegaDepth dataset [see Reference 23 below], which also has clean depth maps computed using multi-view stereo. Scenes that are in the PhotoTourism test set are removed from the training set.

[0092] Results: Table 3 shows outdoor pose estimation on the PhotoTourism dataset. Matching SuperPoint and SIFT features with the deep mid-end matcher results in significantly higher pose accuracy (AUC), precision (P), and matching score (MS) than manual or other learned methods. The deep mid-end matcher outperforms all baselines at all relative pose thresholds when applied to both SuperPoint and SIFT. Most notably, the accuracy of the resulting matching is very high (84.9%), as the deep mid-end matcher reinforces similarities by "gluing" local features together. [Table 3]

[0093] 5.4. Understanding Deep Mid-End Matchers

[0094] Ablation Study: To evaluate our design decisions, we repeat the indoor ScanNet experiments, but this time focusing on different deep mid-end matcher variants. Table 4 shows the ablation of the deep mid-end matcher on ScanNet with SuperPoint local features. All deep mid-end matcher blocks are useful and provide substantial performance gains. Differences relative to the full model are shown. The optimal matching layer alone improves over the baseline nearest neighbor matcher, but the GNN accounts for most of the gain provided by the deep mid-end matcher. Both cross-attention and position encoding are important for effective gluing, and deeper networks further improve accuracy. [Table 4]

[0095] Attention Visualization: Understanding the proposed technique would not be complete without attempting to visualize the attention patterns of the deep mid-end matcher throughout the matching. The wide diversity of self- and cross-attention patterns is shown in Fig. 7, reflecting the complexity of the learned behavior. Fig. 7 shows the visualization of attention, i.e., self- and cross-attention masks αij at different layers and heads. The deep mid-end matcher can learn the diversity of patterns and focus on global or local contexts, self-similar, distinct features, and matching candidates.

[0096] 6. Conclusion

[0097] In this disclosure, we have described what we call the "deep mid-end matcher", an attention graph neural network inspired by the success of transformers in NLP, for local feature matching. We believe that the data association component of the 3D reconstruction pipeline has not received the attention it deserves from the research community, and an effective learning-based mid-end is our solution. The deep mid-end matcher enhances the receiving field of local features, deemphasizing features whose correspondences are missing, and effectively performs both the roles of ContextDesc and positive correspondence classification. Importantly, the internal structure of the deep mid-end matcher is learned globally from real-world data. Our results on 2D / 2D feature matching show significant improvements over the existing state-of-the-art.

[0098] Our discussion here is a strong case for the use of a learnable middle-end in a modern deep learning-based feature matching pipeline as an alternative to hand-designed heuristics. Part of our future work will focus on evaluating deep middle-end matchers inside a full 3D reconstruction pipeline.

[0099] Various exemplary embodiments of the present invention are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the more broadly applicable aspects of the present invention. Various modifications may be made to the described invention, and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process acts, or steps to the objective, spirit, or scope of the present invention. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0100] The present invention includes methods that may be practiced using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, launch, or otherwise act to provide the device needed in the subject method. The methods recited herein may be carried out in any order of the recited events, and in the recited order of events, that is logically possible.

[0101] Exemplary aspects of the invention, together with details regarding material selection and manufacture, have been described above. As for other details of the invention, these can be understood in conjunction with the patents and published documents referenced above, and generally understood or understood by those skilled in the art. The same can be true with respect to the method-based aspects of the invention in terms of additional acts as generally or theoretically adopted.

[0102] In addition, although the present invention has been described with reference to several embodiments that optionally incorporate various features, the present invention should not be limited to those described or shown as possible for each variation of the present invention. Various modifications and equivalents that can be made to the described invention (whether listed herein or not included for some brevity) can be substituted without departing from the true spirit and scope of the present invention. In addition, when a range of values ​​is provided, it is to be understood that each intervening value between the upper and lower limits of that range and any other stated or intervening value in that stated range is encompassed within the scope of the present invention.

[0103] It is also envisioned that any optional feature of the described inventive variation may be described or claimed independently or in combination with any one or more of the features described herein. Reference to a single object includes the possibility that there are multiple identical objects present. More specifically, as used in this specification and the claims associated herewith, the singular forms "a," "an," "said," and "the" include multiple supports unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the subject article in the above description and in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this description is intended to serve as a prelude for the use of such exclusive terms, such as "solely," "only," and equivalents, or the use of "negative" limitations in connection with the recitation of claim elements.

[0104] Without the use of such exclusive terms, the term "comprising" in claims associated with this disclosure shall permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be viewed as changing the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given their broadest possible, commonly understood meaning while maintaining claim legitimacy.

[0105] The scope of the present invention should not be limited to the examples and / or subject specification provided, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0106] 7. Appendix A - Further Experimental Details

[0107] Homography Estimation:

[0108] The test set contains 1,024 pairs of 640x480 images. Homographies are generated by applying random perspective, scaling, rotation, and translation to the original full-size images to avoid boundary artifacts. We evaluate the 512 top-scoring feature points detected by SuperPoint with a non-maximum suppression (NMS) radius of 4 pixels. Correspondences are considered correct if they have a reprojection error lower than 3 pixels. When estimating homographies with RANSAC, we use the opencv function findHomography with 3,000 iterations and a correct correspondence threshold of 3 pixels.

[0109] Indoor pose estimation:

[0110] The overlap score between two images A and B is the average ratio of pixels in A that are visible in B (and vice versa) after taking into account missing depth values ​​and occlusions (by checking for consistency within depth using relative error). We train and evaluate using an overlap range of 0.4 to 0.8. For training, we sample 200 pairs per scene at each reference time point, similar to that in

[15] . The test set is generated by subsampling the sequences by 15, followed by sampling 15 pairs per 300 sequences. We resize all ScanNet images and depth maps to VGA 640x480. We detect up to 1,024 SuperPoint features (using a publicly available trained model with an NMS radius of 4) and 2,048 SIFT features (using the OpenCv implementation). We use an epipolar threshold of 5.10e-4 when calculating accuracy and matching scores. Pose segmentation is computed by first estimating the fundamental matrix using OpenCv's findEssentialMat and RANSAC, followed by recoverPose, with a 1-pixel correct correspondence threshold divided by the average focal length. In contrast to previous works [28, 59, 6], we use explicit integration rather than a rough histogram to compute a more accurate AUC.

[0111] Outdoor Pose Estimation:

[0112] For training on Megadepth, the overlap score is the proportion of triangulated feature points visible in the two images, as in

[15] . At each reference timepoint, we sample pairs with overlap scores within [0.1,0.7]. For evaluation on the PhotoTourism dataset, we use all 11 scenes and the overlap scores calculated by Ono

[30] , with a selection range of [0.1,0.4]. Images are resized so that their longest edge is smaller than 1,600 pixels. For both SIFT and SuperPoint, we detect 2,048 feature points (with an NMS radius of 3). Other evaluation parameters are the same as those used in the indoor evaluation.

[0113] Training a deep middle-end matcher:

[0114] First 200K / 100K / 50K runs to train on homography / indoor / outdoor data Over a number of iterations, with an initial constant learning rate of 10e-4, followed by 900,000 iterations. We use the Adam optimizer followed by an exponential decay of 0.999998 / 0.999992 / 0.999992 until the number of iterations. When using SuperPoint features, we employ batches with 32 / 64 / 16 image pairs and a fixed number of feature points per image: 512 / 400 / 1,024. When using SIFT features, we use 1,024 feature points and 24 pairs. Due to the limited number of training scenes, the outdoor model is initialized with a homography model. Prior to feature encoding, feature points are normalized by the maximum edge of the image.

[0115] The ground truth correspondences M and unmatched sets I and J are generated by first computing an M×N reprojection matrix between all detected feature points using the ground truth homography or pose and depth maps. Correspondences are cells along both rows and columns that have the smallest reprojection error below a given threshold, i.e., 3, 5, and 3 pixels for homography, indoor, and outdoor matching, respectively. For homography, unmatched feature points are simply those that do not appear in M. For indoor and outdoor matching, due to errors in depth and pose, unmatched feature points must additionally have a minimum reprojection error greater than 15 and 5 pixels, respectively. This allows us to ignore indications for feature points whose correspondences are ambiguous, while still providing some supervision through synchorn regularization.

[0116] 8.References

[0117] Each of the following references is incorporated herein by reference in its entirety and is referenced in the above description. [Table 5-1] [Table 5-2] [Table 5-3] [Table 5-4] [Table 5-5] [Table 5-6]

Claims

[Claim 1] The invention described in this specification.

Images