Context matching

Abstract

The present disclosure relates to context matching. Feature descriptor matching is rephrased as a graph matching problem. Key points from a query image and a reference image are initially matched and filtered based on the matching. For a given key point, a feature graph is constructed based on neighboring key points around the given key point. The feature graph is compared to a corresponding feature graph of the reference image for the matching key point. Based on the comparison, relocalization data is obtained.

Description

Background Technology

[0001] Computer vision typically relies on the identification of key points in images, from which feature descriptors can be determined to obtain information about a scene. Typically, a descriptor network, trained to recognize feature descriptors from reference images, can be used to determine these descriptors. For example, images of a scene can be captured at different times or when the scene has different characteristics. These images can then be used to identify key points, which can be used for detection, image retrieval, localization, and other applications.

[0002] Searching for good feature correspondences is a fundamental step in tasks such as visual relocalization. However, mismatches are often unavoidable because local features can encounter visual ambiguity. An improved technique is needed for matching features across the entire image. Attached Figure Description

[0003] Figures 1A to 1B The techniques for keypoint matching according to one or more implementation schemes are illustrated in flowchart form.

[0004] Figure 2 A flowchart of a technique for keypoint matching according to one or more implementation schemes is shown.

[0005] Figure 3 A flowchart is shown of a technique for refining an initial set of keypoint matching according to one or more implementation schemes.

[0006] Figure 4 A flowchart illustrating a technique for comparing key point matching contexts according to one or more additional embodiments is shown.

[0007] Figure 5 A simplified system diagram for keypoint matching according to one or more implementation schemes is shown in block diagram form.

[0008] Figure 6 A computer system according to one or more implementation schemes is shown in block diagram form. Detailed Implementation

[0009] This disclosure relates to systems, methods, and computer-readable media for identifying keypoint matching based on contextual information. Generally, keypoint descriptors are determined for keypoints. These keypoint descriptors can describe visual descriptors and / or deep learning descriptors. In some embodiments described herein, keypoint descriptors can be obtained by applying a trained network to image data of keypoints. The trained network can be trained to produce a binarized feature descriptor. This binarized feature descriptor can be used to identify keypoint matching in a reference image.

[0010] In some implementations, constraints may be applied to the feature matching process to address matching uncertainty. In some implementations, a distance ratio test may be applied, which imposes a margin between the nearest neighbor and the remaining neighbors. Furthermore, the matching can be further refined by removing mismatched feature points or feature points with more than one match. For each remaining keypoint node, a graph including the spatial representation of that keypoint and its spatial neighbors can be generated. This graph can be compared to the corresponding graph of the object keypoint in each reference image to determine the graph distance for that particular reference image. This graph distance may be based on a visual and / or geometric comparison of the graph. In some implementations, keypoints may be filtered out or weighted based on inconsistent nearest neighbors based on this graph distance.

[0011] In some implementations, contextual matching is used in the filtering phase of the relocation technique to filter out feature matches with inconsistent nearest neighbors. For a single feature pair with low match confidence, if consistent neighboring matches exist around each feature, the feature match score should be increased due to the highly similar nearest neighbors. On the other hand, based on the context of a reliable match, similar feature pairs lacking that reliable match are more likely to be false positives and should be assigned a low graph match score.

[0012] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0013] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0014] Figures 1A to 1B A flowchart illustrates a technique for contextual keypoint matching according to one or more implementations. Contextual keypoint matching can be used for, for example, relocalization techniques, image matching, camera pose determination, etc. This flowchart... Figure 1A Starting at point 105, the system obtains a query image 102 and a reference image 104. For example, the query image 102 may be an image captured from a camera at the device requesting location information. In some embodiments, the query image 102 may be associated with depth information or other sensor information that can be used to determine location information. The reference image 104 may be obtained from local storage of the reference image or from network storage that includes the reference image. In some embodiments, the reference image 104 may be associated with the same user profile as the query image 102. The reference image 104 may include an image of a physical environment that is the same as or similar to the physical environment captured by the query image 102. Although a single reference image 104 is shown, it should be understood that multiple reference images may be used to determine location information or otherwise determine location information during a context matching process. In some embodiments, the process may be repeated for each of the multiple reference images. Figures 1A to 1B The steps described herein.

[0015] The flowchart continues at box 110, where the system identifies image patches of first keypoints in reference image 104 and query image 102. In some embodiments, the keypoints may be feature points in query image 102, which can be used to compare the scene depicted in query image 102 with reference image 104. Therefore, corresponding keypoints can be identified in reference image 104. Each keypoint may be associated with an image patch including image data surrounding the keypoint, from which a description of the keypoint can be generated. Thus, query image 102 may include image patch 106, while reference image 104 may include a keypoint region from which a corresponding reference descriptor can be determined. The flowchart continues at box 115, where the system identifies keypoint matching. Thus, as shown, keypoint 112 of query image 102 is compared with keypoint 114 of reference image 104. In some embodiments, feature descriptors of the query image keypoints and feature descriptors of the reference image keypoints are compared to determine a feature distance. In some embodiments, feature vectors can be obtained for both object keypoints and reference keypoints, and a vector distance can be determined from the two feature vectors.

[0016] Continue to refer to Figure 1B The flowchart continues at box 120, where the system identifies the matching of nearest neighbor keypoints within the query image and corresponding reference keypoints in the reference image. In some embodiments, nearest neighbors are selected by identifying feature points spatially closest to the object keypoint in the image. In some embodiments, a predetermined number of feature points closest to the object feature point can be selected. Alternatively or additionally, a portion of the image surrounding the object keypoint can be used to identify spatially close neighbor keypoints.

[0017] The flowchart continues at box 125, where the system reduces neighbor keypoints to remove mismatched keypoints, resulting in a reduced set of matching keypoints 128. In some implementations, only neighbor keypoints with one-to-one matches in the reference image may be considered. In some implementations, a distance ratio test may be additionally or alternatively applied to the keypoints, which imposes a margin between the nearest neighbor and the remaining neighbors in terms of feature similarity, reducing the features considered to those most likely to result in a strong match. While keypoint reduction is described herein as occurring after the selection of neighbor nodes, in some implementations, keypoint reduction may be performed globally across the entire image or a portion of the image earlier in the context matching process.

[0018] At box 130, a visual comparison is performed between the graph of the query keypoint and the graph of the reference keypoint. The graph of the query keypoint may be generated from object keypoint 112 and neighboring keypoints, which were selected at 120 and retained after reduction of keypoint 128. The graph from the reference keypoint may be generated based on keypoints that match keypoints in the query keypoint graph. Visual comparison 132 may involve determining the similarity between visual features of the query graph and the reference graph. This visual comparison may include, for example, comparing the overall shape of the set of nodes that constitute the graph.

[0019] The flowchart ends at box 135, where, according to some embodiments, the geometric similarity between the query graph and the reference graph is compared. Comparing the geometric similarity between the query graph and the reference graph may include, for example, analyzing the edge nodes of each graph to determine the degree of similarity of edge configurations across the entire graph. In some embodiments, it may be assumed that the relative order of features should be consistent across different viewpoints. According to some embodiments, the results of visual and / or geometric comparisons of the graphs can be used as weighting parameters for determining inliers and outliers. Therefore, feature points with strong contextual comparisons can be weighted more heavily as matches compared to feature points with weak contextual comparisons.

[0020] Figure 2 A flowchart of a technique for keypoint matching according to one or more implementation schemes is shown. Although Figure 2 The various processes shown are illustrated in a specific order, but it should be understood that these processes can be executed in different orders. Furthermore, not all processes may need to be executed to train the descriptor network.

[0021] The flowchart begins at 205, where the system acquires image data captured from a camera. This image data may be captured by a camera from which positioning information (such as camera pose information, position information, etc.) is to be determined. The image data may include, for example, color images, black and white images, etc. Furthermore, in some embodiments, the image data may include depth information or other sensor information from which the system can determine positioning information.

[0022] The flowchart continues at box 210, where the descriptor module uses the query image data from step 205 to determine keypoints. In one or more embodiments, a detector is applied to the image to identify keypoints. This detector may be a feature detector, such as SIFT or SURF, which identifies features or points of interest (i.e., keypoints) in the image.

[0023] The flowchart continues at box 215, where an image patch is cropped. In one or more embodiments, an image patch may be cropped from image data captured at 205 based on keypoints detected at 210. The image patch of the object keypoints may include image data surrounding the keypoints, from which descriptor information can be determined.

[0024] At box 220, a set of keypoint matches is obtained based on the cropped image patches. In some implementations, a ternary loss function is used to determine keypoint matches. Keypoint matches can be determined by comparing the binarized descriptors of the keypoints to identify matching keypoints between the query image and one or more reference images. In some implementations, FREAK, SIFT, etc., can be used to determine keypoint matches. According to some implementations, keypoint matching can be optimized by utilizing binarized descriptors during the matching process to reduce the necessary computational resources. By using a ternary loss function, matched keypoints are only used if the match is strong enough to effectively distinguish them from other candidate keypoints, as will be referred to below. Figure 3 As stated above.

[0025] The flowchart continues at box 225, where a feature map is generated based on the matching of this set of keypoints. The feature map can be generated by identifying one or more neighboring keypoints of a specific keypoint to determine the geometry. In some implementations, the feature map can be generated by identifying the spatial relationships between object keypoints and constrained neighboring keypoints. That is, when generating the feature map, only keypoints with one-to-one matches can be considered.

[0026] At box 230, the system will compare the generated feature map with the feature map used for corresponding feature points from the reference image. In some implementations, for visual matching, spatial features of the map from the query image and the map from the reference image can be considered. Additionally or alternatively, edge similarity can be considered in the case of geometric matching. For example, the relative positions of adjacent keypoints can be compared in the query image and the reference image.

[0027] The flowchart ends at box 235, where the system determines camera pose or other positioning information based on this comparison. According to one or more embodiments, this positioning data can provide descriptor data or other positioning data based on identified keypoints from the query image and reference image.

[0028] Figure 3 A flowchart illustrating a technique for refining an initial set of keypoint matching, according to one or more implementation schemes, is shown. Although Figure 3 The various processes shown are illustrated in a specific order, but it should be understood that these processes can be executed in different orders. Furthermore, not all processes may need to be executed to train the descriptor network.

[0029] The flowchart begins at box 305, where the system obtains its first set of key points. These key points can be compared to the above description. Figure 2 The keypoint is obtained as described in box 210. The keypoint can be determined by using a feature detector that identifies keypoints or feature points in the image. Examples of feature detectors may include a difference of Gaussian (“DoG”) detector or other scalar-invariant keypoint detectors.

[0030] At box 310, the trained network is applied to each image patch of the image, including the keypoints, to obtain a binarized feature vector. That is, the neural network can be trained using image patches from the training image to generate a binarized descriptor. Thus, a binarized feature descriptor is obtained for each keypoint in the image. Alternatively, the feature vector can be obtained in other ways, or it can be generated in a non-binarized format.

[0031] The flowchart continues at box 315, where an initial set of matches is determined between a first set of keypoints from the query image and keypoints in one or more reference images. This initial set of matches can be determined, for example, using a nearest neighbor search to compare the binarized feature vectors of the keypoints from the query image with the binarized feature vectors of the reference images. Alternatively, other methods can be used to determine keypoint matches from feature vectors.

[0032] The flowchart ends at box 320, where a threshold constraint is applied to the nearest neighbor distance between matches to reduce the initial matches to a revised list. In some embodiments, the threshold constraint may be applied globally to feature points on the image, or to feature points within a portion of the image surrounding a given keypoint (such as an image patch), or to feature points within a portion of the image from which a feature map is to be identified. To determine the similarity between feature points, the threshold constraint may include a distance ratio test that applies a threshold between the nearest neighbor distance and the second nearest neighbor distance. Thus, the remaining keypoints for localization are those keypoints determined to be the most likely to produce useful results because they are significantly closer matches to the next nearest candidate keypoint. In some embodiments, the initial matches may be further reduced to remove mismatched feature points or feature points with more than one match. In other words, in some embodiments, feature points associated only with a one-to-one match of a specific reference image may be used for localization purposes.

[0033] Figure 4 A flowchart illustrating a technique for comparing keypoint matching contexts, according to one or more additional embodiments, is shown. Although Figure 4The various processes shown are illustrated in a specific order, but it should be understood that these processes can be executed in different orders. Furthermore, not all processes may need to be executed to train the descriptor network.

[0034] The flowchart begins at box 405, where a set of feature vectors for the image is obtained. As described above, feature vectors for a given keypoint can be obtained. In some embodiments, a portion of the image data, including features (i.e., image patches), can be provided as input to a trained neural network that provides feature descriptors for the keypoints based on the provided image data. According to some embodiments, the neural network can be trained to provide either binary feature vectors as descriptors for the keypoints or to provide non-binary feature vectors.

[0035] The flowchart continues at block 410, where, in some embodiments, the system identifies a portion of the image associated with a feature vector. For example, this portion of the image may include an image patch. In some embodiments, the size of the image portion may be a predetermined size or a user-selectable size. In some embodiments, the size of the image portion may be dynamically determined to encapsulate a predetermined number of feature points, enabling the generation of a feature map.

[0036] At box 415, one or more spatial neighbors are identified in this portion of the image. According to one or more embodiments, spatial neighbors are identified in the query image based on spatial proximity to object feature points. In some embodiments, this feature is optional. Figure 3 The reduced feature set. Spatial neighbors may include, for example, all spatial neighbors within a predetermined distance, a predetermined number of neighbors closest to the feature point, etc. According to some embodiments, the larger the spatial coverage used, the larger the feature map can be and the more statistically significant it is. However, the larger the feature map, the more expensive it becomes to perform the matching. Therefore, the size of the graph and / or the area used to generate the graph can be modified automatically or via user input based on whether resource costs and / or accuracy should be prioritized. At box 420, a feature map is constructed based on one or more spatial neighbors as feature vectors. In some embodiments, the feature map may be the geometric relationship between object keypoints and spatial neighbors.

[0037] The flowchart continues at 425, where a corresponding reference feature map is obtained from the reference image. In some implementations, this reference map can be obtained by finding, for example... Figure 3 The matching reference keypoints and matching reference feature points for each spatial neighbor obtained at position 415 in the query image are used to generate the reference feature map. This reference feature map can indicate the geometric relationship between the corresponding matches of the query keypoints and the query spatial neighbors. In some embodiments, the matching reference keypoints can be located using a lookup table associated with a reference image including the reference keypoints.

[0038] At box 430, the graphical distance between the feature map and the reference feature map is calculated to measure similarity. This graphical distance can be determined in several ways. In some implementations, the graphical distance can be determined based on the matching of the two feature maps. The graphical distance D(G) is used to determine this distance. i G j An exemplary formula for ), where G i =(V i E i ) refers to querying the feature map, and G j This refers to the reference feature map. For the reference match fj extracted at position j in the reference image, a feature map G is constructed. j Node V i This includes the object feature vector fi for key points of the object and its spatial neighbors {f}. m|m≠i In some implementations, {f m It can be retrieved by indexing a lookup table that stores feature matches and their locations. Edge E i Defined according to the self-similarity method. In some implementations, in fi and its neighbors {f} m Calculate the Hamming distance between {d} i，m}

[0039] The flowchart ends at box 435, where weights are assigned to the feature vectors based on graph distance. The two feature maps can be matched based on the following graph distance:

[0040]

[0041] In some implementations, the graph distance is the distance between all nodes such as d. i，j and {d m，n A reweighted combination of the distances between graphics within the given range. d i，j weight Based on the distance {d} within all graphs i，m The relative sorting d between} i，j .also, This can be approximated by such sorting and via an increasing function. Normalize this sorting. In some implementations, The smaller d i，j The more it should be reduced, the better it will promote graphic similarity. In some implementations, w m，n It can indicate edge similarity, which can be used to measure the distance d between nodes in a graph. m，n Reweighting. Furthermore, α can be an additional weighting parameter based on the two distance terms. Predefined α values ​​can be used or determined based on how these two terms should contribute.

[0042] In some implementations, the graph matching process can be end-to-end learnable, such as through deep learning algorithms; however, the aforementioned techniques can be efficient re-matching and filtering mechanisms that can be used in conjunction with current feature matching techniques such as SLAM.

[0043] In some implementations, the image matching technique can be improved by additionally considering the geometric context. That is, the description above regarding boxes 430-435 describes a technique for comparing feature maps over the entire query image and reference images using visual context. Furthermore, in some implementations, geometric context can be utilized. It can be assumed that the angular order of edges in the image can be maintained across the entire viewpoint. Therefore, the geometric context can be determined to account for geometric similarity. An example of determining geometric similarity could be the additional term λd. kendall (τ i , τ j ), where τ i Representing graphic G i Middle node V i The angle is sorted, and d kendall ∈[0,1] is the normalized Kendall tau distance that measures the inconsistency between two sorted lists, and λ = 100 is the weighting parameter. Therefore, in some implementations, the uniform graph matching distance can be defined as:

[0044]

[0045] This graphical distance can be used to effectively filter out initial matches with low confidence. Such graphical matching methods provide a fast filtering mechanism, which can then be followed by standard RANSAC in a SLAM relocation pipeline.

[0046] See Figure 5 This diagram illustrates a simplified block diagram of an electronic device 500 according to one or more embodiments of the present disclosure. The electronic device 500 may be part of a multi-functional device. Furthermore, according to one or more embodiments, various components presented in the electronic device 500 may be distributed across one or more connected electronic devices. The electronic device 500 may additionally be connected via a network interface 550 to one or more other network devices 570, within which various functions may be included, or across the network interface, which may be distributed across a network 560. Exemplary networks include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 500 is used to perform feature matching techniques, such as for relocation.

[0047] Electronic device 500 may include processor 510, such as a central processing unit (CPU). Processor 510 may be a system-on-a-chip (SoC) such as those present in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 510 may include multiple processors of the same or different types. Electronic device 500 may also include memory 520. Memory 520 may include one or more different types of memory that can be used in conjunction with processor 510 to perform device functions. For example, memory 520 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 520 may store various programming modules for execution by processor 510, including descriptor module 522, feature matching module 524, relocation module 526, and other applications 528. Electronic device 500 may also include storage device 530. Storage device 530 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (e.g., CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (e.g., electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)).

[0048] The electronic device 500 may also include one or more cameras 540 or other sensors. In one or more embodiments, at least one of the one or more cameras may be a conventional RGB camera. Furthermore, the camera may include a stereo camera or other multi-camera system, a time-of-flight camera system, etc. Camera 540 may include a lens assembly, a sensor, and other circuitry that can be used together to capture RGB images.

[0049] Electronic device 500 allows users to interact with extended reality (XR) environments. Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0050] Storage device 530 can be used to store various data and structures that can be used to provide techniques for providing positioning information for images. Storage device 530 may include, for example, image data 534 and descriptor network 538. Image data 534 may include, for example, images captured by camera 540. Image data 534 may include visual data and depth data of the images.

[0051] In addition to training data, storage device 530 may also include a descriptor network 538. Descriptor network 538 may be a network trained to provide descriptors for the provided image, for example, in the form of descriptor vectors. In one or more embodiments, the descriptor network may be trained on image patches of image data associated with keypoints in the training image. In some embodiments, the descriptor network may be trained to provide binarized descriptors for the provided image data. Although the various components described within storage device 530 are depicted as being within a single storage device, it should be understood that these various components or the data within those components may be distributed across multiple storage devices or even across multiple electronic devices (e.g., across networks).

[0052] According to one or more embodiments, memory 520 may include one or more modules comprising computer-readable code executable by one or more processors 510 to perform functions. The memory may include, for example, a descriptor module 522, which may be used, for example, to obtain keypoint descriptors of an image from image data 534. In some embodiments, descriptor module 522 may obtain portions of an image that include specific keypoints, such as image patches of a given image. Descriptor module 522 may apply a descriptor network 538 to an image or a portion of an image to obtain keypoint descriptors indicating descriptions of specific keypoints. The keypoint descriptor may be a binary descriptor in the form of a compact binary string.

[0053] The memory 520 may also include a feature matching module 524 that can use keypoint descriptors to match keypoints with a reference image, such as reference data 572 from the network device 570. According to one or more embodiments, the feature matching module 524 can use the keypoint descriptors generated by the descriptor module 522 to identify keypoint matches, for example, in the reference data 572. The feature matching module 524 can then provide the matching data to the relocation module 526 for use in the relocation process.

[0054] According to one or more embodiments, the repositioning module 526 may utilize feature descriptors to determine the positioning information of the current image. For example, the camera pose may be determined based on matching data by comparing keypoint descriptors from the current image with a reference image (e.g., from reference data 572).

[0055] Although electronic device 500 is described as including the numerous components described above, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. Furthermore, additional components may be used, and some combination of the functions of any components may be possible.

[0056] Now for reference Figure 6This document illustrates a simplified functional block diagram of an exemplary multi-functional electronic device 600 according to one embodiment. Each electronic device may be a multi-functional electronic device, or may have some or all of the components described herein. The multi-functional electronic device 600 may include a processor 605, a display 610, a user interface 615, graphics hardware 620, device sensors 625 (e.g., proximity / ambient light sensors, accelerometers, and / or gyroscopes), a microphone 630, an audio codec 635, a speaker 640, communication circuitry 645, digital image capture circuitry 650 (e.g., including a camera system), a video codec 655 (e.g., supporting a digital image capture unit), a memory 660, a storage device 665, and a communication bus 670. The multi-functional electronic device 600 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0057] Processor 605 executes instructions necessary for the operation of many functions performed by device 600 (e.g., the generation and / or processing of images as disclosed herein). Processor 605 may, for example, drive display 610 and may receive user input from user interface 615. User interface 615 allows a user to interact with device 600. For example, user interface 615 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, and / or touchscreen. Processor 605 may also be, for example, a system-on-a-chip, such as those present in mobile devices, and includes a dedicated graphics processing unit (GPU). Processor 605 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 620 may be dedicated computing hardware for processing graphics and / or assisting processor 605 in processing graphics information. In one embodiment, graphics hardware 620 may include a programmable GPU.

[0058] Image capture circuit 650 may include two (or more) lens assemblies 680A and 680B, each lens assembly having a separate focal length. For example, lens assembly 680A may have a shorter focal length relative to the focal length of lens assembly 680B. Each lens assembly may have a separate associated sensor element 690. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 650 can capture still images and / or video images. The output from image capture circuit 650 can be processed at least in part by one or more video codecs 655 and / or processors 605 and / or graphics hardware 620, and / or dedicated image processing units or pipelines incorporated within circuit 650. Images thus captured may be stored in memory 660 and / or storage device 665.

[0059] The sensor and camera circuitry 650 can capture still images and video images that can be processed at least in part by devices including one or more video codecs 655 and / or processors 605 and / or graphics hardware 620 and / or dedicated image processing units incorporated within the circuitry 650. Images thus captured can be stored in memory 660 and / or storage device 665. Memory 660 may include one or more different types of media used by processor 605 and graphics hardware 620 to perform device functions. For example, memory 660 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 665 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 665 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Memory 660 and storage device 665 can be used to tangibly hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 605, such computer program code can implement one or more of the methods described herein.

[0060] It should be understood that the above description is intended to be exemplary and not restrictive. The material has been presented to enable any person skilled in the art to make and use the disclosed matters protected by the claims, and is provided in the context of a particular embodiment, variations of which will be readily apparent to a person skilled in the art (e.g., some embodiments of the disclosed embodiments may be used in combination with each other). Therefore, Figures 1 to... Figure 4 The specific arrangement of the steps or actions shown, or Figure 1 and Figures 5 to 6 The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by referring to the appended claims and the full scope of their equivalents. In the appended claims, the terms "comprising" and "wherein" are used as common Chinese equivalents to the corresponding terms "including" and "characterized in".

Claims

1. A method for relocalization, comprising: obtaining a first set of keypoints for a query image; determining a set of initial matches between the first set of keypoints and a set of reference keypoints in a reference image to obtain a first set of matched keypoints in the first set of keypoints and a second set of matched keypoints in the set of reference keypoints; constructing a feature map comprising a geometric spatial relationship between a first keypoint in the first set of matched keypoints and one or more second keypoint in the first set of matched keypoints, wherein the one or more second keypoint correspond to spatial neighbors of the first keypoint; identifying a first reference keypoint in the second set of matched keypoints from the reference image, wherein the first reference keypoint matches the first keypoint in the first set of matched keypoints; identifying one or more second reference keypoint in the second set of matched keypoints from the reference image, wherein the one or more second reference keypoint matches the one or more second keypoint in the first set of matched keypoints; generating a reference feature map comprising a geometric spatial relationship between the first reference keypoint in the second set of matched keypoints and the one or more second reference keypoint in the second set of matched keypoints; comparing a geometric pattern of the feature map with a geometric pattern of the reference feature map; and determining relocalization data for the query image based on the comparison.

2. The method of claim 1, wherein determining the set of initial matches further comprises removing keypoints from the first set of keypoints that do not match the set of reference keypoints.

3. The method of any one of claims 1-2, further comprising filtering the set of initial matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance in the set of initial matches.

4. The method of any one of claims 1-2, wherein obtaining the first set of keypoints comprises: obtaining a first feature vector for each keypoint in the first set of keypoints, wherein the set of initial matches is determined by comparing the first feature vector to one or more feature vectors of the set of reference keypoints.

5. The method of claim 4, wherein the first feature vector is obtained from a neural network trained to generate a first binary feature vector based on input image data.

6. The method of any one of claims 1-2, wherein the feature map comprises a spatial representation of a plurality of the first set of keypoints in at least a portion of the query image.

7. The method of claim 6, wherein the reference feature map comprises a spatial representation of a keypoint for a corresponding feature vector.

8. The method of any one of claims 1-2, wherein the first set of keypoints is obtained from a first portion of the query image, and wherein the set of initial matches is determined from a corresponding portion of the reference image. ​ 9. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to: obtain a first set of keypoints for a query image; determine a set of initial matches between the first set of keypoints and a set of reference keypoints in a reference image to obtain a first set of matching keypoints in the first set of keypoints and a second set of matching keypoints in the set of reference keypoints; construct a feature map comprising a geometric spatial relationship between a first keypoint in the first set of matching keypoints and one or more second keypoin ts in the first set of matching keypoints, wherein the one or more second keypoints correspond to spatial neighbors of the first keypoint; identify a first reference keypoint in the second set of matching keypoints from the reference image, wherein the first reference keypoint matches the first keypoint in the first set of matching keypoints; identify one or more second reference keypoints in the second set of matching keypoints from the reference image, wherein the one or more second reference keypoints match the one or more second keypoints in the first set of matching keypoints; generate a reference feature map comprising a geometric spatial relationship between the first reference keypoint in the second set of matching keypoints and the one or more second reference keypoints in the second set of matching keypoints; compare a geometric pattern of the feature map with a geometric pattern of the reference feature map; and determine, based on the comparison, relocalization data for the query image.

10. The non-transitory computer-readable medium of claim 9, wherein the computer-readable code to determine the set of initial matches further comprises computer-readable code to remove from the first set of keypoints keypoints that do not match the set of reference keypoints.

11. The non-transitory computer-readable medium of any one of claims 9-10, further comprising computer-readable code to filter the set of initial matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance in the set of initial matches.

12. The non-transitory computer-readable medium of any one of claims 9-10, wherein the computer-readable code to obtain the first set of keypoints comprises computer-readable code to: obtain a first feature vector for each keypoint in the first set of keypoints, wherein the set of initial matches is determined by comparing the first feature vector to one or more feature vectors of the set of reference keypoints.

13. The non-transitory computer-readable medium of any one of claims 9-10, wherein the feature map comprises a spatial representation of a plurality of the first set of keypoints in at least a portion of the query image, and wherein the reference feature map comprises a spatial representation of a keypoint for a corresponding feature vector. ​ 14. The non-transitory computer-readable medium of any one of claims 9-10, wherein the first set of keypoints is obtained from a first portion of the query image, and wherein the initial set of matches is determined from a corresponding portion of the reference image.

15. A system for relocalization, comprising: one or more processors; and one or more computer-readable media comprising computer-readable code executable by the one or more processors to: obtain a first set of keypoints for a query image; determine an initial set of matches between the first set of keypoints and a set of reference keypoints in a reference image to obtain a first set of matching keypoints in the first set of keypoints and a second set of matching keypoints in the set of reference keypoints; construct a feature graph comprising a geometric spatial relationship between a first keypoint in the first set of matching keypoints and one or more second keypoint in the first set of matching keypoints, wherein the one or more second keypoint correspond to spatial neighbors of the first keypoint; identify a first reference keypoint in the second set of matching keypoints from the reference image, wherein the first reference keypoint matches the first keypoint in the first set of matching keypoints; identify one or more second reference keypoint in the second set of matching keypoints from the reference image, wherein the one or more second reference keypoint match the one or more second keypoint in the first set of matching keypoints; generate a reference feature graph comprising a geometric spatial relationship between the first reference keypoint in the second set of matching keypoints and the one or more second reference keypoint in the second set of matching keypoints; compare a geometric pattern of the feature graph with a geometric pattern of the reference feature graph; and determine relocalization data for the query image based on the comparison.

16. The system of claim 15, wherein the computer-readable code to determine the initial set of matches further comprises computer-readable code to remove from the first set of keypoints keypoints that do not match the set of reference keypoints.

17. The system of any one of claims 15-16, further comprising computer-readable code to filter the initial set of matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance in the initial set of matches.

18. The system of any one of claims 15-16, wherein the computer-readable code to obtain the first set of keypoints comprises computer-readable code to: obtain a first feature vector for each keypoint in the first set of keypoints, wherein the initial set of matches is determined by comparing the first feature vector to one or more feature vectors of the set of reference keypoints. ​ 19. The system of claim 18, wherein the first feature vector is obtained from a neural network trained to generate a first binarized feature vector based on input image data.

20. The system of any one of claims 15-16, wherein the feature map comprises a spatial representation of a plurality of the first set of keypoints in at least a portion of the query image.

Images