Determining location of a device within an environment comprising planar surfaces

Abstract

A method (800) for determining the location of a device within an environment comprising a set of two or more planar surfaces (PSs), the set of two or more PSs comprising a first PS, PS A, and a second PS, PS_B, wherein a first set of points of interest (POIs) is associated with PS A and a second set of POIs is associated with PS B. The method includes detecting a PS, PS D, in an image of the environment that was captured using an image sensor of the device. The method further includes determining Pols associated with PS D using the captured image. The method further includes using the Pols associated with PS D to determine the PS within the set of two or more PSs in the environment to which PS D corresponds. The method further includes, after determining the PS within the set of PSs to which PS D corresponds, determining the location of the device within the environment. Wherein using the Pols associated with PS D to determine the PS within the set of PSs of the environment to which PS D corresponds comprises obtaining a first transform value using the determined Pols and the Pols associated with PS A; and determining whether the first transform value satisfies a first condition.

Description

DETERMINING LOCATION OF A DEVICE WITHIN AN ENVIRONMENTCOMPRISING PLANAR SURFACESTECHNICAL FIELD

[0001] Disclosed are a method for determining the location of a device within an environment comprising a set of two or more planar surfaces, a corresponding computer program, a corresponding carrier, and an apparatus operable for determining the location of a device within an environment comprising a set of two or more planar surfaces.BACKGROUND

[0002] Image based localization allows a computing device (e.g., a smartphone or other mobile computing device) to locate its position in an environment using previously captured images or a three-dimensional (3D) map. The computing device may locate itself by comparing visual information of captured images with that of the previously captured images. The visual information may include information regarding planar structural elements (e.g., walls). Comparing the visual information may include performing local feature matching between two or more images. Local feature matching includes first recognizing features of the same scene across images with slightly different viewpoints and then establishing correspondences between them.

[0003] Matching local features is a critical step in many visual pipelines, as it produces the association of the geometric data between two sources of visual information required for registration, which is the process of transforming different sets of data into a common coordinate system (e.g., relative pose between two images, or relative pose between image and a map). Local feature matching, however, is highly sensitive to perturbations on the visual content of the image (e.g., dynamic changes in the environment, lighting variations). Local feature matching is also more challenging in an interoperable scenario (i.e., the collaboration of devices that use different algorithms to process images).

[0004] A local feature (also known as, a point-of-interest (Pol) of an image may include a combination of a keypoint and a local descriptor. Keypoints are salient points of an image (e.g., features from accelerated segment test (FAST), difference-of-gaussian (DoG), and Harris). The keypoints may be expressed as a vector of xy-coordinates. The keypoints may be described with the local descriptors, which consist of compact vectors that represent the local intensity values in the region around a keypoint (e.g., scale invariant feature transform (SIFT), binary robust independent elementary features (BRIEF), and orientedFAST and rotated BRIEF (ORB). Each type of local descriptor may be designed in a different manner making a direct comparison between different descriptors challenging. For example, local descriptors may be fitted to certain detectors (e.g., ORB descriptor requires a rotation- aware detector for optimal results).

[0005] Local feature matching typically consists of associating the keypoints of an image to those in another image with the most similar local descriptor. The matching may be refined by ensuring distinctiveness through comparison with the second-best match. Some local feature matching methods leverage the spatial distribution of the matches in the images to improve the results, by means of a spatial grid and polar constraints. Other methods may use a Graph Neural Network to add high-level learned knowledge to the local feature association problem. Another method may enable matching between local features by employing a learned latent space of local descriptors while assuming the same detector is used. Yet another method may employ a convolutional neural network that processes red, blue, and green (RGB), depth, and normal vectors from planar surfaces to produce a descriptor for planes. Coplanar matches may then be fed to a robust optimization algorithm to allow for visual localization.SUMMARY

[0006] Certain challenges presently exist. For example, a computing device attempting to locate itself in an environment may use a different image processing algorithm and / or different sensors than the device which previously captured images of the environment (e.g., iPhone vs Android, drone vs robotic vacuum cleaner, Microsoft Hololens vs NReal XR glasses). As such, the local features generated by the two devices may contain different types of keypoints and local descriptors. Local feature matching algorithms, however, require the same type of keypoints and local descriptors. For example, keypoints first detected and described with SIFT, must be matched against keypoints extracted and described with the same SIFT detector+descriptor. Therefore, a process for local feature matching and localization utilizing different types of local features is highly desired.

[0007] Accordingly, in one aspect there is provided a method for determining the location of a device within an environment comprising a set of two or more planar surfaces (PSs), the set of two or more PSs comprising a first PS, PS A, and a second PS, PS_B, wherein a first set of points of interest (POIs) is associated with PS A and a second set of POIs is associated with PS_B. The method includes detecting a PS, PS D, in an image of theenvironment that was captured using an image sensor of the device. The method further includes determining Pols associated with PS D using the captured image. The method further includes using the Pols associated with PS D to determine the PS within the set of two or more PSs in the environment to which PS D corresponds. The method further includes, after determining the PS within the set of PSs to which PS D corresponds, determining the location of the device within the environment. Wherein using the Pols associated with PS D to determine the PS within the set of PSs of the environment to which PS D corresponds comprises obtaining a first transform value using the determined Pols and the Pols associated with PS A; and determining whether the first transform value satisfies a first condition.

[0008] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of a computing apparatus causes the computing apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium. In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitry coupled to the memory.

[0009] An advantage of the embodiments disclosed herein is efficiently matching local features of different types when performing image-based localization.

[0010] Another advantage of the embodiments disclosed herein is reducing the number of false associations in local feature matching by comparing and verifying information in each individual plane in a separate manner.

[0011] Another advantage of the embodiments disclosed herein is reducing computational costs by only exploiting information detected on structural elements.

[0012] Another advantage of the embodiments disclosed herein is efficiently associating visual information of different images under challenging conditions. For example, the challenging conditions may include heterogeneous pipelines and dynamic environments.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0014] FIG. 1 A illustrates a block diagram of an environment according to an embodiment.

[0015] FIG. IB illustrates a block diagram of a 3D map according to an embodiment.

[0016] FIG. 2A illustrates a block diagram of an environment according to an embodiment.

[0017] FIG. 2B illustrates a block diagram of captured images according to an embodiment.

[0018] FIG. 3 is a flowchart illustrating a process according to an embodiment.

[0019] FIG. 4 is a flowchart illustrating a process according to an embodiment.

[0020] FIG. 5 illustrates a block diagram of a matching process according to an embodiment.

[0021] FIG. 6 is a flowchart illustrating a process according to an embodiment.

[0022] FIG. 7 illustrates a block diagram according to an embodiment.

[0023] FIG. 8 is a flowchart illustrating a process according to an embodiment.

[0024] FIG. 9 is a block diagram of a computing apparatus according to an embodiment.

[0025] FIG. 10 illustrates a graph showing results of a case study.DETAILED DESCRIPTION

[0026] FIG. 1 A illustrates an environment 100, according to an embodiment. Environment 100 includes three planar surfaces 104A, 104B, and 104C. A first device 102 may generate a 3D map of environment 100. First device 102 includes an image sensor configured to capture one or more images (e.g., video recording or a series of still images) of environment 100. To generate the 3D map, first device 102 may process the captured images using a 3D reconstruction algorithm (e.g., Structure-from-Motion (SfM) or Simultaneous Localization and Mapping (SLAM)) that uses a first type of detector and / or descriptor configuration (e.g., SIFT detector with SIFT descriptor). Additionally, each image may be processed to detect and segment the structural planes that it contains. In some embodiments, first device 102 may transmit the captured images to a remote computing device 108 for processing the images.

[0027] FIG. IB illustrates an example 3D map 112 of environment 100 generated by first device 102. 3D map 112 includes representations of the three planar surfaces 104A, 104B, and 104C. Each of planar surfaces 104A, 104B, and 104C comprises a set of points of interest (Pols) 106 associated with the planar surface. 3D map 112 may be stored at first device 102, remote computing device 108, or another computing device.

[0028] FIG. 2 A illustrates a second device 202 located within environment 100, according to an embodiment. Second device 202 may determine its location in environment 100 using image-based localization. Second device 202 includes an image sensor configured to capture one or more images (e.g., video recording or a series of still images) of environment 100. Second device 202 may process the captured images using a second type of detector+descriptor configuration (e.g., FAST detector with BRIEF descriptor or ORB detector with ORB descriptor). In some embodiments, second device 202 may transmit the captured images to a remote computing device 208 for processing the images.

[0029] Fig. 2B illustrates an output of the image processing performed by second device 202. When processing the captured images, second device 202 may detect two planar surfaces 204A and 204B. Second device 202 may also determine a set of Pols 206 associated with each of the planar surfaces. Second device 202 may transmit a request for 3D map 112 to first device 102 or device 108. After receiving 3D map 112, second device 202 may determine its location in environment 100 using the visual information (e.g., planar surfaces and Pols) of the captured images by second device 202 and of 3D map 112.

[0030] First device 102 and second device 202, however, may use two different image processing algorithms. For example, first device 102 may use a first algorithm with SIFT detector with a SIFT descriptor while second device 202 may use a second algorithm with a FAST detector with a BRIEF descriptor. As such, Pols 106 determined by first device 102 and Pols 206 determined by second device 202 may have different types of keypoints and descriptors.

[0031] The embodiments described herein illustrate an efficient method for performing image-based localization with different types of local features. Here, the amount of potential false associations of local feature matching is reduced because the embodiments first determine whether two planar surfaces are likely to correspond by comparing their local features and then verify that the two planar surfaces can be the same real-world surface viewed from two different positions. Consequently, associating visual information onstructural planes with that from non- structural items (e.g., furniture) may be avoided, which results in improved matching results.

[0032] The embodiments described herein also reduce computational costs by only storing / saving information detected on structural elements. This allows 3D models (or the local features from the images) to occupy significantly less space.

[0033] The embodiments described herein additionally illustrate an efficient method that is able to associate the visual information of different images under challenging conditions. For example, the methods described herein may match local features in heterogeneous pipelines (e.g., keypoints detected at different locations which are described differently) and in dynamic environments where the visual information contained by structural elements is less likely to change.

[0034] In one embodiment, a process may associate keypoints detected on structural planes (e.g., walls) from two different images, which results in an association that is more robust in the interoperable case. For example, the interoperable case may include a heterogeneous pipeline where different devices use different keypoint detectors and local descriptors. The process may perform a prior detection and segmentation of the structural planes in each image, which allows the process to detect and describe only those Pols that lie in the structural planes. For example, the planes may have properties that help with the association even under challenging conditions, resulting in a more robust association. With this process, one may provide a solution to the interoperability keypoint detector problem, making it possible to localize two devices using different keypoint detectors based on each other’s information.

[0035] The process may produce coplanar constraints over the local features lying on structural planes by means of the projective properties of these surfaces. The process may rely on the intuition that, even in the case of heterogeneous detectors, keypoints share a similar spatial distribution or arrangement even from different perspectives. This intuition is especially relevant with regard to features lying on planes, where the resulting projective transformations maintain a set of verifiable properties. Finally, this process solves the plane- to-plane correspondence problem that ensures global consistency, which avoids various-to- one plane correspondences.

[0036] Embodiments described herein improve local feature matching between two images through spatial verification of the visual information detected on structural planes of ascene (e.g., walls), which is particularly beneficial in the case of disparate keypoints arising in an interoperable scenario. Such verification leverages the visual properties of planar surfaces (e.g., homography), enforcing matches between coplanar keypoints. The processes herein may receive as input the local features detected on walls from two images and output a consistent association between them, improving the robustness under challenging scenarios such as heterogeneous detectors or dynamic environments.

[0037] In one embodiment, device 102 (e.g., iPhone) may visit an environment (e.g., environment 100), record the environment, and store associated information (e.g., images, keypoints). These images may be employed to build a 3D map or model from its measurements, given a localization algorithm (e.g., simultaneous location and mapping (SLAM), Structure-from-Motion) that uses a detector+descriptor configuration of type A (e.g., SIFT detector with SIFT descriptor). To reduce the computational load of the map, the localization algorithm may only take into account the visual information stored in the walls.

[0038] Second device 202 (e.g., Microsoft Hololens XR glasses), which encodes images through a configuration of type B (e.g., FAST detector with BRIEF descriptor or ORB detector with ORB descriptor), may aim to obtain an accurate localization with respect to the previously recorded images or with respect to the constructed map.

[0039] Overview

[0040] The input of a process according to an embodiment may include a pair of images. The process may perform the association of local features between both images by applying coplanar constraints on the visual information detected on structural elements.

[0041] The process may comprise a first plane-to-plane local feature matching step, which produces a set of per-image interplane local feature correspondences. A second step may include a coplanarity-constrained feature matching verifier, which evaluates if each set of plane-to-plane correspondences is feasible.

[0042] The process may include preprocessing for each single image. The preprocessing may include:

[0043] 1) Structural planes detection and segmentation. The planes from structural elements present in each image may be detected and segmented prior to performing the association of local features, by means of a dedicated method (e.g., Sigma-FP, convolutional neural network (CNN)-based semantic segmentation) that also avoids artifacts resulting ofthe masking process (e.g., objects silhouettes). The outcome of this step may be an image where the walls have been segmented.

[0044] 2) Structural plane keypoint detection and description. The salient geometric points of the structural planes of each image may be detected and then described using the local appearance around them (e.g., ORB, BRISK). The detector+descriptor configuration can be either homogeneous (e.g., ORB detector + ORB descriptor, SIFT detector + SIFT descriptor) or heterogeneous (e.g., BRISK detector + ORB descriptor, SIFT detector + BRIEF descriptor).

[0045] The first step may perform matching between local features of the structural planes in both images. Such matching may be carried out through dedicated algorithms (e.g., fast library for approximate nearest neighbor (FLANN)), which accounts for the visual similarity of the local descriptors. In order to ensure distinctiveness of the match, the process may also run a refinement step (e.g., Cross-check, Lowe’s ratio).

[0046] The association between the local features at each plane may be performed at instance level (i.e., the method matches the local features of the i-th structural plane of the first image against those features of the j -th plane in the second one). Consequently, if the first image contains N planes and the second one M, the output of this step may include a set of NxM sets of inter-plane local feature correspondences (whose number may vary in each case, depending on the number of detected features in the different planes).

[0047] Coplanarity Verification

[0048] The process may include coplanarity verification. The coplanarity verification step may include a spatial verification process which ensures that a set of inter-plane local feature matches from two different images are consistent with the projective properties of planes.

[0049] The process may verify the local feature correspondence between images in a plane-to-plane manner by means of its projective properties (e.g., homography). After verifying the feasibility of the individual local feature matches for each pair of planes, the process may compute a globally consistent solution. In some embodiments, the process may be used in the case of two heterogeneous keypoint detectors with a more similar nature (e.g., using two corner-based detectors such as ORB and BRISK).

[0050] First, the process may ensure that there is a sufficient number of matches T_min, since the geometric verification for coplanar sets of associated local features mayrequire a minimum number of inliers. The steps of the process below may focus on the spatial properties of each local feature, that is, on its keypoint.

[0051] The process may then perform a verification process based on the geometric properties of the planar surfaces in images (e.g., homography). It may attempt to estimate a projective transformation between each pair of surfaces of the first and second images, which may become degenerated if the matching or the spatial distribution of the keypoints in both planes are not consistent. In order to deal with the case of heterogeneous detectors (where keypoints may not be detected in the same manner), the alignment estimation error of the projective transform (i.e., the final error after finding the best possible alignment, for example, the projective error) may be set higher than in the case using a same detector+descriptor configuration.

[0052] The resulting projective transformation produces non-degenerate results when keypoints are sufficiently coplanar and there is a minimum covisibility between the matched planar surfaces, which is likely to happen in the case of hierarchical localization pipelines (which first retrieve close images in a coarse manner through whole image descriptors as Bag-of-Words (BoW) or Net VLAD (a convolutional neural network inspired by the vector of locally aggregated descriptor (VLAD) algorithm).

[0053] The verification of the plane-to-plane matching may include verifying the feasibility of the geometric components of the projective transform estimated in the previous step (a rotation and a translation), which also may use the calibration parameters of the camera.

[0054] The verification of the rotation may consider that the matches preserve a consistent orientation in the distribution of the keypoints of both planes (e.g., walls are not seen from an inverted perspective). Such verification can be performed, for example, through the determinant of the rotational submatrix HR (2X2 upper-left submatrix) from the estimated projective transform H: det(WR) > 0 (1)

[0055] The translation component of the planar projective transform may depend on the scale, and since the scale problem is solved (depth information is available from the map images), one may check its feasibility. The transform may be declared as feasible if the translation is lower than a threshold r_t, which is set by taking into account the scale and nature of the environment (e.g., in most indoor environments, the same part of the scenecould be observed from locations spaced a few meters apart and the threshold may be set to a few meters, e.g., 8 meters). This verification may be supported by the fact that typically, wrong plane-to-plane matches are prone to lead to ill-scaled translation estimates.

[0056] After the feasibility of the transformation is verified and considering only the number of inliers K considered for the transformation estimation, a score s_(i,j) may be assigned to the plane-to-plane matching set using, for example, the very number of inliers of the match or the properties of the transformation (e.g., ratio between inliers considered for transformation estimation and total number of initial matches).

[0057] In the case that a plane-to-plane match cannot be verified, it may be classified as unfeasible and is not considered for the final step.

[0058] Once the initial set of plane-to-plane matches is pruned to a verified subset with an associated feasibility score, the process may perform a verification of the global consistency.

[0059] This process may be performed by applying a linear assignment solver (e.g., Hungarian or Sinkhorm algorithms) over the set of verified plane-to-plane matches, which ensures that each plane in the first image is matched with a single plane in the second image and maximizes the accumulated feasibility score by taking into account the feasibility of each plane-to-plane match. The algorithm avoids obtaining matches for non-feasible planes.

[0060] Finally, the matching result between the keypoints of both images is obtained by gathering the local feature matches of each pair of assigned planes from the outcome of the last step.

[0061] In some embodiments, the process may be extended by leveraging the information not comprised in structural planes such as items and objects (e.g., furniture). Concretely, if parts of the image that belong to objects are known, the process may also leverage the visual information it contains by forcing to match local features from objects in one image with those in the second image. Furthermore, the process can be enhanced by exploiting semantic information from images (i.e., images whose pixels are annotated with the object class they represent), ensuring feature matches between parts of the images that are showing the same object class.

[0062] Referring back to FIG. IB, first device 102 may perform process 300 to generate 3D map 112. First device 102 may repeatedly perform the steps of process 300 as itcaptures images from environment 100. In some embodiments, another computer device, such as remote computing device 108, may perform one or more of the steps of process 300.

[0063] FIG. 3 is a flowchart illustrating a process 300, according to an embodiment. Process 300 may begin with step s302. Step s302 comprises first device 102 capturing one or more images of environment 100 using its image sensor. In some embodiments, first device 102 may capture a video of environment 100. For example, first device may be embodied as an iPhone and may use its built-in camera to take images of environment 100.

[0064] Step s304 comprises extracting Pols from the captured images. Pols may be extracted and described using a feature detector+descriptor algorithm (e.g., SIFT, BRIEF and ORB). The feature detector+descriptor algorithm may process the images in two stages.First, keypoints of the Pols are detected by comparing the intensity of each pixel with the surrounding pixels. First device 102 may extract Pols 106 from the captured images, each of which may include a keypoint and a descriptor.

[0065] Step s306 comprises detecting one or more planar surfaces in the captured images. The planar surfaces may be determined using a method that produces the segmentation masks required to know which pixel belongs to planar surfaces in the image (e.g., Sigma-FP, convolutional neural network (CNN)-based semantic segmentation). In some embodiments, the planar surfaces may be embodied as walls of a room or a structure. First device 102 may detect that environment 100 comprises three planar surfaces, planar surface 104A, planar surface 104B, and planar surface 104C. Each of planar surfaces 104A, 104B, and 104C is associated with a set of Pols 106.

[0066] Step s308 comprises generating a 3D map. A 3D map may be generated using Pols and planar surfaces with a 3D reconstruction algorithm (e.g.: COLMAP, RTABMAP). A 3D reconstruction algorithm may search for Pols which appear in multiple images and reproject them into a 3D space by means of camera geometry. First device 102 may generate 3D map 112 using a 3D reconstruction algorithm with planar surfaces 104A-C with their associated Pols 106. 3D map 112 of environment 100 may be the product of comparing and projecting the two-dimensional information (Pols) from multiple images gathered in the same environment into a three-dimensional representation. The result is a set of 3D-annotated Pols that reflects the most salient elements of the mapped environment. After generating 3D map 112, first device 102 may store 3D map 312 for later use by another computing device, e.g., the second device 202.

[0067] In some embodiments, step s308 may be optional. First device 102 may process the captured images of step s302 and store them for later use. In such embodiments, second device 202 may determine its location based on the processed images.

[0068] Referring back to FIG. 2A, second device 202 may perform a process 400 to determine its location using image-based localization. Second device 202 may repeatedly perform the steps of process 400 as it captures images from environment 100. In some embodiments, another computing device, such as remote computing device 208, may perform one or more of the steps of process 400.

[0069] FIG. 4 is a flowchart illustrating process 400, according to an embodiment. Process 400 may begin with step s402. Step s402 comprises second device 202 capturing one or more images of environment 100 using its image sensor. For example, second device 202 may be embodied as an Android phone and may use its built-in camera to take images of environment 100. Second device 202 may be located at the same, or a different, part of environment 100 as first device 102.

[0070] Step s404 comprises extracting Pols from the captured images. Pols may be extracted and described using a feature detector+descriptor algorithm (e.g., SIFT, BRIEF and ORB). The feature detector+descriptor algorithm may process the images in two stages.First, keypoints of the Pols are detected by comparing the intensity of each pixel with the surrounding pixels. Second device 202 may extract Pol 206 from the captured images, each of which may include a keypoint and a descriptor. In some embodiments, second device 202 may utilize a different image processing algorithm than first device 102 outputting a different type of Pol. In some embodiments, second device 202 may utilize the same image processing algorithm than first device 102 outputting a different type of Pol.

[0071] Step s406 comprises detecting one or more planar surfaces in the captured images. The planar surfaces may be determined using a method that produces the segmentation masks required to know which pixel belongs to planar surfaces in the image (e.g., Sigma-FP, CNN-based semantic segmentation). Second device 202 may detect that the captured images include two planar surfaces, planar surface 204 A and planar surface 204B. Each of planar surfaces 204A and 204B may include a set of Pols 206.

[0072] Step s408 comprises second device 202 requesting 3D map 112 from first device 102. In some embodiments, second device 202 may request 3D map 112 from another computing device storing 3D map 112.

[0073] Step s410 comprises matching Pols of the planar surfaces. Second device 202 may compare and match Pols 206 with Pols 106 from first device 102. Second device 202 may match Pol 206 with Pols 106 by comparing their descriptors and searching for the most similar Pols. In some embodiments, first device 102 and second device 202 may use different Pol detection and / or description algorithms. As such, the keypoints and descriptors of Pol 106 will differ from the keypoints and descriptors of Pol 206. For example, SIFT descriptors are 128-byte float, ORB descriptors are 32-byte bool, and BRISK descriptors are 64-byte bool. SIFT detects blobs as keypoints and BRISK / ORB / FAST detect corners as keypoints.

[0074] Pol matching may be performed in a planar surface to planar surface manner. Second device 202 may perform searches for matching Pol for each planar surface of planar surfaces 204 with each planar surface of planar surfaces 104. The result of step s410 is a set of matches between the Pols lying on each pair of planar surfaces. For example, if planar surfaces 204 contains N planar surfaces and planar surfaces 104 contains M planar surfaces, the output of this step consists of a set of NxM sets of inter-planar surfaces Pol correspondences.

[0075] FIG. 5 is an illustrative example of the matching performed by second device 202 in step s410. The matching may be performed using dedicated algorithms (e.g., fast library for approximate nearest neighbor (FLANN)) which accounts the visual similarity of the local descriptors of the Pols. In order to ensure distinctiveness of the match, second device 202 may use an additional refinement step (e.g., Cross-check, Lowe’s ratio). Second device 202 may compare the Pols 206 of planar surfaces 204 A and 204B with the Pol 106 of planar surfaces 104A, 104B, and 104C of 3D map 112.

[0076] For example, planar surface 204A may be associated with a vector of Pols. Planar surface 204A = {Pol 206 A1, Pol 206 A2, Pol 206 A3}. Each Pol 206 may be associated with a coordinate (keypoint) vector (c = {x,y}) and a descriptor vector (d = {dl, d2, ...}). As illustrated in FIG. 5, the output of the matching for planar surface 204A may include two matches with planar surface 104A (Pol 206 A2 isEquiv Pol 106 A2, Pol 206 A3 isEquiv Pol 106 A4), three matches with planar surface 104B (Pol 206 A1 isEquiv Pol 106 B1, Pol 206 A2 isEquiv Pol 106 B2, Pol 206 A3 isEquiv Pol 106 B3), and one match with planar surface 104C (Pol 206 A3 isEquiv Pol 106 C3). Wherein “isEquiv” identifies that two Pol are equivalent to each other. Second device 202 may perform a similar process for planar surface 204B.

[0077] Referring back to FIG. 4, step s412 comprises verification of per-planar surface matches. Each of the sets of inter-planar surface matches may be verified using the spatial properties of the planar surfaces. For the matches between the planar surfaces, second device 202 may perform a process 600.

[0078] Process 600 begins with step s602. Step s602 comprises determining whether a planar surface pair includes a minimum number of Pol matches. If the planar surface pair includes the minimum number of Pol matches, process 600 proceeds to step s604. If not, this set of matches for the planar surface pair may be considered unfeasible and process 600 may progress to step 612.

[0079] For example, second device 202 may determine planar surface 204 A has two matches with planar surface 104A (Pol 206 A2 isEquiv Pol 106 A2, Pol 206 A3 isEquiv Pol 106 A4), three matches with planar surface 104B (Pol 206 A1 isEquiv Pol 106 B1, Pol 206 A2 isEquiv Pol 106 B2, Pol 206 A3 isEquiv Pol 106 B3), and one match with planar surface 104C (Pol 206 A3 isEquiv Pol 106 C3). If the minimum number of Pol matches is set to 2, then planar surface 204A is likely to correspond with either planar surfaces 104 A or 104B since each pair has a number of matches greater than or equal to the minimum number. Planar surface 204A is unlikely to correspond with planer surface 104C since the number of matches is less than the minimum number. As such, second device 202 will only progress planar surfaces 104 A and 104B to step s604 for planar surface 204 A.

[0080] Step s604 applies a homography to assess the spatial feasibility of the set of remaining matches. Homography may validate that the spatial distribution of the matched Pols in both planar surfaces are similar and can be geometrically verified. Given a set of matches between Pols that lie in two planar surfaces or planes, homography produces the transform H (namely, a translation t and a rotation H r). The homography function may only use the keypoints (the xy coordinates) of the Pols.

[0081] Homography is a perspective transformation that describes how the perception of a plane varies when observed from different points. The transformation H encodes a 2D rotation, a 2D translation, a scale variation, and a shear transformation, and may be embodied as a 3x3 Homography matrix:

[0082] Normalizing the matrix by setting i33= 1, the homography matrix H may be deconstructed into equation (3):H = KTK~ (3)

[0083] Where T is a transformation matrix which encodes a 3 -dimensional rotation and a 3-dimensional translation t =tyfz], where tx, ty, and tz is the position of second device 202 relative to the position of first device 102 in the x, y, and z direction. This matrix is typically expressed in homogeneous coordinates as:

[0084] K is a calibration matrix of the camera of the localizer device (e.g., second device 202), a matrix that stores the parameters of such camera (e.g., focal length, optical center, and radial distortion coefficients) and helps to determine the relation between a 3D point in the world and its 2D projection in the image.

[0085] From transformation H, second device 202 may extract:

[0086] Where H r may be embodied as a 2x2 matrix that reflects part of the 2- dimensional (2D) affine transformation. This part of the affine transformation predicts the 2D rotation of the camera with respect to the concerned plane. Process 600 may later check that the affine transformation is orientation-preserving.

[0087] From transformation T, second device 202 may also extract:

[0088] Where t is a vector representing a 3-dimensional (3D) translation of the camera.

[0089] Step 606 comprises determining whether the rotation from the homography H r is satisfies a feasibility condition. A homography applied over an inadequate match produces an incongruent rotation. For example, that the planar surfaces are seen upside-down or in an inverted perspective. If the rotation submatrix produces an incongruent rotation, then the match is considered unfeasible and process 600 proceeds to step s612. If the homography rotation H r is feasible, process 600 proceeds to step s608.

[0090] To verify H r, second device 202 may check whether the determinant of H r is positive, i.e., det( / f_r) > 0 (7)

[0091] If the determinant of Hj- is lower than 0, it means that the transformation does not preserve the orientation of the point distribution.

[0092] FIG. 7 illustrates an example of determining whether H r is feasible when comparing planar surface 104B and planar surface 204A. FIG. 7 comprises two different embodiments of planar surface 204A, planar surface 204A_l and planar surface 204A_2. When the computation is performed with planar surface 204A 1, the preserving homography has rotated, but the distribution of the points is the same (the order POI 206 A1, POI 206 A2, and POI 206 A3 is still clockwise). As such, det( / f_r) > 0, the orientation is preserved, and planar surface 204A_l is feasible. Planar surface 204A_2, however, is nonpreserving with a different distribution (now POI 206 A1, POI 206 A2, and POI 206 A3 is counterclockwise). As such, det( / f_r) < 0, the orientation is not preserved, and planar surface 204A 2 is not feasible. In FIG. 7, a homography with non-preserving orientation means that the camera is seeing the image from the other side of the planar surface, which is impossible.

[0093] Step 608 comprises determining if the translation component t of the transform is up-to-scale in environment 100 by comparing it with respect to a transformation threshold r_t. Second device 202 may determine if the translation component t is adequate by comparing the Pols of the matching planer surfaces. An inadequate match between Pols produces a homography with an incongruent translation value. For example, the homography estimates that the camera has moved 100 meters, but the map represents a small room. If the translation component is inadequate, process 600 proceeds to step s612. If the translation component is adequate, process 600 proceeds to step s610.

[0094] To verify t, second device 202 may perform: verify t(t, r_t) (8)

[0095] If the norm of the translation t is lower than threshold r_t, the planar surface pair is considered to be feasible. r_t may be set by taking into account the scale and nature of the environment (e.g., the dimensions of the 3D map). For example, in most indoorenvironments, the same part of the scene may be observed from locations spaced a few meters apart, but not for example by a distance of 100 meters.

[0096] The translation may be verified in terms of the scale of the environment. For that, one may verify that the 12-norm of the translation vector is lower than a certain threshold that depends on the scene scale:

[0097] For example, in a small room of 9 square meters, a feasible value for i t may be set to 3. So, if the extracted translation vector takes the values t = [8 —5 4], it will be classified as unfeasible, since its 12-norm is ||t||2= 10.24 > 3.

[0098] Step s610 comprises computing a score. The score for a planar surface pair may be computed using its properties, for example: the number of Pols matched, the reprojection error of the homography, etc. The score may also be determined, for example, using the number of inliers of the match or the properties of the transformation (e.g., ratio between inliers considered for transformation estimation and total number of initial matches). For example, second device 202 may calculate:Feasibility score Ad = Score(Matches Ad) IF is feasible Ad (10)

[0099] Where Feasibility score Ad is the score of the planar surface pair, Score(Matches Ad) is a score based on the number of matching Pols, and is feasible Ad is a function identifying if the planar surface pair was determined to be feasible at steps s602, s606, and s608. Second device 202 may assign a first match score to the pair of planar surfaces 204 A and 104 A and a second match score to the pair of planar surfaces 204 A and 104B. The second match score may be greater than the first match score because the pair of planar surfaces 204 A and 104B has more matching Pols than the pair of planar surfaces 204 A and 104 A.

[0100] Step s612 comprises classifying a planar surface pair as unfeasible. If all planar surface pairs are classified as unfeasible, second device 202 may be prompted to change positions in environment 100 and attempt localization again.

[0101] Referring back to FIG. 4, step s414 comprises verifying a global consistency. Verifying the global consistency may include applying a global solver that ensures global consistency in the resulting matches. The global solver may apply a linear assignment solver (e.g., Hungarian or Sinkhorm algorithms) over the set of verified planar surface matches. Thelinear assignment solver may ensure that each plane is matched with a single plane and maximizes the accumulated feasibility score by taking into account the feasibility of each planar surface match.

[0102] For example, in step s610, second device 202 may have computed a score for each planar surface match as illustrated in table 1 below.Table 1

[0103] In Table 1, the match of planar surface 204 A and planar surface 104C did not receive a score as their match was previously determined to be not feasible. Second device 202 may determine planar surface 204A is associated with planar surface 104B and planar surface 204B is associated with planar surface 104C as this pairing maximizes an overall feasibility score.

[0104] Step s416 comprises using the verified matches to estimate the pose of second device 202. Second device 202 may input the verified matches into a dedicated solver (e.g., COLMAP, hLoc), which may estimate the pose of the second device 202. Second device 202 may estimate it is positioned to the right of the original location of first device 102.

[0105] FIG. 8 is a flowchart illustrating a process 800, according to an embodiment for determining the location of a device within an environment comprising a set of two or more planar surfaces (PSs), the set of two or more PSs comprising a first PS, PS A, and a second PS, PS_B, wherein a first set of points of interest (POIs) is associated with PS A and a second set of POIs is associated with PS_B.

[0106] Process 800 may begin with s802. Step s802 comprises detecting a PS, PS D, in an image of the environment that was captured using an image sensor of the device. Step s804 comprises determining Pols associated with PS D using the captured image. Step s806 comprises using the Pols associated with PS D to determine the PS within the set of two ormore PSs in the environment to which PS D corresponds. Step s808 comprises, after determining the PS within the set of PSs to which PS D corresponds, determining the location of the device within the environment. Wherein using the Pols associated with PS D to determine the PS within the set of PSs of the environment to which PS D corresponds comprises obtaining a first transform value using the determined Pols and the Pols associated with PS A and determining whether the first transform value satisfies a first condition.

[0107] In some embodiments, the first transform value is derived from a rotation transform, or the first transform value is derived from a translation transform.

[0108] In some embodiments, using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds further comprises: obtaining a second transform value using the Pols associated with PS D and the Pols associated with PS A; and determining whether the second transform value satisfies a second condition.

[0109] In some embodiments, the first transform value is derived from a rotation transform, and determining whether the first transform value satisfies the first condition comprises determining whether the first transform value satisfies a rotation condition.

[0110] In some embodiments, the second transform value is derived from a translation transform, and determining whether the second transform value satisfies the second condition comprises determining whether the translation transform value satisfies a translation condition.

[0111] In some embodiments, process 800 further comprises obtaining a homography matrix, H, using the Pols associated with PS D and the Pols associated with PS A.

[0112] In some embodiments, the first transform value is the rotation transform value, and process 800 further comprises using H to obtain a rotation matrix, H r, which allows for predicting a rotation of the image sensor with respect to the PS, and the rotation transformation value is the determinant of H r.

[0113] In some embodiments, process 800 further comprises using H to obtain a transformation matrix, T, encoding a three-dimensional (3D) rotation and a 3D translation vector, t, wherein the second transformation value is obtained using the translation vector t.

[0114] In some embodiments, the second transformation value is the norm of the translation vector t.

[0115] In some embodiments, using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds further comprises assigning a first match score to PS A based on a number of Pols associated with PS D that match one of the Pols associated with PS A.

[0116] In some embodiments, using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds comprises: assigning a second match score to PS_B based on a number of Pols associated with PS D that match one of the Pols associated with PS_B; determining that the first match score assigned to PS A is greater than the second match score assigned to PS_B; and as a result of determining the first match score is greater than the second match score, determining that PS D corresponds to PS A.

[0117] In some embodiments, assigning the first match score to PS A comprises: determining a total number of the Pols associated with PS D that match at least one Pol associated with PS A; and assigning the first match score based on the number, wherein there is a positive correlation between the first match score and the number.

[0118] In some embodiments, a number of Pols associated with PS D match at least one Pol associated with PS A, and using the Pols associated with PS D and the Pols associated with PS A to determine whether PS D corresponds to PS A further comprises: determining the number of Pols associated with PS D that match at least one Pol associated with PS A; and determining whether the number is greater than a threshold.

[0119] In some embodiments, the Pols associated with PS D are a first type of Pol (e.g., the Pols associated with PS D are determined using a first Pol detection and description algorithm), the Pols associated with PS A are a second type of Pol (e.g., the Pols associated with PS A are determined using a second Pol detection and description algorithm), and the first type of Pol and the second type of Pol are different (e.g., the first Pol detection and description algorithm is different than the second Pol detection and description algorithm).

[0120] In some embodiments, the set of two or more PSs are associated with a 3D map of the environment.

[0121] In some embodiments, the first set of Pols and the second set of POIs are included in the 3D map.

[0122] FIG. 9 is a block diagram of a computing device 900 (e.g., first device 102, second device 202, remote computing device 108 and 208) according to some embodiments,that can perform process 800. As shown in FIG. 9, computing device 900 may comprise: processing circuitry (PC) 902, which may include one or more processors (P) 955 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., second device 202 may be a distributed computing apparatus comprising two or more computers or a monolithic computing apparatus consisting of a single computer); at least one network interface 948 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 945 and a receiver (Rx) 947 for enabling computing device 900 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 948 is connected (physically or wirelessly) (e.g., network interface 948 may be coupled to an antenna arrangement comprising one or more antennas for enabling computing device 900 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 908, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 902 includes a programmable processor, a computer readable storage medium (CRSM) 942 may be provided. CRSM 942 may store a computer program (CP) 943 comprising computer readable instructions (CRI) 944. CRSM 942 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 944 of computer program 943 is configured such that when executed by PC 902, the CRI causes computing device 900 to perform the steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, computing device 900 may be configured to perform the steps described herein without the need for code. That is, for example, PC 902 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0123] Case Study: Visual Localization

[0124] To demonstrate the validity of the embodiments disclosed herein, a case study was performed through Visual Localization as the application problem. A proper feature matching was crucial to obtain accurate results.

[0125] This case study was carried out in the Robot@VirtualHome dataset, by using the Hierarchical-Localization tool for Visual Localization and Sigma-FP for planar surfacedetection and segmentation. The environment was first reconstructed through SfM using a certain configuration (note that, for the coplanarity-constrained matching, the reconstructed model only stores visual information from structural planes, i.e., planar surfaces). Pose estimation results were compared for a set of query images, using a typical dense matching method (Nearest Neighbor) and the embodiments described herein. For the query, both homogeneous and heterogeneous cases were considered, the latter by varying the keypoint detector and employing a common local descriptor (e.g., BRIEF).

[0126] FIG. 10 illustrates a graph 1000, according to an embodiment. Graph 1000 illustrates the results on Visual Localization, as the number of correctly localized queries under different position thresholds, comparing the Nearest Neighbor matching with the embodiments described herein. The results show that constraining the matching between coplanar features increases the estimation accuracy by more than a 10% for both homogeneous and heterogeneous scenarios. In fact, the embodiments described herein achieved 100% of accuracy in the homogeneous case, while it is able to obtain a substantial improvement in the heterogeneous.

[0127] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0128] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other devices are used to relay the message from the source device to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more devices are used to relay the message from the sender to the receiving device). Further, as used herein “a” means “at least one” or “one or more.”

[0129] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration.Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (800) for determining the location of a device within an environment comprising a set of two or more planar surfaces, PSs, the set of two or more PSs comprising a first PS, PS A, and a second PS, PS_B, wherein a first set of points of interest, POIs, is associated with PS A and a second set of POIs is associated with PS_B, the method comprising: detecting (s802) a PS, PS D, in an image of the environment that was captured using an image sensor of the device; determining (s804) Pols associated with PS D using the captured image; using the Pols associated with PS D to determine (s806) the PS within the set of two or more PSs in the environment to which PS D corresponds; and after determining the PS within the set of PSs to which PS D corresponds, determining (s808) the location of the device within the environment, wherein using the Pols associated with PS D to determine the PS within the set of PSs of the environment to which PS D corresponds comprises: obtaining a first transform value using the determined Pols and the Pols associated with PS A; and determining whether the first transform value satisfies a first condition.

2. The method of claim 1, wherein the first transform value is derived from a rotation transform, or the first transform value is derived from a translation transform.

3. The method of claim 1 or 2, wherein using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds further comprises: obtaining a second transform value using the Pols associated with PS D and the Pols associated with PS A; and determining whether the second transform value satisfies a second condition.

4. The method of claim 1, 2, or 3, wherein the first transform value is derived from a rotation transform, anddetermining whether the first transform value satisfies the first condition comprises determining whether the first transform value satisfies a rotation condition.

5. The method of claim 3 or 4, wherein the second transform value is derived from a translation transform, and determining whether the second transform value satisfies the second condition comprises determining whether the translation transform value satisfies a translation condition.

6. The method of claim 2, 3, 4, or 5, further comprising: obtaining a homography matrix, H, using the Pols associated with PS D and the Pols associated with PS A.

7. The method of claim 6, wherein the first transform value is the rotation transform value, the method further comprises using H to obtain a rotation matrix, H r, which allows for predicting a rotation of the image sensor with respect to the PS, and the rotation transformation value is the determinant of H r.

8. The method of claim 6 or 7, further comprising: using H to obtain a transformation matrix, T, encoding a three-dimensional, 3D, rotation and a 3D translation vector, t, wherein the second transformation value is obtained using the translation vector t.

9. The method of claim 8, wherein the second transformation value is the norm of the translation vector t.

10. The method of any one of claims 1-9, wherein using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds further comprises: assigning a first match score to PS A based on a number of Pols associated with PS D that match one of the Pols associated with PS A.

11. The method of claim 10 wherein using the Pols associated with PS D to determine the PS within the set of PSs to which PS D corresponds comprises: assigning a second match score to PS_B based on a number of Pols associated with PS D that match one of the Pols associated with PS_B; determining that the first match score assigned to PS A is greater than the second match score assigned to PS_B; and as a result of determining the first match score is greater than the second match score, determining that PS D corresponds to PS A.

12. The method of claim 10 or 11, wherein assigning the first match score to PS A comprises: determining a total number of the Pols associated with PS D that match at least one Pol associated with PS A; and assigning the first match score based on the number, wherein there is a positive correlation between the first match score and the number.

13. The method of any of claims 1-12, wherein a number of Pols associated with PS D match at least one Pol associated with PS A, and using the Pols associated with PS D and the Pols associated with PS A to determine whether PS D corresponds to PS A further comprises: determining the number of Pols associated with PS D that match at least one Pol associated with PS A; and determining whether the number is greater than a threshold.

14. The method of any of claims 1-13, wherein the Pols associated with PS D are a first type of Pol, the Pols associated with PS A are a second type of Pol, and the first type of Pol and the second type of Pol are different.

15. The method of any of claims 1-14, wherein the set of two or more PSs are associated with a 3D map of the environment.

116. The method of claim 15, wherein the first set of Pols and the second set of POIs are included in the 3D map.

17. A computer program (943) comprising instructions (944), executable by processing circuitry (902) of an apparatus (900), for configuring the apparatus to perform the method of any one of claims 1-16.

18. A carrier containing the computer program of claim 17, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (942).

19. An apparatus (900) operable for determining the location of a device within an environment comprising a set of two or more planar surfaces, PSs, the set of two or more PSs comprising a first PS, PS A, and a second PS, PS_B, wherein a first set of points of interest, POIs, is associated with PS A and a second set of POIs is associated with PS_B, the apparatus comprising: a memory (942); and processing circuitry (902) coupled to the memory (942), wherein the apparatus (900) is configured to perform a method comprising: detecting (s802) a PS, PS D, in an image of the environment that was captured using an image sensor of the device; determining (s804) Pols associated with PS D using the captured image; using the Pols associated with PS D to determine (s806) the PS within the set of two or more PSs in the environment to which PS D corresponds; and after determining the PS within the set of PSs to which PS D corresponds, determining (s808) the location of the device within the environment, wherein using the Pols associated with PS D to determine the PS within the set of PSs of the environment to which PS D corresponds comprises: obtaining a first transform value using the determined Pols and the Pols associated with PS A; and determining whether the first transform value satisfies a first condition.

20. The apparatus of claim 19, wherein the apparatus is further configured to perform the method of any one of embodiments 2-16.

Images