Image-based positioning

Abstract

A method for image-based localization includes capturing, at a camera device, a plurality of images of a real-world environment. A first set of image features is detected in a first image of the plurality of images. The first set of image features is transmitted to a remote device configured to estimate a pose of the camera device based on image features detected in the plurality of images before additional sets of image features are detected in other images of the plurality of images. When the additional sets of image features are detected in the other images of the plurality of images, the additional sets of image features are transmitted to the remote device. An estimated pose of the camera device is received from the remote device.

Description

BACKGROUND

[0001] Mobile computing devices, robots, drones, and other devices can utilize image-based localization techniques to estimate their position and orientation in a physical environment. For example, one or more images captured by a device can be compared to a pre-computed map representing a 3D reconstruction of a space. SUMMARY

[0002] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is neither intended to identify key or essential features of the claimed subject matter nor is it used to limit the scope of the claimed subject matter. Further, there is no intention to use the

[0003] A method for image-based localization includes capturing, at a camera device, a plurality of images of a real-world environment. A first set of image features is detected in a first image of the plurality of images. The first set of image features is transmitted to a remote device configured to estimate a pose of the camera device based on image features detected in the plurality of images before a set of additional image features is detected in other images of the plurality of images. The set of additional image features is transmitted to the remote device when the set of additional image features is detected in the other images of the plurality of images. An estimated pose of the camera device is received from the remote device. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1A AND Figure 1B Capturing a plurality of images of a real-world environment is schematically illustrated.

[0005] Figure 2 An example method for image-based localization is illustrated.

[0006] Figure 3 Data exchange between a camera device and a remote device is schematically illustrated.

[0007] Figure 4 An example flowchart for image-based localization is illustrated.

[0008] Figure 5 An example of data exchanged between a camera device and a remote device during image-based localization is schematically depicted.

[0009] Figure 6 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0010] A wide variety of devices and related technologies can utilize image-based localization techniques. Examples can include wearable computing devices, such as head-mounted display (HMD) devices that provide augmented reality and / or virtual reality experiences, mobile devices (e.g., smartphones, tablets), robotic machines, and self-driving vehicles. These and other devices can use image-based localization techniques to determine their position and orientation (i.e., pose) in a real-world environment. Such devices are referred to herein as “camera devices.” While camera devices typically include or are configured to interface with at least one camera, camera devices can have any suitable hardware configuration.

[0011] Some approaches to image-based localization utilize prior knowledge of a digital three-dimensional (3D) map or scene model of a real-world environment. A digital environment map is essentially a spatial database that includes geometric data. In some examples, the geometric data includes points defined in three dimensions (e.g., x, y, z) distributed throughout a 3D space to form a 3D point cloud reconstruction of a real-world environment. The 3D points can be associated with feature descriptors (e.g., multi-dimensional vectors) extracted from source images or videos used to construct the map. Thus, the 3D points can be referred to as map features of the digital environment map.

[0012] Given a digital environment map of a real-world environment and one or more images or videos captured using a camera device, an image-based localization task can estimate the 3D position (x, y, z) and 3D orientation (roll, pitch, yaw) of the camera device within the real-world environment. Camera localization algorithms can solve this task in two steps—first matching 2D point features in the one or more captured images to 3D map features stored as 3D points in the 3D map. Next, the 6 degrees of freedom (6DOF) pose of the camera can be computed using the 2D point to 3D point matches and their underlying coordinates.

[0013] In some cases, camera localization can be performed by a device different from the camera device used to capture the images, thereby potentially saving processing, battery, and / or storage resources of the camera device. For example, images captured by a camera device can be transmitted to a remote device (e.g., a server) that has access to a digital environment map of a real-world environment and performs the processing steps for estimating the pose of the camera device. Moreover, transmitting the images to a remote device alleviates the need to share the digital environment map with third parties.

[0014] In this context, this disclosure primarily describes image-based localization, where images are transmitted from a camera device to a remote device, and the remote device then estimates the pose of the camera device. However, the image-based localization techniques described herein can be implemented on a single device that captures images of the real-world environment, maintains a digital environment map, and performs image-based localization to estimate its own pose.

[0015] While image-based localization can sometimes be performed on a single image, it is generally desirable to use several different images, either captured simultaneously (e.g., from different viewpoints) and / or sequentially (e.g., from the same viewpoint). This provides additional data that can be used to output a more accurate pose estimate for the camera device. For example, in some cases, a single image of a real-world environment may not contain (e.g., in the form of unique image features) enough detail to resolve a camera device to a particular pose with a reasonably high confidence level. By using more images of the real-world environment for image-based localization, the likelihood that the images collectively contain enough detail to estimate the camera device's pose with a relatively high confidence level is relatively high. This problem can be particularly pronounced in certain real-world environments. For example, an image taken in front of a building facade containing multiple rows of visually similar windows may be difficult to reconcile with any particular pose without additional detail.

[0016] However, performing image-based localization on several images of the same real-world environment can present different challenges. For example, in a single-image scene, a camera device can capture an image, extract image features from the image, and transmit the image (or image features) to a remote device, which can then estimate the camera device's pose. In contrast, in a multi-image scene, capturing multiple images, extracting image features from each of the several images, and transmitting each image (or set of image features) to a remote device can take relatively more time. Then, the remote device can take relatively longer to estimate the camera device's pose because more computation may be required when more image features are provided.

[0017] Therefore, this disclosure relates to an image-based localization technique that uses multiple images of a real-world environment to estimate the pose of a camera device. However, as will be discussed in more detail, embodiments of this disclosure can reduce the time required for the image-based localization process by performing the steps of the process in parallel rather than performing all steps in a strictly non-overlapping sequence. In other words, and in a non-limiting example, the camera device can capture multiple images of the real-world environment, begin extracting image features from specific images before capturing other images, and transmit these image features to a remote device as soon as they are detected, rather than after image features from all images have been extracted. Similarly, the remote device can begin attempting to estimate the pose of the camera device once it receives at least some image features, even while the camera device is still extracting additional image features from images of the real-world environment. It is worth noting that this can be done regardless of whether the image features are extracted at the camera device, the remote device, or other suitable device. Furthermore, even once a single image feature is detected, the remote device can begin attempting to estimate the pose of the camera device without waiting for each image feature to be detected in a single specific image.

[0018] In this way, image-based localization can achieve the advantages of using multiple images (e.g., more accurate pose estimation) while at least partially mitigating potential drawbacks (e.g., potentially longer latency before returning the estimated pose). For example, the image-based localization techniques described herein can improve the operational efficiency of camera devices by requiring fewer local processing steps on the camera device to accurately determine its pose. This can save electrical power and processing power from the camera device, allowing that power to be allocated to other tasks. Furthermore, the amount of time required to determine the camera device's pose can be reduced by the techniques described herein, since pose estimation can be generated before processing all captured images of the real-world environment. In some cases, pose estimation may even be generated before a single image of the real-world environment has been processed.

[0019] Figure 1A A user 100 is schematically depicted in a real-world environment 102, including structure 104. User 100 has a camera device 106 equipped with one or more cameras 108. In this example, the camera device takes the form of an HMD (Head-Down Display). However, as discussed above, the camera device can take any suitable form and include any suitable set of hardware resources.

[0020] Figure 1BMultiple images 110A to 110C depict a real-world environment 102. These images can be used to estimate the pose of a camera device relative to the real-world environment. When the camera device is an augmented reality or virtual reality device, this allows the camera device to display a virtual image with a desired real-world perceived positioning, for example, appearing as if it is in a fixed position within the real-world environment (i.e., world-locked), or appearing as if it is maintaining a fixed position relative to the user (i.e., body-locked). However, any suitable camera device can be used, and such a camera device can use the estimated camera pose for any suitable purpose.

[0021] Furthermore, image 110A is shown as including multiple image features represented by black circles 112. Such image features can be detected in an image in any suitable manner, as described in more detail below. Furthermore, as used herein, the term "image feature" refers to computer data that describes or otherwise represents one or more portions of an image. In other words, while black circles are used to indicate the location where an image feature is detected in image 110A, this is merely for visual clarity. In practical implementations, "image features" may simply describe the visual content of an image without being displayed at any point or otherwise graphically represented graphically.

[0022] As discussed above, images 110A to 110C of the real-world environment 102 captured by camera device 106 can be transmitted to a remote device (e.g., a server) for camera pose estimation. For example, the remote device may have previously acquired a digital environment map of the real-world environment, which includes multiple map features. These map features can be characterized as a 3D point cloud representing a reconstruction of the real-world environment. By determining the correspondence between map features in the digital environment map and image features detected in the images, the pose of the camera device when capturing images 110A to 110C can be inferred.

[0023] However, as discussed above, while using multiple images to estimate the camera's pose can increase the accuracy of the resulting pose estimate, it can also increase the total time required to complete the process. Therefore, Figure 2An example method 200 for image-based localization is illustrated, which, when used, can at least partially reduce the time required to estimate the pose of a camera device from multiple images. Method 200 involves communication between the camera device and a remote device. However, it is worth noting that both the camera device and the remote device can have any suitable hardware configuration and form factor. Furthermore, in many cases, the operations performed at the camera device and the remote device can occur substantially simultaneously. As discussed in more detail below, image capture, image feature detection, and camera pose estimation can at least partially overlap, such that the pose of the camera device is estimated before all real-world images have been captured, or all image features in the captured images have been detected.

[0024] Furthermore, the term "remote device" refers to any suitable computing device configured to engage with a camera device and perform image-based localization to estimate the camera device's pose. Any suitable physical separation may exist between the camera device and the remote device. Communication between the camera device and the remote device can be achieved in any suitable manner. As an example, the camera device and the remote device can use a wired communication coupling or a wireless connection via Wi-Fi, Bluetooth, or any other suitable wireless communication protocol. When communication is conducted over a network, the network can take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, a cellular network, or a combination thereof, and may include the Internet. Furthermore, as discussed above, in some cases, image-based localization can be performed on a single device (e.g., the camera device) without involving additional devices.

[0025] At point 202, method 200 includes: capturing multiple images of a real-world environment. This is relative to... Figure 1A and 1B The illustration shows that camera device 106 captures multiple images 110A to 110C of the real-world environment 102. Furthermore, Figure 3 An example camera device 300 is schematically shown, configured to use one or more cameras 302 to capture multiple images 304A and 304B.

[0026] In some examples, camera 302 may include one or more RGB cameras, IR cameras, and / or depth cameras (such as time-of-flight, structured light, or other forms of depth cameras). The camera device may include any suitable number of cameras, including single cameras, camera pairs (e.g., stereo camera pairs), camera arrays comprising multiple cameras, etc. Furthermore, the camera device may include multiple separate types of cameras, such as RGB cameras and depth cameras.

[0027] In some cases, multiple images captured by a camera device can be captured substantially simultaneously. For example, in a device with four onboard cameras, all four cameras can be activated substantially simultaneously, resulting in the simultaneous capture of four different images with different perspectives. These images can be optionally combined into a single composite image using appropriate image editing techniques. Given that the various hardware components of the camera device may operate at slightly different speeds or have slightly different associated latency requirements, it is not necessary to capture multiple images precisely simultaneously; rather, they can be captured close enough that the camera device's pose is unlikely to change meaningfully between capturing the various images. For example, within a span of a thousandth of a second, the camera device's pose is unlikely to change large enough to interfere with the downstream pose estimation step. In some implementations, images captured even a tenth of a second apart can be described as "substantially" captured, depending on the accuracy requirements of the final estimated pose.

[0028] When a camera device includes multiple cameras, the cameras can have any suitable spatial relationship with respect to each other and to other hardware components of the camera device. Typically, the spatial arrangement of the multiple cameras is known to the camera device in some way. For example, the camera device may include manufacturer calibration data indicating the relative 3D position of each of the multiple cameras. Alternatively or additionally, the spatial arrangement of the multiple cameras can be specified in another suitable manner—for example, by the user during a manual calibration process. In some cases, some information regarding the spatial arrangement of the multiple cameras can be transmitted to a remote device, as will be discussed in more detail below.

[0029] Additionally or alternatively, multiple images captured by a camera device in a real-world environment can be captured sequentially by the same camera and / or multiple different cameras. In some cases, the time interval between sequentially captured images can be relatively small—for example, on the order of a fraction of a second—although any suitable interval can be used. Furthermore, the interval between sequentially captured images does not need to be regular. In some cases, images can be captured sequentially and simultaneously—for example, a device equipped with multiple cameras can sequentially trigger two or more cameras simultaneously multiple times.

[0030] Return to temporarily Figure 2 At position 204, method 200 includes: detecting a first image feature set in a first image among a plurality of images. This in Figure 3 As shown, camera device 300 is illustrated as including image 304A and associated image feature set 306A. Furthermore, in Figure 1B In the image, the locations where image features have been detected are indicated by black circles 112. It should be understood that... Figure 1BEach black circle shown can have associated image features, and each individual image captured by the camera device can have any number (e.g., dozens, hundreds, thousands or more) of such features.

[0031] like Figure 1B As shown, the two-dimensional distribution of image features can reflect the geometric properties and visual appearance of objects existing in the physical environment at the time the image was captured. In this example, image features include features corresponding to the shape, dimensions, and other characteristics of structure 104.

[0032] Image features can take many suitable forms and can be detected in many suitable ways. As an example, an image feature can be a Scale Invariant Feature Transform (SIFT) feature. A SIFT feature includes SIFT keypoints that store geometric information associated with the feature, including its 2D location, scale, and one-dimensional angular orientation. A SIFT feature also includes a feature descriptor, implemented as a 128-dimensional vector. Multi-step processes can be used to detect SIFT features in an image. However, it should be understood that any suitable approach for extracting image features can be used, and such image features do not have to be SIFT features. As another example, the detected image feature can be a Speed-Up Robust Feature (SURF) feature. Typically, an image feature includes some description of the underlying image data associated with the image feature (e.g., expressed as a feature vector) and an indication of the two-dimensional (2D) pixel location of the image feature in the corresponding image of the real-world environment.

[0033] Return to temporarily Figure 2 At 206, method 200 includes transmitting a first set of image features to a remote device before an additional set of image features is detected in other images of a plurality of images captured by the camera device. It is noteworthy that the entire set of image features detected in the first image does not need to be transmitted to the remote device immediately. Instead, in some examples, transmission of image features can begin as soon as a single image feature is detected in the first image. Furthermore, the image features detected in the first image can be transmitted in any suitable order. This can include the order in which image features are detected, the order in which spatially adjacent image features are grouped together, the order in which image features are prioritized as more useful or discriminative for pose estimation (e.g., due to the “uniqueness” of each image feature), etc. Next, at 208, method 200 includes transmitting an additional set of image features to a remote device when the additional set of image features is detected in other images of a plurality of images. These additional sets of image features can also be transmitted at any time and in any order.

[0034] A remote device can be configured to estimate the camera device's pose based on image features detected in multiple images. Furthermore, the remote device can be configured to begin attempting to estimate the camera device's pose upon receiving any image features, even if such image features are insufficient to estimate a pose with sufficiently high confidence. Therefore, as the remote device continues to receive image features from the camera device, it can progressively estimate additional poses of the camera device until one or more poses with sufficiently high confidence are finally estimated for reporting to the camera device. Notably, in some cases, the remote device can estimate a sufficiently high-confidence pose of the camera device before receiving image features from all images captured by the camera device. Therefore, this saves time compared to other multi-image localization techniques, where pose estimation is not performed until image features are extracted and analyzed from each captured image.

[0035] This is Figure 3 The image is shown in the middle. Figure 3 In this embodiment, camera device 300 includes two distinct images 304A / 304B, each with an associated set of image features 306A / 306B. This information can be transmitted to remote device 308 for camera pose estimation. However, as discussed above, the camera device can begin transmitting image features to the remote device after detecting the first set of image features 306A or even a single image feature in image 304A, rather than waiting for the image features to be detected in each captured image. In other words, the remote device can begin attempting to estimate the pose of camera device 300 based on one or more image features 306A before detecting all image features 306B or even before capturing image 304B.

[0036] Figure 4 The diagram illustrates an example flowchart for image-based localization. At 400, camera device 300 captures images such as... Figure 1A and 1B The illustrated environment 102 consists of one or more images of the real-world environment. As discussed above, in some cases, camera device 300 may capture multiple images substantially simultaneously and / or sequentially capture one or more images. At 402, camera device 300 may optionally extract image features from the image captured at 400. It is worth noting that image features do not necessarily need to be detected at the camera device. In other words, image features can be detected at the camera device, a remote device, or any other suitable device.

[0037] continue Figure 4At point 404, camera device 300 transmits information to remote device 308. The specific information transmitted by the camera device can vary depending on whether image features are detected at the camera device or the remote device. If image features are detected at the camera device, the information transmitted by the camera device may include the image features, or may not include the actual image in which the features were detected. However, if image features are detected at the remote device, the camera device can transmit an image of the real-world environment without first detecting any image features.

[0038] Figure 5 A specific example of information 500 that can be transmitted from a camera device to a remote device is illustrated schematically. This information can be encapsulated or formatted in any suitable manner. Information 500 can be divided between any suitable number of discrete data packets or transmissions. Furthermore, it should be understood that... Figure 5 The specific data presented here is presented as a non-limiting example. In various implementations, the camera device may transmit more information, less information, and / or different information to a remote device.

[0039] exist Figure 5 In the example, information 500 includes an image 502 of the real-world environment captured by the camera device. However, depending on the implementation, the camera device may or may not transmit the complete captured image to the remote device. For example, if image features are detected at the camera device, the data transmitted from the camera device to the remote device may optionally omit the complete image of the real-world environment to facilitate the detection of image features in such an image. If image features are detected at the remote device, the camera device typically transmits an image of the real-world environment it has captured to the remote device so that the image features can be detected by the remote device.

[0040] Information 500 also includes an image feature set 504 corresponding to image 502. Therefore, in Figure 5 In the example, image features are detected by a camera device. Each image feature includes a descriptor 506 and a 2D pixel location 508 in which the image feature is detected in image 502. Each descriptor 506 includes some representation of the visual content of image 502 in which the corresponding image feature is detected—for example, in the form of a multidimensional feature vector.

[0041] Information 500 also includes camera information 510, which can refer to any suitable metric or specification applicable to the camera used by the camera device for capturing image 502. As an example, camera information 510 can specify the camera type (e.g., RGB, IR, brand / model), zoom parameters, spatial relationship of the camera relative to other cameras on the camera device, etc. For instance, as discussed above, in some cases, the camera device can be configured to capture multiple images of a real-world environment substantially simultaneously using multiple cameras. In such cases, the information transmitted to the remote device can include information indicating the spatial relationship of each of the respective different cameras on the camera device relative to each other. This can take any suitable form. For example, camera information 510 can specify the physical distance and orientation of the camera used to capture image 502 relative to other cameras and / or other suitable device components or reference frames on the camera device. As another example, camera information 510 can include a reference name or label for the camera used to capture image 502, such as "inner camera" or "left camera," which can be used when the remote device has information about the geometry and configuration of the camera device.

[0042] As shown, information 500 includes the capture time 512 of image 502. The capture time can be expressed in any suitable manner. For example, it can be relative to the system clock of the camera device, a reference time shared between the camera device and a remote device, or the time elapsed since the start of a particular process (e.g., the first transmission may occur at time 0), etc.

[0043] Information 500 also includes the relative pose 514 of the camera device per image during image capture 502. Typically, if the camera device sequentially captures images of the real-world environment, the camera device's pose may change between sequential image captures. Therefore, to facilitate more efficient pose estimation by a remote device, the camera device can track its own pose relative to some internal reference frame and transmit this information to the remote device. As used herein, the term "relative pose" refers to the pose tracked by the camera device, which typically does not fully resolve to an absolute position relative to the surrounding real-world environment. For example, after capturing a first image of the real-world environment, the camera device can track its own movement between capturing successive images of the real-world environment and transmit such information to a remote device. Therefore, "relative pose" differs from the "estimated pose" or "absolute pose" output by the remote device, which defines the camera device's position and orientation relative to a different reference frame corresponding to the real-world environment (e.g., a world-locked coordinate system or other suitable reference frame).

[0044] The relative orientation of the camera equipment can be determined and specified in any suitable manner. Figure 5Various examples of different types of information can be used by a camera device to track its movement as it captures images of the real-world environment. Specifically, information 500 includes data 516 from the camera device's inertial measurement unit (IMU), data 520 from a magnetometer, and data 522 from a Global Positioning System (GPS) receiver. As the camera device moves, its movement can be additionally or alternatively tracked based on the optical flow 518 observed by one or more cameras. For example, if the camera device moves while the surrounding real-world environment remains stationary, any changes in the size, location, or orientation of real-world objects visible to the camera device can be attributed to and used to estimate this movement. As an additional example, the camera device's movement can be estimated based on data 524 and / or 526 indicating changes in any Wi-Fi / cellular / other signals in the real-world environment, which can correspond to the camera device's movement.

[0045] It is worth noting that the relative pose of the camera device does not need to be calculated by the camera device itself. Instead, information used to calculate the relative pose (e.g., IMU data 516, optical flow data 518, magnetometer data 520, GPS data 522, Wi-Fi data 524, and / or cellular data 526) can be transmitted to another suitable device, which can use this information to calculate the relative pose of the camera device. Therefore, in some examples, even if a remote device is struggling to output the absolute pose of the camera device, the remote device can calculate the relative pose of the camera device based on information received from the camera device. It is also worth noting that the relative pose of the camera device can be calculated in any suitable manner and based on any suitable information, regardless of which device(s) performs the calculation. For example, in addition to or instead of any other relevant information, the remote device can calculate the relative pose of the camera device based on captured image features and / or any image features detected in the image.

[0046] Return to Figure 4After the camera device sends information to the remote device at 404, the camera device determines at 406 whether to capture more images of the real-world environment. If yes, the camera device captures another image at 400. If no, the camera device aborts image capture at 408. Image capture can be aborted after a predetermined number of images have been captured, after a predetermined amount of time has elapsed, after a user command, after the camera device enters a low-power or off mode, after the camera device receives an instruction from another device (e.g., a remote device), after the camera device receives an estimated pose exceeding a confidence threshold, and / or for any other suitable reason. Furthermore, after image capture is aborted at 408, a new image can subsequently be captured at 400 in response to any suitable trigger (e.g., the device pose estimate falls below a threshold confidence level).

[0047] Return to temporarily Figure 2 At 210, method 200 includes: receiving an estimated pose of a camera device. As discussed above, the estimated pose specifies the location and / or orientation of the camera device when capturing an image of a real-world environment.

[0048] Remote devices can estimate the camera's pose in any suitable way. Now, regarding... Figure 4 A specific process used for pose estimation is described. The data transmitted by the camera device at point 404 may include the information above. Figure 5 Following the information discussed, the remote device receives the information at 410. At 412, if image feature detection occurs on the remote device, the remote device can detect image features in the image of the real-world environment transmitted by the camera device. It is worth noting that image feature detection can be performed at the remote device even while the remote device is continuing to receive information from the camera device (e.g., additional images).

[0049] At point 414, the remote device will match image features detected in multiple images of the real-world environment with one or more map features in a digital environment map representing the real-world environment. Figure 3 In this diagram, remote device 308 is shown to include a digital environment map 310 and a corresponding set of map features 312. By identifying the correspondence between detected image features and map features of the digital environment map, the remote device can estimate the pose of the camera device, which is... Figure 3 The middle part is shown as attitude estimation 314.

[0050] In some examples, the map features of a digital environment map can take the form of a 3D point cloud representing a reconstruction of the real-world environment. A digital environment map can include any suitable number of points—in some examples, a 3D point cloud can include 20,000, 30,000 or more 3D points representing various locations on a surface in the real-world environment.

[0051] Such digital environment maps can be generated in any suitable manner. Generally, the real-world environment will be mapped using a suitable imaging device that captures multiple images or videos of the real-world environment from different locations. These images can be grayscale images, color images, depth images, etc. As discussed above, image features can be detected in each image. By detecting image features common to more than one captured image, spatial relationships between images can be determined. When the same image feature is detected in multiple images captured from different locations, the distance between each image feature and the imaging system can be inferred from the observed parallax offset. Additionally or alternatively, depth information from a depth camera can be used to identify the dimensions of the real-world environment and the distance between the imaging system and each image feature. In some implementations, the pose of the imaging system can be tracked as it captures multiple images, providing additional information that can be used to stitch the multiple images into a 3D representation of the real-world environment.

[0052] As noted above, each map feature in the map features is associated with a feature descriptor extracted from the source image used to construct the 3D map. The feature descriptor can include a multidimensional vector encoding the appearance of the surface surrounding the 3D map feature. For example, an image patch (such as 32×32 pixels) surrounding a given pixel where a map feature is detected can encode the appearance of the surface around that point into a more compact feature descriptor.

[0053] It is not necessary to generate feature descriptors for every 3D map feature in the 3D point cloud. In some examples, feature descriptors can be generated selectively as they differentiate map features (such as corners, high-texture areas, etc.). Additionally, in some examples, the initially generated point cloud can be extracted to reduce the density of map features included in the cloud. For example, a subset of representative map features can be selected from the point cloud, and unselected features within a predetermined distance of the selected 3D map features can be discarded. In this way, the resulting extracted 3D point cloud can be loaded more quickly, and the 3D cloud is easier for remote devices to process.

[0054] Additionally or alternatively, 3D maps may include other suitable information describing the real-world environment. As examples, such information may include lines, planes, and / or other higher-level objects; Wi-Fi / Bluetooth identifiers and / or signal strength; GPS coordinates; gravity vectors; fundamental direction vectors (e.g., magnetic north); and / or any other suitable information that may be used to determine the positioning of a camera device relative to the real-world environment.

[0055] Estimating the pose of a camera device can involve determining the correspondence between image features detected at 2D pixel locations in an image of a real-world environment and map features with 3D map locations in a digital environment map. This can be accomplished by searching the 3D map for feature descriptors associated with each image feature and identifying the 3D map features with the most similar feature descriptors in the 3D point cloud. In other words, determining the correspondence can include identifying a set of image features that have feature descriptors that match the feature descriptors of the 3D map features in the 3D map. Thus, 2D points detected in an image of a real-world environment correspond to 3D points associated with 3D map features, giving a set of 2D-to-3D point correspondences. This feature descriptor matching step can be implemented using one of many nearest neighbor matching techniques. The L2 distance between descriptor vectors can be used to calculate the feature similarity.

[0056] It is important to note that not all determined correspondences must be correct. In practice, a number of incorrect correspondences may be identified. This is because the underlying feature descriptors may be ambiguous, and it does not always allow for sufficient differentiation between feature descriptors of image features and multiple feature descriptors associated with 3D points in the map.

[0057] Furthermore, each image of the real-world environment can include items, objects, people, etc., that do not exist when mapping the real-world environment. Therefore, one or more image features can be detected in various images captured by camera devices that are not represented in the digital environment map. Typically, such image features do not correspond to any 3D map features. This can be advantageous in some scenarios because any privacy details depicted in multiple images may not correspond to any 3D map features in the digital environment map, thus limiting the amount of information about privacy details that a remote device can infer. The map may also contain a certain number of map features that do not match any of the image features in the multiple images.

[0058] Return to Figure 4Once the correspondence between image features and map features is identified at 414, this correspondence is passed to another process or application configured to estimate the pose of the camera device at 416. It is worth noting that, as with other steps in the image-based localization process described herein, image features can be matched with map features and passed to pose estimation at any appropriate time. In other words, the identified correspondence can be passed to pose estimation before matching or even extracting all image features from a particular image, before transmitting or capturing all images of the real-world environment, etc.

[0059] Thus, at point 418, the remote device determines whether there are additional image features to match with map features in the digital environment map. If so, the remote device matches such image features at point 414. If not, the remote device aborts image feature matching at point 420. It is important to note that aborting image feature matching at point 420 does not signify the end of the entire image-based localization process, as any matched image features can be used to estimate the camera device's pose. Furthermore, if additional images and / or image features are received from the camera device, the remote device can resume matching image features at point 414.

[0060] Once the correspondence between image features and map features is identified, this correspondence can be used at position 422 to estimate the camera pose. Pose estimation can be performed in any suitable manner. In some examples, estimating the camera pose can be an iterative randomization process where correspondences between 2D pixel locations of image features and 3D map locations of map features are identified for multiple candidate camera poses, and the consistency of these candidate camera poses with the underlying correspondence data is evaluated. In one example process, the Random Sample Consensus (RANSAC) method can be used to estimate the camera pose based on the determined correspondences. In such cases, a minimal or non-minimal solver can be used to compute multiple candidate camera poses based on a subset of the determined correspondences. However, it should be understood that other suitable methods can be used to estimate camera poses without using the RANSAC framework.

[0061] First, a remote device can identify a subset of correspondences from the entire defined set of 2D image feature to 3D map feature correspondences. In some cases, the number of correspondences in the identified subset can be the minimum number of data points required to compute a finite number of solutions using a method known as a minimal solver. For example, when fitting a 2D line to a set of 2D points, at least two unique data points are needed to find a unique 2D line that passes exactly through the 2D points (a single solution). On the other hand, a single data point can correspond to an infinite number of potential solutions. Therefore, for this line fitting example, two data points are the minimum number required by the minimal solver. In contrast, a non-minimal solver can use more than two data points.

[0062] Continuing with the 2D line fitting example, multiple subsets of 2D data point pairs can be selected. In the RANSAC framework, these subsets are randomly selected. If a minimum solver is applied to each subset (i.e., 2D point pairs), different latent solutions can be found from each subset. However, some latent solutions (2D lines) are more consistent with the input data than others. In other words, some latent lines pass through more 2D points in the dataset than others. In RANSAC, for each latent solution identified by the minimum solver, the solution (i.e., the 2D line) is compared with other 2D data points in the dataset to determine which points are consistent with the proposed solution (i.e., “in-bound” values) and which points are inconsistent with the proposed solution (i.e., “out-of-bound” values). Any solution with the highest ratio of in-bound to out-of-bound values ​​or with at least a threshold number of all in-bound values ​​can be accepted as the actual solution—for example, the 2D line best suited to the set of 2D points.

[0063] Returning to the camera pose estimation problem, depending on the available information, the RANSAC solver can output a finite number of solutions or potential pose candidates from a subset of identified correspondences. In other words, for each subset of identified correspondences, the remote device can compute a set of candidate camera poses that satisfy the subset of correspondences.

[0064] When some information about the camera's pose is known, the number of correspondences required by the minimum solver can be reduced in some cases. For example, when capturing multiple images of a real-world environment, the camera device can detect the gravity vector relative to its current pose (e.g., via an onboard 3-axis accelerometer). In some cases, this gravity vector can be included in the information transmitted from the camera device to a remote device. Therefore, the two degrees of freedom of the camera's 3D orientation (corresponding to roll and pitch) may be known, thus reducing the complexity of estimating the remaining degrees of freedom.

[0065] In addition, multiple images of the real-world environment are captured sequentially and transmitted by the camera device as described above. Figure 5In the case of relative pose information discussed, this relative pose information can be used when estimating the pose of the camera device. In other words, the remote device can use relative pose information to further narrow down the potential set of estimated poses of the camera device, rather than independently estimating the pose of each image captured by the camera device. For example, the high-confidence pose of the camera device is consistent with each image transmitted to the remote device and the movement of the camera device between image captures.

[0066] After the candidate camera poses for each subset of correspondences are calculated, the remote device can identify the optimal overall camera pose from each candidate camera pose set. In other words, the remote device identifies several camera pose hypotheses, at least one of which is likely to correspond to the actual pose of the camera. Generally, the optimal overall camera pose is more consistent with the overall correspondence set compared to other candidate camera poses. In other words, the remote device can determine, for each candidate camera pose, how many of the identified correspondences are in-bound and how many are out-of-bounds, and can identify the optimal overall camera pose based on this.

[0067] Depending on the implementation and the specific pose estimation technique used, each estimated camera device pose may include a corresponding confidence value. In the case of RANSAC, this confidence value may be proportional to the in-bounds / out-of-bounds ratio described above, where pose estimates with more in-bounds values ​​(e.g., more correspondences consistent with the pose than inconsistent ones) have relatively higher confidence values. Furthermore, the remote device does not need to estimate a single pose of the camera device and then stop. Instead, as the remote device receives additional image and / or image features from the camera device, it can progressively output multiple estimated poses of the camera device based on the identified correspondences between image features and map features. Since receiving additional image features typically provides the remote device with more data that can be used to generate more accurate pose estimates, each estimated pose may have a corresponding confidence value proportional to the amount of correspondence identified when each corresponding estimated pose is output.

[0068] Therefore, at various times, the remote device can generate any number of different pose estimates for the camera device, where any or all of these pose estimates can be transmitted to the camera device. In other words, in some cases, the estimated pose received by the camera device can be a preliminary estimated pose received later, before receiving a final pose estimate with higher confidence. The preliminary estimated pose can be generated by the remote device and received by the camera device at any suitable time, including before the camera device completes capturing images of the real-world environment, before detecting the image feature set of each image of the real-world environment captured by the camera device, and before all detected image features are matched with corresponding map features, etc.

[0069] In some examples, the remote device can output the camera device's pose estimate before even a single image of the real-world environment has been fully analyzed to detect image features. In other words, when capturing a single image of the real-world environment, image features can be extracted from the image at either the camera device or the remote device. The remote device can then begin attempting to determine the absolute pose of the camera device before each image feature in the image is detected or otherwise received by the remote device. In this way, the image-based localization techniques discussed in this paper can produce pose estimates faster than conventional single-image pose estimation techniques, which typically do not begin generating pose estimates until a single captured image has been fully analyzed to detect image features.

[0070] As discussed above, before the initial pose estimate is estimated, the initial pose estimate can have an associated confidence value proportional to the amount of image features transmitted to the remote device. However, whether the initial pose estimate is transmitted to the camera device can depend on whether the confidence value associated with the initial pose estimate exceeds one or more confidence thresholds.

[0071] Return to Figure 4 After the remote device estimates the camera device pose at point 422, it determines at point 424 whether the estimated pose exceeds a final confidence threshold. This threshold may be relatively high, such that any estimated pose with a confidence value higher than the final confidence threshold is designated as the final estimated pose, at which point image-based localization can be aborted.

[0072] If the confidence value of the estimated pose does not exceed the final confidence threshold, the remote device can determine whether the confidence value exceeds one or more lower confidence thresholds. For example, at 426, the remote device determines whether the estimated pose exceeds the preliminary confidence threshold. If not, at 428, the estimated pose is not sent to the camera device, and the remote device can optionally continue generating pose estimates until a pose exceeding the preliminary confidence threshold is estimated or the process stops for another reason. If yes, the estimated pose can be designated as the preliminary estimated pose, sent to the camera device at 430, and received by the camera device at 432.

[0073] It is worth noting that even after sending the initial pose estimate to the camera device, the remote device can continue to receive images and / or image features from the camera device and continue to generate new pose estimates. Therefore, even after receiving the initial pose estimate, the camera device can receive one or more additional estimated poses, which have been estimated based on more detected image features than the initial estimated pose. Thus, the one or more additional estimated poses can have correspondingly higher confidence values. Finally, the remote device can estimate the pose of the camera device at point 424 with a confidence value exceeding the final confidence threshold. This final estimated pose can then be sent to the camera device at point 434 and received at the camera device at point 436.

[0074] It is worth noting that the actual pose of the camera device can change between capturing images of the real-world environment and receiving the corresponding pose estimate. In other words, by the time the camera device receives the estimated pose from the remote device, the estimated pose may no longer be accurate due to movement of the camera device. Therefore, in some scenarios, the remote device can report the estimated camera pose along with a timestamp or other time-lapse indicator of the capture time relative to one or more images of the real-world environment on which the estimated pose is based. The camera device can then update the estimated pose based on its own internal motion tracking data, which includes, for example, the information mentioned above... Figure 5 Any relative attitude data described in the relative attitude data. For example, if a remote device indicates that the estimated attitude is specific to a point in time 0.25 seconds ago and the camera device determines that it has moved forward 3 mm within that time, the estimated attitude can be adjusted forward by 3 mm.

[0075] Additionally or alternatively, the remote device may anticipate the movement of the camera device when estimating its pose. For example, based on information received from the camera device (e.g., relative pose information) and / or information inferred during pose estimation, the remote device may determine that the camera device is moving at a specific rate in a specific direction. By using the measured or estimated network delay between the remote device and the camera device, the remote device may determine, for example, that it will take 0.25 seconds for the estimated pose to reach the camera device. Therefore, the remote device may modify its estimated pose to account for the movement of the camera device during the 0.25-second delay window. For example, if the remote device determines that the camera device is translating to the right at a specific rate, the remote device may estimate the pose of the camera device, which includes an extrapolated rightward movement proportional to the observed network delay. Similar extrapolation may be used to update the estimated pose to account for the movement of the camera device since the capture time of the image of the real-world environment on which the estimated pose is based.

[0076] After the final pose estimate is received, the camera device can terminate the image-based localization process in certain situations. For example, the camera device can interrupt the acquisition of new images of the real-world environment, the detection of image feature sets, or the transmission of image feature sets to a remote device, and the reception of pose estimates from the remote device. Similarly, the remote device can stop receiving images and / or image features from the camera device and can stop generating new pose estimates. At this point, tracking the pose of the camera device can be done by the camera itself—for example, via a suitable positioning / or orientation sensor of the camera device.

[0077] However, in other cases, image-based localization does not need to stop after any particular estimated pose is sent to the camera device. Instead, image-based localization can continue indefinitely—for example, as long as the camera device is in use, it can continue to capture new images of the real-world environment and receive new pose estimates from a remote device. This can facilitate potentially high-precision pose tracking for camera devices lacking other onboard motion sensors such as an IMU. Alternatively, image-based localization can be selectively aborted and resumed—for example, at any time when the camera device is uncertain of its absolute pose, a remote device can be used to re-establish a more certain camera pose.

[0078] The methods and processes described herein can be attached to a computing system of one or more computing devices. Specifically, such methods and processes can be implemented as executable computer applications, network-accessible computing services, application programming interfaces (APIs), libraries, or combinations of the above and / or other computing resources.

[0079] Figure 6A simplified representation of a computing system 600 is schematically illustrated, which is configured to provide any to all computing functions described herein. The computing system 600 may take the form of one or more camera devices, virtual / augmented / mixed reality computing devices, mobile communication devices (e.g., smartphones), tablet computers, home entertainment computers, gaming devices, mobile computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, personal computers, network-accessible server computers, and / or other computing devices. Non-limiting examples of the computing system 600 may include camera devices 106 and 300 and remote device 308.

[0080] The computing system 600 includes a logic subsystem 602 and a storage subsystem 604. The computing system 600 may optionally include a display subsystem 606, an input subsystem 608, a communication subsystem 610, and / or... Figure 6 Other subsystems not shown.

[0081] Logical subsystem 602 includes one or more physical devices configured to execute instructions. For example, the logical subsystem may be configured to execute instructions as part of one or more applications, services, or other logical constructs. The logical subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logical subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logical subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logical subsystem may optionally be distributed across two or more separate devices that may be remotely located and / or configured for coordinated processing. Aspects of the logical subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0082] Storage subsystem 604 includes one or more physical devices configured to temporarily and / or permanently store computer information, such as data and instructions executable by the logical subsystem. When the storage subsystem includes two or more devices, these devices may be co-located and / or remotely positioned. Storage subsystem 604 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices. Storage subsystem 604 may include removable and / or built-in devices. The state of storage subsystem 604 can be transformed—for example, to maintain different data—when the logical subsystem executes instructions.

[0083] Various aspects of the logic subsystem 602 and the storage subsystem 604 can be integrated into one or more hardware logic components. Such hardware logic components may include, for example, application-specific integrated circuits (PASIC / ASIC), application-specific standard products (PSSP / ASSP), system-on-a-chip (SOC), and complex programmable logic devices (CPLD).

[0084] The logic subsystem and storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used collectively to refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functionality. In other words, "machine" is never an abstract concept but always has a tangible form. A machine can be instantiated by a single computing device, or a machine can include two or more sub-components instantiated by two or more different computing devices. In some implementations, a machine includes local components (e.g., software applications executed by a computer processor) that collaborate with remote components (e.g., cloud computing services provided by a network of server computers). The software and / or other instructions that give a particular machine its functionality can optionally be stored as one or more unexecuted modules on one or more suitable storage devices.

[0085] When included, display subsystem 606 can be used to present a visual representation of the data held by storage subsystem 604. This visual representation may take the form of a graphical user interface (GUI). Display subsystem 606 may include one or more display devices utilizing virtually any type of technology. In some implementations, display subsystem may include one or more virtual reality displays, augmented reality displays, or mixed reality displays.

[0086] When included, the input subsystem 608 may include or be coupled to one or more input devices. Input devices may include sensor devices or user input devices. Examples of user input devices include a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or be coupled to a selected Natural User Input (NUI) component system. Such a component system may be an integrated component system or a peripheral component system, and may handle the conversion and / or processing of input actions on-board or off-board. Example NUI component systems may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; and head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition.

[0087] When included, the communication subsystem 610 can be configured to communicatively couple the computing system 600 to one or more other computing devices. The communication subsystem 610 may include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal area network, a local area network, and / or a wide area network.

[0088] The methods and processes disclosed herein can be configured to give users and / or any other person control over any privacy-sensitive and / or potentially sensitive data, including, for example, images captured by a camera device and estimated camera device pose. Data can be disposed of in accordance with privacy and / or security standards whenever it is stored, accessed, and / or processed. When user data is collected, the user or other stakeholders can specify how the data is used and / or stored. User data can only be collected with the utmost respect for user privacy whenever it is collected for any purpose (e.g., user data can only be collected if the user who owns the data provides affirmative consent and / or the user who owns the data is likely to be notified when the user data is collected). If the data is to be accessed by anyone other than the user or used in any decision-making process, the user's consent may be obtained before the data is used and / or published. Users can opt in and / or opt out of data collection at any time. After data has been collected, users can issue commands to delete the data and / or restrict access to the data. All potentially sensitive data may optionally be encrypted and / or anonymized where feasible to further protect user privacy. Users can specify data, metadata, or portions of statistics / results from processed data for distribution to other parties, for example, for further processing. Data can be kept completely private and / or confidential, for example, by temporarily decrypting it only for processing, or by decrypting it only for processing on the user's device and otherwise storing it in encrypted form. Users can store and control the encryption keys for encrypted data. Alternatively or additionally, users can designate a trusted third party to store and control the encryption keys for encrypted data, for example, to provide users with access to the data according to appropriate authentication protocols.

[0089] In one example, a method for image-based localization includes: capturing multiple images of a real-world environment at a camera device; detecting a first set of image features in a first image of the multiple images; transmitting the first set of image features to a remote device, configured to estimate the pose of the camera device based on the image features detected in the multiple images, before an additional set of image features is detected in other images of the multiple images; transmitting an additional set of image features to the remote device when the additional set of image features is detected in other images of the multiple images; and receiving the estimated pose of the camera device from the remote device. In this example or any other example, multiple images of the real-world environment are captured substantially simultaneously. In this example or any other example, each of the multiple images is captured by a different corresponding camera of the camera device, and the method further includes: transmitting information to the remote device indicating the spatial relationship of each corresponding camera among the different corresponding cameras of the remote device relative to each other. In this example or any other example, multiple images of the real-world environment are captured sequentially. In this example or any other example, the method further includes: transmitting information to the remote device about the movement of the camera device between the capture of the sequential images of the multiple images. In this example or any other example, the estimated pose of the camera device is estimated based on the correspondence between identified image features transmitted to the remote device and map features in a digital environment map corresponding to the real-world environment. In this example or any other example, each detected image feature includes an indication of a 2D pixel location at which the corresponding image feature in the real-world environment is identified, and the identified correspondence lies between the 2D pixel location of the image feature transmitted to the remote device and the 3D map location of the map feature in the digital environment map. In this example or any other example, the identified correspondence between the 2D pixel location and the 3D map location is input to a Random Sample Consensus (RANSAC) solver, which outputs the estimated pose of the camera device. In this example or any other example, the estimated pose of the camera device is a preliminary estimated pose and is received before an additional set of image features for each of multiple images is detected, and the preliminary estimated pose has an associated confidence value proportional to the amount of image features transmitted to the remote device before estimating the preliminary estimated pose. In this example or any other example, the method further includes: receiving one or more additional estimated poses after the initial estimated pose, the additional estimated poses being estimated based on more detected image features than the initial estimated pose and having correspondingly higher confidence values. In this example or any other example, the method further includes: receiving a final estimated pose with a corresponding confidence value exceeding a final confidence threshold, and after receiving the final estimated pose, suspending the transmission of the additional image feature set to the remote device.

[0090] In one example, a computing device includes one or more cameras; and a logic machine configured to: capture multiple images of a real-world environment via the one or more cameras; detect a first set of image features in a first image of the multiple images; transmit the first set of image features to a remote device before additional set of image features are detected in other images of the multiple images, the remote device being configured to estimate the pose of the camera device based on the image features detected in the multiple images; transmit additional set of image features to the remote device when additional set of image features are detected in other images of the multiple images; and receive the estimated pose of the camera device from the remote device. In this example or any other example, different corresponding cameras of the camera device capture multiple images of the real-world environment substantially simultaneously, and the logic machine is also configured to transmit information to the remote device indicating the spatial relationship of each of the different corresponding cameras of the camera device relative to each other. In this example or any other example, multiple images of the real-world environment are captured sequentially, and the logic machine is also configured to transmit information to the remote device regarding the movement of the camera device between the capture of the sequential images of the multiple images. In this example or any other example, the estimated pose of the camera device is estimated based on the correspondence between the image features transmitted to the remote device and the identified map features in a digital environment map corresponding to the real-world environment. In this example or any other example, each detected image feature includes an indication of a 2D pixel location at which the corresponding image feature in the real-world environment is identified, and the identified correspondence lies between the 2D pixel location of the image feature transmitted to the remote device and the 3D map location of the map feature in the digital environment map. In this example or any other example, the estimated pose of the camera device is a preliminary estimated pose and is received before an additional set of image features is detected in each of a plurality of images. In this example or any other example, the preliminary estimated pose has an associated confidence value proportional to the amount of image features transmitted to the remote device before estimating the preliminary estimated pose, and the logic is further configured to receive one or more additional estimated poses after the preliminary estimated pose, which are estimated based on more detected image features than the preliminary estimated pose and have correspondingly higher confidence values. In this example or any other example, the logic machine is also configured to receive a final estimated pose with a corresponding confidence value exceeding the final confidence threshold, and to abort the transmission of additional image feature sets to the remote device after receiving the final estimated pose.

[0091] In one example, a computing device includes a logic machine configured to: receive from a camera device multiple images of a real-world environment captured by one or more cameras of the camera device; as the multiple images of the real-world environment are received, detect a set of image features in each of the multiple images; as the set of image features is detected, identify a correspondence between the detected image features and map features of a digital environment map corresponding to the real-world environment; as the correspondence is identified, progressively output multiple estimated poses of the camera device based on the identified correspondences, wherein each estimated pose has a corresponding confidence value proportional to the amount of the correspondence identified when each corresponding estimated pose is output; report a first pose to the camera device after estimating a preliminary pose of the camera device having a corresponding confidence value exceeding a preliminary confidence threshold and before detecting the set of image features in each of the multiple images; and report a second pose to the camera device after estimating a second pose of the camera device having a corresponding confidence value exceeding a final confidence threshold higher than the preliminary confidence threshold.

[0092] This invention is presented by way of example and with reference to the accompanying drawings. Components, process steps, and other elements that may be substantially the same in one or more of the drawings are identified in a harmonized manner and described with minimal repetition. However, it should be noted that elements identified in a harmonized manner may also differ to some extent. It should also be noted that some drawings may be schematic and not drawn to scale. Various drawing scales, aspect ratios, and numbers of components shown in the drawings may be intentionally distorted to make certain features or relationships easier to view.

[0093] It should be understood that because many variations are possible, the configurations and / or methods described herein are exemplary in nature, and these particular embodiments or examples should not be construed as limiting. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in a different order, in parallel, or omitted. Similarly, the order of the processes described above may be changed.

[0094] The subject matter of this disclosure includes all novel and obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.

Claims

1. A method for image-based localization, comprising: At the camera device, capture multiple images of the real-world environment; Detect a first image feature set in the first image among the plurality of images; Before the additional image feature set is detected in the additional images among the plurality of images, the first image feature set is transmitted to a remote device configured to estimate the pose of the camera device. The remote device receives a first estimated pose of the camera device, the first estimated pose having been estimated based on a set of first image features detected in the first image among the plurality of images; When the additional image feature set is detected in the additional image among the plurality of images, the additional image feature set is transmitted to the remote device; as well as The remote device receives a second estimated pose of the camera device, the second estimated pose having been estimated based on a set of multiple image features detected in two or more of the plurality of images, and the second estimated pose having a correspondingly higher confidence value; The first estimated pose of the camera device is a preliminary estimated pose, and is received before the additional set of image features for each of the plurality of images is detected, and the preliminary estimated pose has an associated confidence value that is proportional to the amount of image features transmitted to the remote device before the preliminary estimated pose is estimated. as well as Receive the final estimated pose with a corresponding confidence value exceeding the final confidence threshold, and after receiving the final estimated pose, stop transmitting the additional image feature set to the remote device.

2. The method of claim 1, wherein the plurality of images of the real-world environment are captured substantially simultaneously.

3. The method of claim 2, wherein each of the plurality of images is captured by a different corresponding camera of the camera device, and the method further comprises: Information is transmitted to the remote device, the information indicating the spatial relationship of each of the different corresponding cameras of the camera device relative to each other.

4. The method of claim 1, wherein the plurality of images of the real-world environment are captured sequentially.

5. The method according to claim 4, further comprising: Information about the movement of the camera device between the capture of sequential images in the plurality of images is transmitted to the remote device.

6. The method of claim 1, wherein the first estimated pose of the camera device is estimated based on the correspondence between the image features transmitted to the remote device and the map features in a digital environment map corresponding to the real-world environment.

7. The method of claim 6, wherein each detected image feature includes an indication of a two-dimensional 2D pixel location, wherein at the 2D pixel location, the image feature is identified in a corresponding image of the real-world environment, and wherein the identified correspondence is between the 2D pixel location of the image feature transmitted to the remote device and the three-dimensional 3D map location of the map feature in the digital environment map.

8. The method of claim 7, wherein the correspondence between the identifiers of the 2D pixel positioning and the 3D map positioning is input into a random sample consistent RANSAC solver, the RANSAC solver outputting the estimated pose of the camera device.

9. The method of claim 1, wherein the second estimated pose is estimated based on more detected image features than the initial estimated pose.

10. A camera device, comprising: One or more cameras; as well as The logic machine is configured as follows: Multiple images of the real-world environment are captured via the one or more cameras; Detect a first image feature set in the first image among the plurality of images; Before the additional image feature set is detected in the additional images among the plurality of images, the first image feature set is transmitted to a remote device configured to estimate the pose of the camera device. The camera device receives a preliminary estimated pose from the remote device, the preliminary estimated pose having been estimated based on a first set of image features detected in the first image among the plurality of images; the preliminary estimated pose is received before the additional set of image features for each of the plurality of images is detected, and the preliminary estimated pose has an associated confidence value proportional to the amount of image features transmitted to the remote device before the preliminary estimated pose is estimated; When the additional image feature set is detected in the additional image among the plurality of images, the additional image feature set is transmitted to the remote device; The remote device receives a second estimated pose of the camera device, the second estimated pose having been estimated based on more detected image features than the initial estimated pose, and the second estimated pose having a correspondingly higher confidence value; as well as Receive the final estimated pose with a corresponding confidence value exceeding the final confidence threshold, and after receiving the final estimated pose, stop transmitting the additional image feature set to the remote device.

11. The camera device of claim 10, wherein the plurality of images of the real-world environment are captured substantially simultaneously by different corresponding cameras of the camera device, and the logic machine is further configured to transmit information to the remote device, the information indicating the spatial relationship of each of the different corresponding cameras of the camera device relative to each other.

12. The camera device of claim 10, wherein the plurality of images of the real-world environment are captured sequentially, and the logic machine is further configured to transmit information to the remote device regarding the movement of the camera device between the capture of the sequential images in the plurality of images.

13. The camera device of claim 10, wherein the preliminary estimated pose of the camera device is estimated based on the correspondence between the image features transmitted to the remote device and the map features in a digital environment map corresponding to the real-world environment.

14. The camera device of claim 13, wherein each detected image feature includes an indication of a two-dimensional 2D pixel location, wherein at the 2D pixel location, the image feature is identified in a corresponding image of the real-world environment, and wherein the identified correspondence is between the 2D pixel location of the image feature transmitted to the remote device and the three-dimensional 3D map location of the map feature in the digital environment map.

15. A computing device, comprising: The logic machine is configured as follows: Receive multiple images of the real-world environment of the camera device captured by one or more cameras of the camera device; When the plurality of images of the real-world environment are received, a set of image features in each of the plurality of images is detected; When the image feature set is detected, the correspondence between the detected image features and the map features of the digital environment map corresponding to the real-world environment is identified; When the correspondence is identified, multiple estimated poses of the camera device are output progressively based on the identified correspondence, wherein each estimated pose has a corresponding confidence value, the confidence value being proportional to the amount of the identified correspondence when each corresponding estimated pose is output; After estimating the initial pose of the camera device having a corresponding confidence value that exceeds the initial confidence threshold, and before detecting the set of image features for each of the plurality of images, the initial pose is reported to the camera device. as well as After estimating a second pose of the camera device having a corresponding confidence value exceeding a final confidence threshold higher than the initial confidence threshold, the second pose is reported to the camera device.

Images