Method and apparatus for determining and / or evaluating a localization map of an image display device - Patents.com

Abstract

To provide a method and apparatus for determining and / or evaluating a localizing map of an image display device.SOLUTION: An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, the map being configured for use by the processing unit to localize the user with respect to the environment. The processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map. The processing unit is configured to obtain the metric by computing the metric or by receiving the metric.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present disclosure relates to an image display device configured to be worn on a user's head, and to methods and apparatus for determining and evaluating a localization map for such an image display device. [Background technology]

[0002] Modern computing and display technologies have facilitated the development of so-called "mixed reality" (MR) systems for "virtual reality" (VR) or "augmented reality" (AR) experiences, where digitally reproduced images or parts thereof are presented to a user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user (i.e., transparency to real-world visual input). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0003] MR systems can generate and display color data, which increases the realism of the MR scenario. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can result in a smooth color MR scenario in the user's memory.

[0004] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility Application No. 14 / 555,585, filed November 27, 2014 (Attorney Docket No. ML.20011.00), the contents of which are expressly and fully incorporated herein by reference in their entirety as if fully set forth.

[0005] The MR system may employ at least a wearable display device (e.g., a display configured to be worn on the head, a helmet-mounted display, or smart glasses) that is loosely coupled to the user's head and thus moves as the user's head moves. When the user's head movement is detected by the display device, the data being displayed may be updated (e.g., "warped") to account for changes in head pose (i.e., the orientation and / or location of the user's head).

[0006] As an example, when a user wearing a display device views a virtual representation of a virtual object on the display and walks around the area in which the virtual object appears, the virtual object can be rendered per viewpoint, giving the user the perception of walking around an object that occupies real space. When a display device is used to present multiple virtual objects, head pose measurements can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion.

[0007] To enable AR, a display device (configured to be worn on the user's head) provides a simultaneous view of both real and virtual objects. With an "optical see-through" display, the user can look through a transparent (or semi-transparent) element in the display system and directly view light from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's view of the real world, and the light from the display projects an image of the virtual content over the see-through view of the real objects in the environment. A camera may be mounted on the display device to capture an image or video of the scene being viewed by the user.

[0008] Current optical systems, such as those in MR systems, optically render virtual content. The content is "virtual" in that it does not correspond to actual physical objects located at discrete positions in space. Instead, the virtual content exists only in the brain (e.g., visual centers) of the user of the display device when stimulated by light beams directed at the user's eyes.

[0009] In some cases, an image display device configured to be worn on a user's head may display virtual objects relative to a real environment and / or allow the user to place and / or manipulate virtual objects relative to the real environment. In such cases, the image display device may be configured to localize the user relative to the real environment such that the virtual objects may be correctly displaced relative to the real environment. Disclosed herein are methods and apparatus for determining and / or evaluating a localization map of an image display device (e.g., an MR device, an AR device, a VR device, etc.). The localization map is configured for use by the image display device for localization of the user. Summary of the Invention [Means for solving the problem]

[0010] The device, configured to be worn on a user's head, includes a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit configured to determine a map based, at least in part, on output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment, the processing unit of the device also configured to determine a metric indicative of the likelihood of success of locating the user using the map.

[0011] Optionally, the processing unit may be configured to determine the metric by calculating the metric.

[0012] Optionally, the processing unit may be configured to determine the metric by receiving the metric.

[0013] Optionally, the processing unit is configured to determine a metric based on the joint visibility of points of interest associated with different camera positions.

[0014] Optionally, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0015] Optionally, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0016] Optionally, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0017] Optionally, the metric indicates the likelihood of success in locating the user in one or more viewing directions.

[0018] Optionally, the processing unit is configured to determine a metric based on the number of times the point of interest is detected from different camera positions.

[0019] Optionally, the processing unit is configured to determine the metric without determining any convex hull.

[0020] Optionally, the metric has a value that is based on the directionality.

[0021] Optionally, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0022] Optionally, the directionality includes a turning direction.

[0023] Optionally, the directivity includes a tilt angle.

[0024] Optionally, the directivity includes a roll angle.

[0025] Optionally, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0026] Optionally, the metric is for one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a value based on a position within the one of the plurality of cells.

[0027] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0028] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment.

[0029] Optionally, the processing unit is also configured to determine a total number of images from the camera system for one of the plurality of cells.

[0030] Optionally, the total number of images is associated with a certain viewing direction for the cell.

[0031] Optionally, the total number of images is associated with a number of viewing directions for the cell.

[0032] Optionally, the camera system is configured to acquire a plurality of images, and the processing unit is configured to determine a metric for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0033] Optionally, the processing unit is also configured to determine an average score by dividing the total score by the number of images in the subset of images.

[0034] Optionally, the average score is the metric.

[0035] Optionally, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0036] Optionally, the processing unit is configured to determine the individual scores by accessing a co-visibility graph that associates the reference points with the multiple images.

[0037] Optionally, the co-visibility graph indicates which of the reference points are visible in multiple images.

[0038] Optionally, the processing unit is configured to determine each of the individual scores by determining a number of reference points detected in a corresponding one of the images in the subset of images.

[0039] Optionally, the processing unit is also configured to determine an area score, indicative of coverage by the map.

[0040] Optionally, the area score is based on the spatial distribution of the data points in the map.

[0041] Optionally, at least one of the cells has a footprint of 2m x 2m.

[0042] Optionally, at least one of the cells also has a predetermined height.

[0043] Optionally, the processing unit is configured to determine the metric by acquiring a plurality of images from the camera system and determining joint visibility values, each joint visibility value indicating a number of reference points detected in a corresponding one of the plurality of images.

[0044] Optionally, the camera system comprises multiple cameras.

[0045] Optionally, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in the first position.

[0046] Optionally, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0047] Optionally, the plurality of cameras comprises a first forward-facing camera.

[0048] Optionally, the plurality of cameras comprises a second forward facing camera.

[0049] Optionally, the plurality of cameras comprises a first side-facing camera.

[0050] Optionally, the plurality of cameras comprises a second side-facing camera.

[0051] Optionally, the processing unit is configured to determine a desired viewing direction of the camera system to refine the value of the metric.

[0052] Optionally, the processing unit is configured to generate a graphic based on the determined desired viewing direction, the graphic being configured to instruct a user to change a current viewing direction of the camera system to the desired viewing direction.

[0053] Optionally, the camera system is configured to capture an image of the environment after a desired viewing direction of the camera system is achieved.

[0054] Optionally, the processing unit is configured to update the map based on the images.

[0055] Optionally, the processing unit is configured to update the metrics based on the updated map.

[0056] Optionally, the processing unit is configured to determine a metric prior to using the map to locate the user relative to the environment.

[0057] Optionally, the processing unit is configured to determine the metric before enabling the device to share content with another device.

[0058] Optionally, the processing unit is configured to determine the metric during a map building session in which the processing unit determines the map.

[0059] Optionally, the processing unit is configured to retrospectively determine the metric by accessing a previously determined map from the non-transient medium.

[0060] Optionally, the processing unit is configured to perform sanitization when the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0061] Optionally, the data includes images from a camera system, and the processing unit is configured to perform sanitization by removing or not considering the images.

[0062] Optionally, the camera system comprises multiple cameras, the data comprises a set of images generated by individual cameras, and the processing unit is configured to perform sanitization by removing or not considering the set of images.

[0063] Optionally, the data comprises an identification of reference points in an image from a camera system, and the processing unit is configured to perform the sanitization by not taking into account the identification of the reference points.

[0064] Optionally, the data represents rays or lines associated with an image from a camera system and a reference point, and the processing unit is configured to perform the sanitization by not taking into account the rays or lines associated with the image.

[0065] Optionally, the processing unit is configured to perform sanitization as part of the local optimization.

[0066] Optionally, the processing unit is configured to perform bundle adjustment and adjust one or more light rays associated with one or more images from the camera system, and the processing unit is configured to perform the bundle adjustment after performing sanitization and removing the data.

[0067] Optionally, the processing unit is configured to perform a bundle adjustment as part of the global optimization.

[0068] Optionally, the processing unit is configured to perform a global optimization based on the images from the camera system and the three-dimensional reference points.

[0069] Optionally, the processing unit is configured to perform a global optimization also based on a relative orientation between the cameras of the camera system.

[0070] Optionally, the processing unit is configured to determine a score for an image obtained from the camera system.

[0071] Optionally, the score is a constraint score.

[0072] Optionally, the score indicates the degree to which the image is constrained.

[0073] Optionally, the processing unit is configured to determine the score based on a Jacobian of the reference point measurements.

[0074] Optionally, the processing unit is configured to determine the score based on an information matrix, which is a diagonal matrix.

[0075] Optionally, the processing unit is configured to determine a score based on a number of reference points detected in the image.

[0076] Optionally, the processing unit is configured to perform data sanitization based on the score.

[0077] Optionally, the processing unit is configured to remove image constraints or to remove the image when performing data sanitization.

[0078] Optionally, the processing unit is configured to remove the constraint on the image or to remove the image when the score falls below a threshold.

[0079] Optionally, the processing unit is configured to determine the map by determining a plurality of map segments and connecting the map segments.

[0080] Optionally, the processing unit is configured to determine a first of the map segments by obtaining images from a camera system and linking the images, the images being generated in sequence by the camera system.

[0081] Optionally, the processing unit is configured to determine an individual score for the image.

[0082] Optionally, the processing unit is configured to obtain an additional image from the camera system, determine a score for the additional image, and initiate a second one of the map segments in response to the score of the additional image from the camera system satisfying a criterion.

[0083] Optionally, the processing unit is configured to initiate a second map segment when the score indicates that the image having a degree of constraint for the first map segment is below a threshold.

[0084] Optionally, the output includes one or more images from a camera system.

[0085] The device, configured to be worn on a user's head, includes a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit configured to determine a map based at least in part on output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment, the processing unit of the device is also configured to obtain a metric indicative of the likelihood of success of locating the user using the map, the processing unit being configured to obtain the metric by calculating the metric or by receiving the metric.

[0086] The device, configured to be worn on a user's head, includes a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit configured to determine a map based at least in part on output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment, and the processing unit configured to determine a score for images obtained from the camera system, the score indicating the degree to which the image is constrained against the map segments to form the map.

[0087] Optionally, the processing unit may be configured to determine the score by calculating the score.

[0088] Optionally, the processing unit may be configured to determine the score by receiving the score.

[0089] Optionally, the processing unit is configured to determine the score based on a Jacobian of the reference point measurements.

[0090] Optionally, the processing unit is configured to determine the score based on an information matrix, which is a diagonal matrix.

[0091] Optionally, the processing unit is configured to determine a score based on a number of reference points detected in the image.

[0092] Optionally, the processing unit is configured to perform data sanitization based on the score.

[0093] Optionally, the processing unit is configured to remove image constraints or to remove the image when performing data sanitization.

[0094] Optionally, the processing unit is configured to remove the constraint on the image or to remove the image when the score falls below a threshold.

[0095] Optionally, the processing unit is configured to perform sanitization when the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0096] Optionally, the data comprises images from a camera system, and the processing unit is configured to perform sanitization by removing or not considering the images.

[0097] Optionally, the camera system comprises multiple cameras, the data comprises a set of images generated by individual cameras, and the processing unit is configured to perform sanitization by removing or not considering the set of images.

[0098] Optionally, the data comprises an identification of reference points in an image from a camera system, and the processing unit is configured to perform the sanitization by not taking into account the identification of the reference points.

[0099] Optionally, the data represents rays or lines associated with an image from a camera system and a reference point, and the processing unit is configured to perform the sanitization by not taking into account the rays or lines associated with the image.

[0100] Optionally, the processing unit is configured to perform sanitization as part of the local optimization.

[0101] Optionally, the processing unit is configured to perform bundle adjustment and adjust one or more light rays associated with one or more images from the camera system, and the processing unit is configured to perform bundle adjustment after performing sanitization, and the image for which the score is determined is one of the one or more images or is different from the one or more images.

[0102] Optionally, the processing unit is configured to perform a bundle adjustment as part of the global optimization.

[0103] Optionally, the processing unit is configured to perform a global optimization based on one or more images from the camera system and the three-dimensional reference points.

[0104] Optionally, the processing unit is configured to perform a global optimization also based on a relative orientation between the cameras of the camera system.

[0105] Optionally, the processing unit is configured to determine the map by determining a plurality of map segments, the plurality of map segments constituting the map segment, and connecting the map segments, a portion of the map constituting one of the map segments.

[0106] Optionally, the camera system is configured to provide additional images, the additional images being generated by the camera system before the image for which the score is determined is generated, and the processing unit is configured to determine a first map segment of the map segments by linking the additional images, the additional images being generated in sequence by the camera system.

[0107] Optionally, the processing unit is configured to determine an individual score for the additional image.

[0108] Optionally, the processing unit is configured to initiate a second one of the map segments in response to the score of the image from the camera system satisfying a criterion.

[0109] Optionally, the processing unit is configured to initiate a second map segment when the score indicates that the image having a degree of constraint for the first map segment is below a threshold.

[0110] Optionally, the processing unit of the device is also configured to determine a metric indicative of the likelihood of success of locating the user using the map.

[0111] Optionally, the processing unit is configured to determine a metric based on the joint visibility of points of interest associated with different camera positions.

[0112] Optionally, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0113] Optionally, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0114] Optionally, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0115] Optionally, the metric indicates the likelihood of success in locating the user in one or more viewing directions.

[0116] Optionally, the processing unit is configured to determine a metric based on the number of times the point of interest is detected from different camera positions.

[0117] Optionally, the processing unit is configured to determine the metric without determining any convex hull.

[0118] Optionally, the metric has a value that is based on the directionality.

[0119] Optionally, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0120] Optionally, the directionality includes a turning direction.

[0121] Optionally, the directivity includes a tilt angle.

[0122] Optionally, the directivity includes a roll angle.

[0123] Optionally, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0124] Optionally, the metric is for one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a value based on a position within the one of the plurality of cells.

[0125] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0126] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment.

[0127] Optionally, the processing unit is also configured to determine a total number of images from the camera system for one of the plurality of cells.

[0128] Optionally, the total number of images is associated with a certain viewing direction for the cell.

[0129] Optionally, the total number of images is associated with a number of viewing directions for the cell.

[0130] Optionally, the camera system is configured to acquire a plurality of images, the plurality of images including an image for which a score is determined, and the processing unit is configured to determine a metric for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0131] Optionally, the processing unit is also configured to determine an average score by dividing the total score by the number of images in the subset of images.

[0132] Optionally, the average score is the metric.

[0133] Optionally, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0134] Optionally, the processing unit is configured to determine the individual scores by accessing a co-visibility graph that associates the reference points with the multiple images.

[0135] Optionally, the co-visibility graph indicates which of the reference points are visible in multiple images.

[0136] Optionally, the processing unit is configured to determine each of the individual scores by determining a number of reference points detected in a corresponding one of the images in the subset of images.

[0137] Optionally, the processing unit is also configured to determine an area score, indicative of coverage by the map.

[0138] Optionally, the area score is based on the spatial distribution of the data points in the map.

[0139] Optionally, at least one of the cells has a footprint of 2m x 2m.

[0140] Optionally, at least one of the cells also has a predetermined height.

[0141] Optionally, the processing unit is configured to determine the metric by acquiring a plurality of images from a camera system, the plurality of images including an image for which a score is determined, and determining a co-visibility value, each of the co-visibility values ​​indicating a number of reference points detected within a corresponding one of the plurality of images.

[0142] Optionally, the camera system comprises multiple cameras.

[0143] Optionally, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in the first position.

[0144] Optionally, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0145] Optionally, the plurality of cameras comprises a first forward-facing camera.

[0146] Optionally, the plurality of cameras comprises a second forward facing camera.

[0147] Optionally, the plurality of cameras comprises a first side-facing camera.

[0148] Optionally, the plurality of cameras comprises a second side-facing camera.

[0149] Optionally, the processing unit is configured to determine a desired viewing direction of the camera system to refine the value of the metric.

[0150] Optionally, the processing unit is configured to generate a graphic based on the determined desired viewing direction, the graphic being configured to instruct a user to change a current viewing direction of the camera system to the desired viewing direction.

[0151] Optionally, the camera system is configured to capture additional images after a desired viewing direction of the camera system is achieved.

[0152] Optionally, the processing unit is configured to update the map based on the additional images.

[0153] Optionally, the processing unit is configured to update the metrics based on the updated map.

[0154] Optionally, the processing unit is configured to determine a metric prior to using the map to locate the user relative to the environment.

[0155] Optionally, the processing unit is configured to determine the metric before enabling the device to share content with another device.

[0156] Optionally, the processing unit is configured to determine the metric during a map building session in which the processing unit determines the map.

[0157] Optionally, the processing unit is configured to retrospectively determine the metric by accessing a previously determined map from the non-transient medium.

[0158] The device, configured to be worn on a user's head, includes a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit configured to determine a map based at least in part on output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment, the processing unit configured to obtain a score for images obtained from the camera system, the score indicating the degree to which the image is constrained relative to the map segments to form the map, and the processing unit configured to obtain the score by calculating the score or by receiving the score.

[0159] A method implemented by an apparatus configured to be worn on a user's head, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, comprising the steps of obtaining, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and determining, by the processing unit, a metric indicative of the likelihood of success of locating the user using the map.

[0160] Optionally, the act of determining a metric includes calculating the metric.

[0161] Optionally, the act of determining the metric includes receiving the metric.

[0162] Optionally, a metric is determined based on the simultaneous visibility of points of interest associated with different camera positions.

[0163] Optionally, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0164] Optionally, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0165] Optionally, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0166] Optionally, the metric indicates the likelihood of success in locating the user in one or more viewing directions.

[0167] Optionally, a metric is determined based on the number of times the point of interest is detected from different camera positions.

[0168] Optionally, the metric is determined by the processing unit without determining any convex hull.

[0169] Optionally, the metric has a value that is based on the directionality.

[0170] Optionally, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0171] Optionally, the directionality includes a turning direction.

[0172] Optionally, the directivity includes a tilt angle.

[0173] Optionally, the directivity includes a roll angle.

[0174] Optionally, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0175] Optionally, the metric is for one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a value based on a position within the one of the plurality of cells.

[0176] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0177] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment.

[0178] Optionally, the act of determining the metric includes determining a total number of images from the camera system that are associated with one of the plurality of cells.

[0179] Optionally, the total number of images is associated with a certain viewing direction for the cell.

[0180] Optionally, the total number of images is associated with a number of viewing directions for the cell.

[0181] Optionally, the camera system is configured to acquire a plurality of images, and a metric is determined for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0182] Optionally, the metric is determined by dividing the total score by the number of images in the subset of images to take the average score.

[0183] Optionally, the average score is the metric.

[0184] Optionally, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0185] Optionally, the individual scores are determined by accessing a co-visibility graph that associates multiple images with a reference point.

[0186] Optionally, the co-visibility graph indicates which of the reference points are visible in multiple images.

[0187] Optionally, each individual score is determined by determining the number of reference points detected in a corresponding one of the images in the subset of images.

[0188] Optionally, the method further comprises determining an area score, indicative of coverage by the map.

[0189] Optionally, the area score is determined based on the spatial distribution of the data points in the map.

[0190] Optionally, at least one of the cells has a footprint of 2m x 2m.

[0191] Optionally, at least one of the cells also has a predetermined height.

[0192] Optionally, the metric is determined by acquiring a plurality of images from a camera system and determining joint visibility values, each joint visibility value indicating a number of reference points detected in a corresponding one of the plurality of images.

[0193] Optionally, the camera system comprises multiple cameras.

[0194] Optionally, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in the first position.

[0195] Optionally, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0196] Optionally, the plurality of cameras comprises a first forward-facing camera.

[0197] Optionally, the plurality of cameras comprises a second forward facing camera.

[0198] Optionally, the plurality of cameras comprises a first side-facing camera.

[0199] Optionally, the plurality of cameras comprises a second side-facing camera.

[0200] Optionally, the method further comprises determining, by the processing unit, a desired viewing direction of the camera system to refine the value of the metric.

[0201] Optionally, the method further includes generating a graphic based on the determined desired viewing direction, the graphic being configured to instruct a user to change the current viewing direction of the camera system to the desired viewing direction.

[0202] Optionally, the method further comprises obtaining an image of the environment from the camera system after a desired viewing direction of the camera system is achieved.

[0203] Optionally, the method further comprises updating the map based on the image.

[0204] Optionally, the method further comprises updating the metrics based on the updated map.

[0205] Optionally, the metric is determined before the map is used to locate the user relative to the environment.

[0206] Optionally, the metric is determined before a device shares content with another device.

[0207] Optionally, the metric is determined during a map building session in which the processing unit determines the map.

[0208] Optionally, the metric is determined retrospectively by accessing a previously determined map from a non-transient medium.

[0209] Optionally, the method further includes a step of performing sanitization, if the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0210] Optionally, the data includes images from a camera system and sanitization is performed by removing or not considering the images.

[0211] Optionally, the camera system comprises multiple cameras and the data comprises a set of images produced by individual cameras, and sanitization is performed by removing or not considering the set of images.

[0212] Optionally, the data comprises an identification of reference points in an image from a camera system, and sanitization is performed by a step that does not take into account the identification of the reference points.

[0213] Optionally, the data represents rays or lines associated with an image from a camera system and a reference point, and the sanitization is performed by a step that does not take into account the rays or lines associated with the image.

[0214] Optionally, sanitization is performed as part of a local optimization.

[0215] Optionally, the method further includes performing a bundle adjustment to adjust one or more light rays associated with one or more images from the camera system, the bundle adjustment being performed after sanitization has been performed to remove the data.

[0216] Optionally, the bundle adjustment is performed as part of the global optimization.

[0217] Optionally, a global optimization is performed based on images from a camera system and three-dimensional reference points.

[0218] Optionally, the global optimization is also performed based on the relative orientation between the cameras of the camera system.

[0219] Optionally, the method further comprises determining, by the processing unit, a score for the image obtained from the camera system.

[0220] Optionally, the score is a constraint score.

[0221] Optionally, the score indicates the degree to which the image is constrained.

[0222] Optionally, the score is determined based on the Jacobian of the baseline measurements.

[0223] Optionally, the score is determined based on an information matrix, which is a diagonal matrix.

[0224] Optionally, the score is determined based on the number of fiducial points detected in the image.

[0225] Optionally, the method further comprises performing data sanitization based on the score.

[0226] Optionally, the act of performing data sanitization includes removing image constraints or removing the image.

[0227] Optionally, image constraints or images are removed when the score falls below a threshold.

[0228] Optionally, the map is determined by determining a number of map segments and connecting the map segments.

[0229] Optionally, the act of determining the plurality of map segments includes determining a first map segment of the map segments by acquiring images from a camera system and linking the images, the images being generated in sequence by the camera system.

[0230] Optionally, the method further comprises determining an individual score for the image.

[0231] Optionally, the method further includes, in response to the score of the additional image from the camera system satisfying a criterion, acquiring an additional image from the camera system, determining a score for the additional image, and initiating a second one of the map segments.

[0232] Optionally, a second map segment is initiated when the score indicates that the image having a degree of constraint for the first map segment falls below a threshold.

[0233] Optionally, the output includes one or more images from a camera system.

[0234] A method implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, comprising the steps of: acquiring, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and acquiring, by the processing unit, a metric indicative of the likelihood of success of locating the user using the map, the act of acquiring comprising, by the processing unit, calculating the metric or receiving the metric.

[0235] A method implemented by an apparatus configured to be worn on a user's head, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, comprising the steps of: acquiring, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and determining, by the processing unit, a score for an image acquired from the camera system, the score indicating the degree to which the image is constrained relative to the map segments to form the map.

[0236] Optionally, the act of determining a score includes calculating the score.

[0237] Optionally, the act of determining the score includes receiving the score.

[0238] Optionally, the score is determined based on the Jacobian of the baseline measurements.

[0239] Optionally, the score is determined based on an information matrix, which is a diagonal matrix.

[0240] Optionally, the score is determined based on the number of fiducial points detected in the image.

[0241] Optionally, the method further comprises performing, by the processing unit, data sanitization based on the score.

[0242] Optionally, the act of performing data sanitization includes removing image constraints or removing the image.

[0243] Optionally, the act of performing data sanitization includes removing the constraint on the image or removing the image when the score falls below a threshold.

[0244] Optionally, sanitization is performed to remove or not consider data that would otherwise provide an undesirable contribution to the map if the data is used to determine the map.

[0245] Optionally, the data comprises images from a camera system and sanitization is performed to remove or not take into account the images.

[0246] Optionally, the camera system comprises multiple cameras, the data comprises a set of images produced by individual cameras, and sanitization is performed to remove or not consider the set of images.

[0247] Optionally, the data comprises an identification of reference points in an image from a camera system, and the sanitization is performed so as to not take into account the identification of the reference points.

[0248] Optionally, the data represents rays or lines associated with an image from a camera system and a reference point, and the sanitization is performed such that it does not take into account the rays or lines associated with the image.

[0249] Optionally, sanitization is performed as part of a local optimization.

[0250] Optionally, the method further includes performing a bundle adjustment to adjust one or more light rays associated with one or more images from the camera system, the bundle adjustment being performed after sanitization is performed and the image for which the score is determined is one of the one or more images or is different from the one or more images.

[0251] Optionally, the bundle adjustment is performed as part of the global optimization.

[0252] Optionally, the method further comprises performing, by the processing unit, a global optimization based on one or more images from the camera system and the three-dimensional reference points.

[0253] Optionally, the global optimization is also performed based on the relative orientation between the cameras of the camera system.

[0254] Optionally, the map is determined by the steps of determining a number of map segments, the number of map segments constituting a map segment, and connecting the map segments, a part of the map constituting one of the map segments.

[0255] Optionally, the camera system is configured to provide additional images, the additional images being generated by the camera system before the image for which the score is determined is generated, and the act of determining the map includes a step of determining a first map segment of the map segments by linking the additional images, the additional images being generated in sequence by the camera system.

[0256] Optionally, the method further comprises determining an individual score for the additional image.

[0257] Optionally, the method further comprises initiating, by the processing unit, a second one of the map segments in response to the score of the image from the camera system satisfying a criterion.

[0258] Optionally, a second map segment is initiated when the score indicates that the image having a degree of constraint for the first map segment falls below a threshold.

[0259] Optionally, the method further comprises determining a metric indicative of the likelihood of success in locating the user using the map.

[0260] Optionally, a metric is determined based on the simultaneous visibility of points of interest associated with different camera positions.

[0261] Optionally, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0262] Optionally, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0263] Optionally, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0264] Optionally, the metric indicates the likelihood of success in locating the user in one or more viewing directions.

[0265] Optionally, a metric is determined based on the number of times the point of interest is detected from different camera positions.

[0266] Optionally, the metric is determined by the processing unit without determining any convex hull.

[0267] Optionally, the metric has a value that is based on the directionality.

[0268] Optionally, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0269] Optionally, the directionality includes a turning direction.

[0270] Optionally, the directivity includes a tilt angle.

[0271] Optionally, the directivity includes a roll angle.

[0272] Optionally, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0273] Optionally, the metric is for one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a value based on a position within the one of the plurality of cells.

[0274] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, the metric having a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0275] Optionally, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment.

[0276] Optionally, the act of determining the metric includes determining a total number of images from the camera system that are associated with one of the plurality of cells.

[0277] Optionally, the total number of images is associated with a certain viewing direction for the cell.

[0278] Optionally, the total number of images is associated with a number of viewing directions for the cell.

[0279] Optionally, the camera system is configured to acquire a plurality of images, the plurality of images including an image for which a score is determined, and a metric is determined for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0280] Optionally, the method further comprises dividing the total score by the number of images in the subset of images to obtain an average score.

[0281] Optionally, the average score is the metric.

[0282] Optionally, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0283] Optionally, the individual scores are determined by accessing a co-visibility graph that associates multiple images with a reference point.

[0284] Optionally, the co-visibility graph indicates which of the reference points are visible in multiple images.

[0285] Optionally, each individual score is determined by determining the number of reference points detected in a corresponding one of the images in the subset of images.

[0286] Optionally, the method further comprises determining an area score, indicative of coverage by the map.

[0287] Optionally, the area score is based on the spatial distribution of the data points in the map.

[0288] Optionally, at least one of the cells has a footprint of 2m x 2m.

[0289] Optionally, at least one of the cells also has a predetermined height.

[0290] Optionally, the metric is determined by the steps of acquiring a plurality of images from a camera system, the plurality of images including the image for which the score is determined, and determining a co-visibility value, each of the co-visibility values ​​indicating the number of reference points detected within a corresponding one of the plurality of images.

[0291] Optionally, the camera system comprises multiple cameras.

[0292] Optionally, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in the first position.

[0293] Optionally, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0294] Optionally, the plurality of cameras comprises a first forward-facing camera.

[0295] Optionally, the plurality of cameras comprises a second forward facing camera.

[0296] Optionally, the plurality of cameras comprises a first side-facing camera.

[0297] Optionally, the plurality of cameras comprises a second side-facing camera.

[0298] Optionally, the method further comprises determining, by the processing unit, a desired viewing direction of the camera system to refine the value of the metric.

[0299] Optionally, the method further includes generating a graphic based on the determined desired viewing direction, the graphic being configured to instruct a user to change the current viewing direction of the camera system to the desired viewing direction.

[0300] Optionally, the method further comprises acquiring an additional image from the camera system after a desired viewing direction of the camera system is achieved.

[0301] Optionally, the method further comprises updating the map based on the additional imagery.

[0302] Optionally, the method further comprises updating the metrics based on the updated map.

[0303] Optionally, the metric is determined before the processing unit uses the map to locate the user relative to the environment.

[0304] Optionally, the metric is determined before a device shares content with another device.

[0305] Optionally, the metric is determined during a map building session in which the processing unit determines the map.

[0306] Optionally, the metric is determined retrospectively by accessing a previously determined map from a non-transient medium.

[0307] A method implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method including the steps of: acquiring, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and acquiring, by the processing unit, a score for an image acquired from the camera system, the score indicating the degree to which the image is constrained relative to the map segments to form the map, the act of acquiring the score including calculating the score or receiving the score.

[0308] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. The present invention provides, for example, the following: (Item 1) 1. A device configured to be worn on a user's head, comprising: a screen configured to present graphics to a user; a camera system configured to view an environment in which the user is located; a processing unit configured to determine a map based at least in part on output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and Equipped with The processing unit of the device is also configured to obtain a metric indicative of a likelihood of success of locating the user using the map, the processing unit being configured to obtain the metric by calculating the metric or by receiving the metric. (Item 2) Item 2. The apparatus of item 1, wherein the processing unit is configured to determine the metric based on simultaneous visibility of points of interest associated with different camera positions. (Item 3) 2. The apparatus of claim 1, wherein the metric indicates a number of reference points that can be used to locate the user relative to the environment. (Item 4) 2. The apparatus of claim 1, wherein the metric indicates a likelihood of success in locating the user in one or more viewing directions. (Item 5) Item 10. The apparatus of item 1, wherein the processing unit is configured to determine the metric based on the number of times a point of interest is detected from different camera positions. (Item 6) Item 1 , the apparatus of item 1 , wherein the processing unit is configured to determine the metric without determining any convex hull. (Item 7) Item 2. The apparatus of item 1, wherein the metric has a value based on directionality. (Item 8) 2. The apparatus of claim 1, wherein the metric is for one of a plurality of cells, each of the cells representing a three-dimensional space of a portion of the environment, and the metric has a value based on a position within the one of the plurality of cells. (Item 9) 2. The apparatus of claim 1, wherein the metric relates to one of a plurality of cells, each of the cells representing a three-dimensional space of a portion of the environment. (Item 10) The camera system is configured to acquire a plurality of images, and the processing unit is further configured to: identifying a subset of said images that belong to the same range of viewing directions; determining an individual score for images in the subset of images; summing said scores to obtain a total score; 10. The apparatus of claim 9, configured to determine the metric for one of the plurality of cells by: (Item 11) Item 11. The apparatus of item 10, wherein the processing unit is also configured to determine an average score by dividing the total score by the number of images in the subset of images. (Item 12) Item 11. The apparatus of item 10, wherein the processing unit is configured to determine the individual scores by accessing a co-visibility graph that associates reference points with the multiple images. (Item 13) Item 13. The apparatus of item 12, wherein the co-visibility graph indicates which of the reference points are visible among those in the multiple images. (Item 14) Item 11. The apparatus of item 10, wherein the processing unit is configured to determine each of the individual scores by determining a number of reference points detected in a corresponding one of the images in the subset of images. (Item 15) Item 1 , the apparatus of item 1 , wherein the processing unit is also configured to determine an area score indicative of coverage by the map. (Item 16) The processing unit includes: acquiring a plurality of images from the camera system; determining simultaneous visibility values, each of the simultaneous visibility values ​​indicating a number of fiducial points detected in a corresponding one of the plurality of images; Item 2. The apparatus of item 1, configured to determine the metric by: (Item 17) Item 2. The apparatus of item 1, wherein the processing unit is configured to determine a desired viewing direction of the camera system to refine the value of the metric. (Item 18) Item 18. The apparatus of item 17, wherein the processing unit is configured to generate the graphic based on the determined desired viewing direction, and the graphic is configured to instruct the user to change a current viewing direction of the camera system to the desired viewing direction. (Item 19) 20. The apparatus of claim 18, wherein the processing unit is configured to update the metric after the desired viewing direction is achieved. (Item 20) 2. The apparatus of claim 1, wherein the processing unit is configured to perform sanitization when the data is used to determine the map to remove or not consider data that would otherwise provide an undesirable contribution with respect to the map. (Item 21) 21. The apparatus of claim 20, wherein the data includes images from the camera system, and the processing unit is configured to perform the sanitization by (1) removing or not considering the images, (2) not considering identification of reference points within the images, and / or (3) not considering light rays or lines associated with the images. (Item 22) 21. The apparatus of claim 20, wherein the processing unit is configured to perform bundle adjustment and adjust one or more light rays associated with one or more images from the camera system, and the processing unit is configured to perform the bundle adjustment after performing the sanitization and removing the data. (Item 23) 2. The apparatus of claim 1, wherein the processing unit is configured to perform optimization based on images from the camera system, three-dimensional reference points, and a relative orientation between the cameras of the camera system. (Item 24) Item 10. The apparatus of item 1, wherein the processing unit is configured to determine a score for an image obtained from the camera system. (Item 25) 25. The apparatus of claim 24, wherein the score indicates the degree to which the image is constrained. (Item 26) 25. The apparatus of claim 24, wherein the processing unit is configured to determine the score based on a Jacobian of reference point measurements. (Item 27) The processing unit is configured to perform data sanitization based on the score; The processing unit is configured to remove the constraints on the image or to remove the image when performing the data sanitization. 25. The apparatus according to item 24. (Item 28) The processing unit includes: determining a plurality of map segments; connecting said map segments; 2. The apparatus of claim 1, configured to determine the map by: (Item 29) 30. The apparatus of item 28, wherein the processing unit is configured to determine a first one of the map segments by acquiring images from the camera system and linking the images, the images being generated in sequence by the camera system. (Item 30) The processing unit includes: acquiring an additional image from the camera system; determining a score for the additional image; and initiating a second one of the map segments in response to the score of the additional image from the camera system satisfying a criterion; and 30. The apparatus of item 29, configured to: (Item 31) Item 31. The apparatus of item 30, wherein the processing unit is configured to initiate the second map segment when the score indicates that the image having a degree of constraint on the first map segment is below a threshold. (Item 32) 1. A method implemented by a device configured to be worn on a user's head, said device having a screen configured to present graphics to a user, a camera system configured to view an environment in which the user is located, and a processing unit, said method comprising the steps of: acquiring, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and obtaining, by the processing unit, a metric indicative of a likelihood of success of locating the user using the map, the act of obtaining including calculating or receiving, by the processing unit, the metric; A method comprising: [Brief description of the drawings]

[0309] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the foregoing and other advantages and objects of the various embodiments of the present disclosure, a detailed description of the briefly described disclosure will be given by reference to specific embodiments thereof as illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered as limiting its scope, the present disclosure will be explained and described with additional specificity and detail through the use of the accompanying drawings.

[0310] [Figure 1] FIG. 1 illustrates another image display system having an image display device according to some embodiments.

[0311] [Diagram 2] FIG. 2 illustrates another image display system having an image display device according to another embodiment.

[0312] [Diagram 3]FIG. 3 illustrates another image display system having an image display device according to another embodiment.

[0313] [Figure 4] FIG. 4 illustrates another image display system having an image display device according to another embodiment.

[0314] [Diagram 5] FIG. 5 illustrates an image display device that displays frames at multiple depth planes.

[0315] [Figure 6] FIG. 6 illustrates a method for determining a map to enable an image display device to locate a user of the image display device and / or perform other functions.

[0316] [Figure 7] FIG. 7 illustrates an example of an environment that is divided into multiple cells.

[0317] [Figure 8A] FIG. 8A illustrates a method for determining a metric that indicates the likelihood of success in locating a user using a map.

[0318] [Figure 8B] FIG. 8B illustrates a graphical representation of the method of FIG. 8A.

[0319] [Figure 9] FIG. 9 illustrates an example of a co-visibility graph.

[0320] [Figure 10] FIG. 10 illustrates the map and location management method.

[0321] [Figure 11]FIG. 11 illustrates a method for sharing content between users of image display devices.

[0322] [Figure 12] FIG. 12 illustrates a technique for determining a map to enable an image display device to locate a user of the image display device and / or perform other functions.

[0323] [Figure 13] FIG. 13 illustrates a method for determining a map to enable an image display device to locate a user of the image display device and / or perform other functions.

[0324] [Figure 14] FIG. 14 illustrates a method performed by an image display device according to some embodiments.

[0325] [Figure 15] FIG. 15 illustrates another method implemented by an image display device according to some embodiments.

[0326] [Figure 16] FIG. 16 illustrates a specialized processing system, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0327] Various embodiments of the present disclosure are directed to methods, apparatus, and articles of manufacture for providing input for a video imaging device configured to be worn on a user's head. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0328] Various embodiments are described below with reference to the figures. It should be noted that the figures are not drawn to scale, and elements of similar structure or function are represented by similar reference numbers throughout the figures. It should also be noted that the figures are intended only to facilitate the description of the embodiments. They are not intended as a comprehensive description of the invention or as limitations on the scope of the invention. In addition, the illustrated embodiment does not necessarily have to have all the aspects or advantages shown. An aspect or advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiment even if not so illustrated or not so explicitly described.

[0329] The following description relates to illustrative VR, AR, and / or MR systems with which the embodiments described herein may be practiced, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the illustrative examples disclosed herein.

[0330] Problem and solution description In some cases, a localization map of the environment is obtained to locate a user of the image display device relative to the user's environment. Real-time tracking images from a camera system of the image display device are then matched against the localization map to locate the user. The success rate of localization depends on the quality of the localization map. Therefore, it would be advantageous to determine a metric to indicate the quality of the map that indicates the likelihood of success for using the map for localization. Various techniques may be employed to determine the metric. In one implementation, the metric is determined based on the simultaneous visibility of reference points captured in different images. The reference points may be map points that represent features of interest, such as corners, edges, etc., and the features of interest may be used to identify objects in the environment for localization purposes. After the user can be successfully localized relative to the user's environment, the user can then use the image display device to place virtual content relative to the environment, retrieve previous content from a previous session, share content with other users in the environment, etc.

[0331] The localization map may be created using a camera system of an image display device. In particular, a user of the image display device assumes different head poses (e.g., turns his / her head) and "scans" the environment. While doing so, the camera system captures images of the environment. A processing unit of the image display device then processes the images and creates the map. In some embodiments, to improve the quality of the map, undesirable data that may contribute to the map may be removed and / or adjusted during the map creation process. In one example, the undesirable data may be images that are not well constrained for the map segment. As images are generated in a sequence to create a map, the images are linked together to form the map segment. Each image may capture a certain number of reference points (e.g., map points). If an image captures many reference points that are also captured by other images, the image may be considered to be well constrained for the segment being created. On the other hand, if an image has only a few reference points and / or reference points in the image are not detected by other images, the image may be considered to be poorly constrained for the segment being created. In some embodiments, images that are not well constrained may be removed and map segments with well constrained images may be connected together to form a localization map.

[0332] 1-4 illustrate various components of an image display system 100 in various embodiments. The image display system 100 includes an image display device 101 and an apparatus 200 for providing input for the image display device 101. The apparatus 200 will be described in more detail below. The image display device 101 may be a VR device, an AR device, an MR device, or any other type of display device. The image display device 101 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned in front of the eye of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other ear canal of the end user 50 to provide stereo / adjustable sound control). The display subsystem 110 is designed to present to the eyes of the end user 50 light patterns that can be comfortably perceived as an augmentation to physical reality, with a high level of image quality and three-dimensional perception, and is also capable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0333] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display, through which the user can view light from real objects directly through a transparent (or semi-transparent) element. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's view of the real world. To this end, the display subsystem 110 comprises a partially transparent display. The display is positioned within the field of view of the end user 50, between the eyes of the end user 50 and the surrounding environment, such that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0334] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby it is combined with direct light from the surrounding environment and transmitted from the display to the user's eye 50. The projection subsystem may be an optical fiber-based projection device, and the display may be a waveguide-based display into which scanned light from the projection subsystem is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers in the light field may be stacked close enough together (i.e., one layer is within the perturbation cone of an adjacent layer) to appear persistent to the human accessory visual system. Additionally or alternatively, picture elements may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field, even when the layers are more sparsely stacked (i.e., one layer is outside the perturbation cone of an adjacent layer). The display subsystem 110 may be monocular or binocular.

[0335] The image display device 101 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and interocular distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes), or any combination of the foregoing. Many of these sensors operate under the assumption that the frame 102, onto which they are affixed, is, in turn, substantially fixed to the user's head, eyes, and ears.

[0336] The image display device 101 may also include a user orientation detection module. The user orientation module may detect the instantaneous position of the end user's 50 head 54 (e.g., via a sensor coupled to the frame 102) and predict the position of the end user's 50 head 54 based on the position data received from the sensor. Detecting the instantaneous position of the end user's 50 head 54 facilitates determining the specific real object the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that real object, and further providing an indication of the location where the virtual object will be displayed. The user orientation module may also track the eyes of the end user 50 based on tracking data received from the sensor.

[0337] The image display device 101 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem may include a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphic processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), e.g., field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0338] The control subsystem of image display device 101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D scene data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the 3D scene into a 2D image) from the 3D data stored in the 3D database and store these frames in the frame buffer. One or more additional integrated circuits may control the loading of frames into and out of the frame buffer and the operation of the image projection assembly of display subsystem 110.

[0339] The various processing components of the image display device 101 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 1-4, the image display device 101 may include a local processing and data module 130 that is operatively coupled to the display subsystem 110 and sensors, such as by wired leads or wireless connectivity 136. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 1), fixedly attached to a helmet or hat 56 (Figure 2), removably attached to a torso 58 of the end user 50 (Figure 3), or removably attached to a waist 60 of the end user 50 in a belt-type configuration (Figure 4). The image display device 101 may also include a remote processing module 132 and a remote data repository 134 operably coupled to the local processing and data module 130 by wired conductors or wireless connectivity 138, 140, etc., such that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130.

[0340] The local processing and data module 130 may comprise a power efficient processor or controller and digital memory such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensors and / or obtained and / or processed using the remote processing module 132 and / or the remote data repository 134, possibly for passing to the display subsystem 110 after processing or retrieval. The remote processing module 132 may comprise one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository 134 may comprise a relatively large digital data storage facility and may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130, allowing for fully autonomous use from any remote module.

[0341] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports to provide wire or optical communication, or one or more wireless interfaces or ports, such as via RF, microwave, and IR, to provide wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In still further implementations, the wired and wireless communication options may differ from those illustrated in FIGS. 1-4. Thus, the particular options for wired or wireless communication should not be considered limiting.

[0342] In some embodiments, the user orientation module is contained within the local processing and data module 130, while the CPU and GPU are contained within the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained within the local processing and data module 130. The 3D database can be associated with a remote data repository 134 or located locally.

[0343] Some image display systems (e.g., VR systems, AR systems, MR systems, etc.) use multiple volume phase holograms, surface relief holograms, or light directing optical elements that incorporate depth plane information to generate images that appear to originate from distinct depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") may be incorporated into or imprinted / embossed on a light directing optical element ("LOE", e.g., a planar waveguide) such that collimated light (a light beam with a substantially planar wavefront) intersects with the diffraction pattern at multiple locations as it is substantially totally internally reflected along the LOE and exits toward the user's eye. The DOE is configured such that the light exiting the LOE through it is converged to appear to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("concentrator").

[0344] For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). It should be understood that by using stacked LOE assemblies, multiple depth planes may be created, with each LOE configured to display an image that appears to originate from a particular depth plane. It should be understood that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Additionally, N, 2N, or 3N stacked LOEs may be used to generate RGB color images at N depth planes.

[0345] To present 3-D virtual content to a user, the image display system 100 (e.g., a VR system, an AR system, an MR system, etc.) projects images of the virtual content into the user's eye such that they appear to originate from different depth planes in the Z direction (i.e., orthogonal away from the user's eye). In other words, the virtual content may not only vary in the X and Y directions (i.e., 2D planes orthogonal to the central visual axis of the user's eye), but may also appear to vary in the Z direction such that the user may perceive objects to be very close, or at infinite distance, or any distance in between. In other embodiments, the user may perceive multiple objects at different depth planes at the same time. For example, the user may see a virtual dragon appearing from infinity and running towards the user. Alternatively, the user may see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at arm's length (approximately 1 meter) from the user at the same time.

[0346] The multi-plane focusing system creates the perception of variable depth by projecting images onto some or all of a number of depth planes located at separate fixed distances in the Z direction from the user's eyes. Now referring to FIG. 5, it should be understood that the multi-plane focusing system may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 5). Although an MR system can include any number of depth planes 150, one exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content in one or more of the six depth planes 150, a 3-D perception is created such that the user perceives one or more virtual objects at variable distances from the user's eyes. Given that the human eye is more sensitive to objects that are close in distance than objects that appear to be far away, more depth planes 150 are generated closer to the eyes, as shown in FIG. 5. In other embodiments, the depth planes 150 may be placed at equal distances away from each other.

[0347] The depth plane positions 150 may be measured in diopters, a unit of refractive power equal to the inverse of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, when generating virtual content at strategically placed depth planes 150, a user may perceive virtual objects in three dimensions. For example, a user may perceive a first virtual object as being close when it is displayed in depth plane 1 while another virtual object appears at infinity in depth plane 6. Alternatively, the virtual object may be displayed first in depth plane 6, then in depth plane 5, and so on until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes may be generated at various cross sections of the coffee cup, giving the user a highly granular 3-D view of the coffee cup.

[0348] In some embodiments, image display system 100 (e.g., VR system, AR system, MR system, etc.) may function as a multi-plane focal system. In other words, all six LOEs may be illuminated simultaneously such that images appearing to arise from six fixed depth planes are generated in rapid succession, with the light source rapidly transmitting image information to LOE1, then LOE2, then LOE3, etc. For example, a portion of a desired image, including an image of the sky at optical infinity, may be injected at time 1, and an LOE that retains collimation of the light (e.g., depth plane 6 from FIG. 5) may be utilized. An image of a closer tree branch may then be injected at time 2, and an LOE configured to create an image that appears to arise from a depth plane 10 meters away (e.g., depth plane 5 from FIG. 5) may be utilized. An image of a pen may then be injected at time 3, and an LOE configured to create an image that appears to arise from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a fast time-sequential (eg, 360 Hz) manner so that the user's eyes and brain (eg, visual cortex) perceive the inputs as all part of the same image.

[0349] The image display system 100 may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) and generate images for a 3-D experience / scenario. As used herein, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to originate from various depth planes matches the vergence and accommodation of the user's eyes for that image and minimizes or eliminates vergence-accommodation conflicts.

[0350] Location specific map

[0351] FIG. 6 illustrates a method for determining a map to enable the image display device 101 to locate a user 50 of the image display device 101. As shown in the figure, when the user 50 is using the image display device 101, the user 50 can move the image display device 101 to achieve different viewing locations and / or orientations. For example, the user 50 can turn his head, turn his body, and / or walk to different locations. In some embodiments, the image display device 101 includes a forward-facing camera. Thus, by moving the image display device 101, the field of view of the forward-facing camera of the image display device 101 will change accordingly. While the user 50 is in different poses, the camera of the image display device 101 generates corresponding images. In the illustrated example, the user 50 achieves three different poses by turning his head, and the forward-facing camera of the image display device 101 generates three images 200a-200c corresponding to the three poses. Each of the images 200a-200c captures an object 202 in the environment. For example, image 200b captures objects 202a-202d, and image 200c captures objects 202b-202e. Depending on the pose achieved by the user 50, some objects in the environment may be captured in multiple images 202 of the camera, and some other objects may be captured in only one image 200. In some embodiments, the processing unit 130 of the image display device 101 is configured to obtain the image 200 from the camera of the image display device 101, perform image processing, extract features from the image 200, and create the map 220. The map 220 may be stored in a non-transitory medium of the image display device 101 and may be used by the processing unit 130 to perform localization of the user 50. Thus, the map 220 functions as a localization map. In the illustrated embodiment, the map 220 is a three-dimensional representation of the environment as seen by different poses of the user 50 .

[0352] In some embodiments, the environment surrounding the user 50 may be divided into multiple cells. In such cases, the above map-building techniques may be employed for the different cells of the environment. FIG. 7 illustrates an example of an environment divided into multiple cells 300. Each cell 300 is a defined three-dimensional space that represents a portion of the environment. Each cell 300 may have a predefined size and shape. For example, each cell 300 may have a footprint of 2m by 2m and a height of 2m. Each cell 300 may have other footprint dimensions and / or other heights in other embodiments. Also, in other embodiments, each cell 300 may not have the rectangular configuration shown and may have other shapes. In the illustrated embodiment, the cells 300 all have the same shape and dimensions. In other embodiments, at least two of the cells 300 may have different individual dimensions and / or shapes.

[0353] In some embodiments, a user 50 of the image display device 101 may head to different locations in the environment corresponding to different cells 300 and may use a camera of the image display device 101 to scan the space within the corresponding cell and obtain different maps for the individual cells of the environment. The maps may be stored in a non-transitory medium of the image display device 101 to enable a processing unit 130 of the image display device 101 to perform localization of the user 50.

[0354] During use of the map to locate user 50, the camera of the image display device 101 acquires an image of the environment based on a certain position and orientation of user 50. Such camera images serve as tracking images (one or more images may be used to create the tracking map) to enable the processing unit 130 of the image display device 101 to track the position and / or posture and / or orientation of user 50. In particular, the processing unit 130 of the image display device 101 processes the images from the camera and determines whether features in the images match certain features in the map 220. If a match is found, the processing unit 130 may then determine the position and orientation of user 50 based on the matched features. In some embodiments, the map 220 may contain less information (e.g., features) than the tracking image provided by the camera of the image display device 101 because certain features may be filtered out or removed when the map 220 is created. This is advantageous because it allows the processing unit 130 to efficiently match the tracking images with the map 220. Also, in some embodiments, map 220 may be referred to as a "canonical map." In some embodiments, there may be one or more canonical maps. More than one canonical map may be used, for example, to accommodate security settings or access permissions for a location or for a virtual content or application. When performing localization, processing unit 130 performs feature extraction, extracting features from the camera image (tracking image) and matching the features with those in the canonical map. For example, processing unit 130 may perform feature extraction by identifying objects, corners, etc. in the image. In one implementation, processing unit 130 is configured to find a six-degree-of-freedom transformation between the tracking image and the canonical map and localize user 50.Once user 50 can be successfully located relative to the environment using the map, processing unit 130 may then enable the user to use the map to place virtual content relative to the environment, retrieve virtual content from a previous session, share virtual content with other users, etc.

[0355] In some embodiments, when there are multiple maps created for multiple cells (such as those described with reference to FIG. 7), the processing unit 130 may be configured to determine the applicable map based on the current location of the user 50. For example, if the user 50 is present at a location in cell No. 4, the processing unit 130 may then retrieve or access the map created for cell No. 4 and may then use such map for locating the user 50 while the user is present in the space corresponding to cell No. 4. When the user 50 moves out of the space of cell No. 4, the processing unit 130 may then stop using the map of cell No. 4 for locating the user. For example, the user 50 may move from cell No. 4 to cell No. 5. In such a case, the processing unit 130 may then use, retrieve or access the map created for cell No. 5 and may then use such map for locating the user 50 while the user is present in the space corresponding to cell No. 5.

[0356] Map Quality Scoring

[0357] As illustrated above, since localization of user 50 relative to the environment is based on matching features of the tracking camera images with features of map 220, the quality of map 220 may correlate with successful user 50 localization. In some embodiments, processing unit 130 of image display device 101 may be configured to determine a metric indicative of the likelihood of success in locating the user using map 220. In some embodiments, processing unit 130 of image display device 101 may be configured to determine such a metric during a map construction session in which user 50 scans the environment using image display device 101. Alternatively, or in addition, processing unit 130 may determine such a metric retroactively after map 220 is constructed and evaluate the quality of the constructed map 220.

[0358] In some embodiments, processing unit 130 may determine the metric by performing a calculation and obtaining the metric. In other embodiments, processing unit 130 may determine the metric by receiving the metric from another component or device and obtaining the metric. As a non-limiting example, the other component or device providing the metric may be a module within image display device 101 or an external device in communication with the image display device, which may be worn by the user or may be physically separate from the user. For example, the external device may be a wireless transmitter, a computer, a handheld or body-worn device, a database, a server, a base station, etc.

[0359] FIG. 8A illustrates an example of a method for determining a metric indicative of the likelihood of success in locating a user 50 using a map. The metric may be considered a measure of the quality of the map. In the illustrated example, the metric is determined for a particular cell 300 of the environment. In some cases, multiple cells 300 may exist in the environment, and the metric may be determined for each of the multiple cells 300. As shown in the figure, while the user 50 is present in the space of the cell 300, the user 50 may assume different poses, allowing the camera of the image display device 101 to capture different images 310 corresponding to the respective poses. In the illustrated example, the user 50 assumes six poses, and the camera of the image display device 101 generates six corresponding images 310a-310f. Image 310a is generated while the user 50 is looking generally in the north direction. Images 310b-310d are generated while the user 50 is looking generally in the east direction. Image 310e is generated while the user is looking generally in a southern direction, and image 310f is generated while the user is looking generally in a western direction.

[0360] In some embodiments, to determine a metric that measures the quality of the map, processing unit 130 first quantifies the pose of user 50 such that images that belong to the same viewing direction or the same range of viewing directions are grouped together. For example, in one implementation, images that have corresponding viewing directions that do not vary more than 30° from a given reference direction may be grouped together. Following the above example, for an east direction (reference direction), processing unit 130 may be configured to group all images 310 that have corresponding viewing directions facing east (reference direction) ±30°. Thus, processing unit 130 may group images 310b-310d because their corresponding viewing directions are within the range of east ±30°. Similar techniques may be employed to group images for other reference directions (e.g., south, west, north, etc.).

[0361] After the processing unit 130 quantifies the poses and groups the images 300 from the camera of the image display device 101 based on the quantified poses, the processing unit 130 may then determine a metric for a given reference direction based on the images 300 in the group. Following the above example, the processing unit 130 may determine a score for each of the three images 310b-310d that belong to the group for the East reference direction. As shown in the figure, the processing unit 130 may determine a score of 1 for image 310b, a score of 2 for image 310c, and a score of 3 for image 310d. In some embodiments, the score for image 310 represents the number of reference points captured in the image. As a non-limiting example, the reference points may be parts of features, parts of objects, points in three-dimensional space, etc. that will be represented by or included with the map 220. The reference points of the map 220 allow the tracked image to be matched with the map 220. Therefore, for a given viewing direction within a cell 300, the more control points there are, the more likely it is that the tracking image can be matched with the map 220 for that given viewing direction.

[0362] Various techniques may be employed to determine the score for the images 310. In one technique, the processing unit 130 may utilize a co-visibility graph to determine an individual score for the images 310. The co-visibility graph is a bipartite graph that links reference points to the images 310 based on which the reference points are visible in any of the images 310. FIG. 9 illustrates an example of a co-visibility graph 400. In an example, there are eight reference points R1-R8, and the identification of images 310b-310d that correspond to a reference direction (i.e., East, in the example) is also shown. In the example of the co-visibility graph 400, image 310b is linked with reference points R1-R4 because these reference points are captured in image 310b. Also, image 310c is linked with reference points R2-R4 and R6-R7 because these reference points are captured in image 310c. Image 310d was linked with fiducial points R5-R8 because these fiducial points were captured in image 310d. In the example shown, only the identification of images 310b-310d and their associated fiducial points are included in the simultaneous visibility graph 400 for illustrative purposes. Note that the simultaneous visibility graph 400 may include other fiducial points detected by other images and other identifications of images, such as images 310a, 310e, 310f, etc. In some embodiments, the simultaneous visibility graph 400 may include identifications of all of the images, all of the fiducial points captured in all of the images, and links associating the images with the fiducial points, generated by the camera of the image display device 101 while the user assumes different poses within a cell 300 of the environment.

[0363] Returning to the above example, the scores for images 310b-310d may be determined by processing unit 130 as 4, 5, and 4, respectively. After processing unit 130 determines the scores for the individual images that belong to the same reference direction (or range of reference directions), processing unit 130 may then combine the scores and determine a composite score. For example, processing unit 130 may add all of the scores to obtain a total score. According to the above example, the total score for the East reference direction would be 4+5+4=13. Processing unit 130 may also divide the total score by the number of images 310 used to derive the total score to obtain an average score. According to the above example, the average score for the East reference direction would be 13 / 3=4.33. The score 4.33 in the above example indicates that, on average, 4.33 reference points are detected in each image for the East reference direction.

[0364] In other embodiments, instead of including all of the reference points linked with each image 310 in the co-visibility graph 400 to determine a score for the corresponding image 310, the processing unit 130 may be configured to include only a subset of the reference points. For example, in other embodiments, the processing unit 130 may be configured to count only reference points linked to two or more images 310 to determine a score. In particular, following the example of the co-visibility graph of FIG. 9, the processing unit 130 may determine a score of 3 for image 310b because reference points R2-R4 captured in image 310b were also captured in image 310c. Reference point R1 is not counted as part of the score because reference point R1 was captured only in image 310b and not in any other image 310. Similarly, the processing unit 130 may determine a score of 5 for image 310c and a score of 3 for image 310d. The processing unit 130 may add all of the scores together to obtain a total score. Following the above example, the total score for the East reference direction would be 3+5+3=11. Processing unit 130 may also divide the total score by the number of images 310 used to derive the total score to obtain an average score. Following the above example, the average score for the East reference direction would be 11 / 3=3.67. The score 3.67 in the above example indicates that, on average, 3.67 reference points are detected in each image captured in at least two of the images for the East reference direction.

[0365] In the above example, the processing unit 130 may also be configured to determine a total number of images 310 from the camera for one of the plurality of cells 300, the total number of images being associated with a certain reference (viewing) direction for the cell 300 (e.g., eastward ±30°). In other embodiments, the reference direction may be other directions, such as northward, southward, westward, etc. Also, in other embodiments, the reference direction may have a range different from 60° (±30°). For example, in other embodiments, the reference direction may have a range less than 60°, such as 50° (e.g., ±25°), 40° (e.g., ±20°), 30° (e.g., ±15°), 20° (e.g., ±10°), 10° (e.g., ±5°), 0°, etc. In other embodiments, the processing unit 130 may be configured to determine a total number of images 310 associated with a certain plurality of viewing directions for the cell 300. For example, if images 310a-310f are all images generated for cell 300, processing unit 130 may determine the total number of images 310 to be six.

[0366] In some embodiments, the metric for indicating the quality of the map 220 may be or include a total score or an average score for a reference direction (e.g., north, east, south, west, etc.). In other embodiments, the metric for indicating the quality of the map 220 may be or include a total score or an average score for different individual reference directions (e.g., north, east, southwest, etc.). In further embodiments, the processing unit 130 may combine the scores for the different directions to determine a cell score for a cell 300. For example, the processing unit 130 may add up the total scores or may add up the average scores for the north, east, south, and west directions to determine the cell score for the cell 300. In such a case, the metric includes the cell score for the cell 300.

[0367] In further embodiments, the metric for indicating the quality of map 220 may be an array or set of values. For example, the metric may include different values ​​for different viewing directions. In such a case, the metric may have a first value associated with a first orientation and a second value associated with a second orientation.

[0368] As shown in the above embodiments, a metric for indicating the quality of the map 220 may have a value that is based on orientation. In some embodiments, the orientation is relative to one or more vertical axes and / or one or more horizontal axes. Also, in some embodiments, the orientation may include turning direction (such as that shown in FIG. 6), tilt angle (e.g., resulting from a user tilting their head and changing their viewing direction), roll angle (e.g., resulting from a user turning their head and changing their viewing direction), or any combination of the foregoing.

[0369] Also, in some embodiments, the metric may have a value based on the location of the user 50 relative to the coordinates of the cell 300. For example, if the user 50 is standing near the eastern boundary of the cell 300 and facing east, the camera will detect fewer reference points compared to a scenario in which the user 50 is standing near the western boundary of the cell 300 and facing east (because in this latter scenario, more objects will be visible within the cell 300). Thus, in some embodiments, the metric may have a first value associated with a first location within the cell 300 and a second value associated with a second location within the cell 300.

[0370] In some embodiments, processing unit 130 may also be configured to determine an area score, which indicates the degree of coverage by the map. In some cases, the area score is based on the spatial distribution of the data points of the map. The area score is advantageous in that it may ensure that the map covers different parts of the environment.

[0371] As shown in the above embodiment, the processing unit 130 is configured to determine a metric based on the simultaneous visibility of points of interest associated with different camera positions. The metric is advantageous because it indicates the number of reference points that can be used to locate the user 50 relative to the environment. Thus, the metric indicates the likelihood of success in locating the user 50 in one or more viewing directions. Also, as shown in the above embodiment, the processing unit is configured to determine a metric based on the number of times the points of interest are detected from different camera positions. Thus, the metric can be easily and efficiently determined without complex calculations and without consuming significant computing resources.

[0372] In the above embodiment, the metric is described as being determined based on different viewing orientations of the user 50 wearing the image display device 101, and the different viewing orientations are achieved by changing the directivity of the camera's field of view. In the above example, the change in directivity to achieve the different view orientations for determining the metric is achieved by the user turning his head or body about a vertical axis or multiple vertical axes. Alternatively, or in addition, the change in directivity to achieve the different view orientations for determining the metric may be achieved by the user turning his head or body about a horizontal axis or multiple horizontal axes. For example, the change in directivity may include a change in the turning angle (such as that shown in FIG. 6), a change in the tilt angle (e.g., resulting from a user tilting his head or body and changing the viewing direction), a change in the roll angle (e.g., resulting from a user rotating his head or body and changing the viewing direction), or any combination of the foregoing.

[0373] As discussed, in some embodiments, processing unit 130 is configured to determine the metric during a map construction session in which processing unit 130 determines map 220. In some cases, processing unit 130 may be configured to determine or update the metric as additional images from the camera of image display device 101 are generated during the map construction session. For example, processing unit 130 may initially determine map 220 based on images 310a, 310b provided by the camera of image display device 101 during a map construction session in which user 50 uses the camera to scan the environment. During the map construction session, processing unit 130 may also determine a metric to indicate the quality of map 220 (initially based on images 310a, 310b). As user 50 continues to scan the environment using the camera, the camera generates additional images based on the user's pose. For example, the camera may then generate image 310c. Processing unit 130 may then determine (e.g., update) map 220 based on new image 310c. Thus, updated map 220 would be based on images 310a-310c. In some cases, updated map 220 may be considered a new map that is built from a previous map. Processing unit 130 may also determine a metric to indicate the quality of updated map 220. For example, processing unit 130 may update previously determined metrics by changing one or more values ​​for the metric, adding one or more values ​​for the metric, and / or deleting one or more values ​​for the metric based on updated map 220. Alternatively, processing unit 130 may determine additional metrics (i.e., in addition to the previously determined metrics) based on updated map 220.

[0374] In other embodiments, instead of updating map 220 based on every newly generated image, processing unit 130 may be configured to update map 220 after the camera generates a certain number of images (e.g., 2, 3, 4, 10, 20, or any arbitrary number). In such cases, processing unit 130 may be configured to determine a metric (e.g., update a previous metric or determine an additional metric) each time map 220 is updated.

[0375] In further embodiments, the processing unit 130 may be configured to update the map 220 after one or more criteria have been achieved, which may or may not involve the number of images generated. For example, the criteria may be a user's command to determine the map 220. In such a case, the image display device 101 may include a user control to allow the user 50 to input a map construction command. When the processing unit 130 receives such a command from the user 50, the processing unit 130 then constructs the map 220 using the latest newly generated image. If the previous map 220 has already been determined, the processing unit 130 may update the map 220 using the most recent image in response to the user's command. Each time the map 220 is updated, the processing unit 130 may also determine a metric for the updated map 220. Thus, the determination of the metric may be performed in response to a user's input.

[0376] As another example, the criterion may be a change in the user's posture orientation (corresponding to the image orientation) within a certain threshold. In such a case, processing unit 130 may be configured to update map 220 using newly generated images as long as the user's posture from which the new image was generated does not deviate from the previous posture by more than a certain threshold. In one implementation, images 310 are stored in a non-transitory medium as they are generated. Processing unit 130 tracks the relative orientation between adjacent images. If the newly generated image 310 has an orientation relative to the previous image 310 that exceeds the threshold, processing unit 130 uses the last set of generated images 310 to update map 220. For example, assume that image 310c has an orientation relative to image 310b of 20°, image 310d has an orientation relative to image 310c of 15°, and image 310e has an orientation relative to image 310d of 55°, and the threshold is 25°. In such a case, if processing unit 130 determines that the relative orientation between images 310d and 310e exceeds the threshold, processing unit 130 then determines (e.g., updates) the map using images 310b-310d. This technique enables processing unit 130 to perform the map update using a set of images that are tightly constrained relative to each other.

[0377] In further embodiments, instead of updating the previous map 220 based on one or more newly generated images from the camera of the image display device 101, the processing unit 130 may be configured to determine an additional map based on one or more newly generated images. For example, with reference to FIG. 8A, assuming that a first map (map portion) has been determined based on images 310b-310d, after the camera generates image 310e, the processing unit 130 may determine that a new and additional map based on the newly generated image 310e is desired. In such a case, the processing unit 130 may determine a second map (map portion) based on image 310e. In some embodiments, the processing unit 130 may be configured to combine the first map and the second map after they are generated to obtain a final map for the cell 300. For example, the processing unit 130 may be configured to link features associated with the first and second maps to form a final map. In some embodiments, during a map-building session, processing unit 130 may be configured to generate a first metric for a first map and a second metric for a second map. In such cases, the first metric and the second metric may be considered separate values ​​(or sets of values) of the metric. Also, in some embodiments, processing unit 130 may combine the first metric and the second metric to determine a combined metric. Thus, as used herein, the term "metric" may refer to one or more values, one or more sets of values, one or more quantifications, any information for quantification purposes, or any information derived from any of the foregoing.

[0378] In still further embodiments, the processing unit 130 may be configured to determine the map 220 after the user 50 has finished scanning the environment using the camera of the image display device 101. In such a case, all images 310 generated while the user 50 is present in the cell 300 are stored in a non-transitory medium. After the scanning is completed, the processing unit 130 then retrieves the stored images 310, processes the images 310, and creates the map 220. In such a case, the processing unit 130 also determines the metrics after the map 220 is created. Thus, instead of determining the metrics during the construction of the map 220, the metrics are determined after the final map 220 is completed.

[0379] As illustrated in the above examples, the term "map," as used herein, may refer to an initially created map, an updated map, a final map, one or more map portions, a combined map resulting from combining map portions, or any combination of the foregoing. Also, the term "map" may refer to any type of map derived from an initially created map, from an updated map, from a final map, from one or more map portions, from a combined map, etc. Additionally, the term "map" may refer to any data structure, any three-dimensional representation, or any type of information derived from features of one or more images, and such data structure, three-dimensional representation, or type of information may form at least a portion of a map or may be any data entity for obtaining a map.

[0380] In any of the embodiments described above, the processing unit 130 may operate the screen of the image display device 101 to display the metrics so that the metrics may be visible to the user 50 as the user scans the environment using the camera of the image display device 101. Alternatively, or in addition, the processing unit 130 may also present the metrics on the screen to inform the user 50 of the quality of the map 220 being used while the processing unit 130 is using the map 220 to locate the user 50 relative to the user's environment. In this manner, the user 50 may selectively decide whether to continue using the map 220 for location purposes or to initiate a map building session to adjust the map 220 (e.g., if the value of the metric indicates that the map has low quality).

[0381] In some embodiments, during a map building session, the processing unit 130 may be configured to determine a desired viewing direction of the camera based on the metric value. For example, the processing unit 130 may be configured to determine the desired viewing direction of the camera to refine the value of the metric. In such a case, the processing unit 130 may be configured to generate a graphic based on the determined desired viewing direction and transmit the graphic for display by the screen of the image display device 101. The graphic is configured to instruct the user 50 to change the current viewing direction of the camera system to the desired viewing direction. For example, the graphic map instructs the user 50 to turn in a direction. In one implementation, the graphic may include a target object and instructions to instruct the user 50 to move (e.g., turn his head) to follow the target object. The processing unit 130 is configured to place the target object at a location within the screen of the image display device 101, where the location of the target object within the screen corresponds to the viewing direction desired to be achieved by the user. The processing unit 130 may also display a fixed object, whose position is fixed relative to the display of the image display device 101. For example, the fixed object may be a cross, a circle, a dot, etc., that is positioned in the center of the screen, and such a fixed object does not move in response to movement by the user (i.e., if the user 50 turns his / her head, the position of the fixed object within the screen does not change). The processing unit may also instruct the user to move the image display device 101 such that the fixed object will align with or occlude the target object. In some cases, after the user 50 moves the image display device 101 such that the fixed object will align with or occlude the target object, the camera system of the image display device 101 may then capture one or more images of the environment while the image display device 101 is in the desired orientation that is achieved.If a map has not been generated, processing unit 130 may create a map using the images and may calculate the metrics based on the map. If an initial map has already been created, processing unit 130 may then update the map based on the newly acquired images and may update the metrics based on the updated map.

[0382] The above process may be repeated by using the processing unit 130 to place the target object in different positions, move the image display device, and guide the user 50 to view the environment in different viewing orientations to build a map. As the map is updated based on newly generated images for new viewing directions, the processing unit 130 calculates a metric based on the current state of the map. In some embodiments, the processing unit 130 is configured to use the value of the metric for the current state of the map to determine the next desired viewing direction. For example, if the metric indicates that the map has a weakness in a certain direction, the processing unit 130 may then determine such a direction and determine a location for placement of the target object within the screen that would guide the user 50 to turn to the desired viewing direction. In other embodiments, instead of determining the position of the target object for placement within the screen of the image display device 101, the position of the target object may be predetermined. For example, a desired trajectory for the image display device 101 may be predetermined, and data representing such desired trajectory may be stored within a non-transitory medium of the image display device 101. In such a case, the camera system of the image display device 101 will capture images during the movement of the user 50, regardless of whether the user 50 successfully moves a fixed object in the screen to the target object. Also, in such a case, because the positioning of the target object in the screen is not based on the value of the metric, the processing unit 130 does not need to calculate the metric while the map is being generated. Instead, the processing unit 130 may calculate the metric after the map is completed (e.g., after the map generation session is over).

[0383] In other embodiments, the processing unit 130 is configured to retroactively determine the metric by accessing a previously determined map 220 from a non-transient media. The non-transient media may be part of the image display device 101. Alternatively, the non-transient media may be stored in a database, such as in a cloud network. In some cases, the processing unit 130 may determine a desired viewing direction of the camera to refine the value of the metric for the previously determined map 220, and may obtain images from the camera of the image display device 101 to refine the map 220, as described above. The refined map 220 may then be stored in the non-transient media.

[0384] In the above embodiment, the processing unit 130 is described with reference to providing a target object for display on the screen of the image display device 101 and instructing the user 50 to follow the target object to improve the value of the metric. In other embodiments, instead of the above technique, the processing unit 130 may instruct the user 50 to move to a location and / or change the viewing direction by presenting text instructions for display on the screen and / or by presenting audio instructions using a speaker. For example, the processing unit 130 may instruct the user 50 to turn 30° to the right, etc. In some embodiments, the desired viewing orientation may be determined by the processing unit 130 to maximize the metric value for a particular cell. Also, in some embodiments, when determining the desired viewing orientation, the processing unit 130 may consider the shortest path and / or the fastest time to obtain a good map.

[0385] FIG. 8B illustrates a graphical representation of the method of FIG. 8A. As shown in the figure, for a given cell, the camera system of the image display device 101 may capture several images. Each image is captured when the image display device 101 is in a certain position and orientation (as represented by a pyramid object with the top representing the camera position and the bottom representing the orientation of the captured image). In the illustrated example, the metric is determined for an east viewing direction. Thus, the processing unit 130 quantifies the pose and selects those images that correspond to an east viewing direction to determine the metric. Note that although more images may be captured for a particular cell 300, in the illustrated diagram, only a subset of the images are shown (i.e., only those images captured while the user was generally looking eastward are shown). For each east-looking image, a score a_i is determined. The metric for a particular cell 300 may be determined by summing all of the scores a_i and taking the sum. In some embodiments, the metric for a particular cell 300 may also be determined by dividing the sum by the number of images N and taking an average score. The determined metric may then be stored in a non-transient medium associated with (1) the determined identity of the cell 300 and (2) the viewing direction (e.g., “East” in the above example). In some embodiments, the metric may also be stored in association with a reference location (e.g., (x,z)). The metric for a particular cell 300 may have different values ​​depending on the reference location. For example, if the reference location is selected to be location 350 (the centroid of cell 300) in the illustrated example, there are only three images where the reference location 350 is East. Hence, only those three images are selected to calculate the metric for the East direction and location 350. On the other hand, if the reference location is selected to be location 352 in the illustrated example, there are five images where the reference location 352 is East. Thus, five images are selected to calculate the metric for the East direction and location 352.

[0386] The above technique for determining a metric to indicate the quality of the map 220 is advantageous because it does not require the processing unit 130 of the image display device 101 to determine any convex hull, which may be computationally complex and consume many processing resources. Also, in some embodiments, the metric may be used to ensure that the map created is good enough for localization before using the map to locate the user 10. In some cases, having a good quality localization map may not only improve the accuracy of localization, but also improve the localization speed. For example, with a good localization map, localization may be performed within 100 ms, more preferably 50 ms or less.

[0387] In other embodiments, the metric may be used by processing unit 130 of image display device 101 to select the best map (among multiple available maps) for locating user 50. For example, in some embodiments, there may be multiple maps created for the same environment (e.g., a particular cell 300). In such a case, processing unit 130 may select the map with the highest value of the metric for use to locate user 50.

[0388] In the above embodiments, the map creation process is described with reference to a camera system capturing images of the environment, and the processing unit is configured to determine a metric based on the concurrent visibility of points of interest associated with different camera positions. In some embodiments, the camera system may include only a single camera (e.g., a forward-facing camera) and capture images for map construction. In other embodiments, the camera system may include two cameras (e.g., two forward-facing cameras) and capture images for map construction. In further embodiments, the camera system of the image display device 101 may include four cameras (e.g., two forward-facing cameras, a first side-facing (e.g., facing left) camera, and a second side-facing (e.g., facing right) camera). Thus, in some cases, the camera positions with which the concurrent visibility of points of interest is associated may include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system. In other cases, the camera positions may include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0389] Also, as discussed, in some embodiments, the camera system may include multiple cameras to generate multiple images for creating the map. In such a case, the multiple images may include a first subset of images generated by the multiple cameras when the camera system of the image display device 101 is in a first position. The multiple images may also include a second subset of images generated by the multiple cameras when the camera system of the image display device 101 is in a second position. Thus, each subset of images is generated by the camera system of the image display device 101 when the image display device 101 is in a certain orientation. In some cases, the processing unit 130 is configured to track images within each of the subsets of images. For example, all images within each subset may be grouped together as a "key rig" associated with a certain orientation of the image display device 101.

[0390] Also, in some embodiments, features related to determining a metric to indicate the quality of the map may be incorporated into a map and localization management method. FIG. 10 illustrates an example of a map and localization management method 500 for managing the creation of localization maps and the use of such maps. In some embodiments, a localization map may be a tracking map that is tied to a head pose and specific to a single session, and / or may be a canonical map to which the tracking map may be localized. In some embodiments, a tracking map may be a canonical map if a quality metric associated with the tracking map is above a threshold. The method 500 starts at item 502. First, the processing unit 130 of the image display device 101 attempts to locate the user 50 of the image display device 101 using the already created map while the user 50 is in an environment corresponding to the created map (item 504). If the processing unit 130 successfully locates the user 50 relative to the environment using the map, the processing unit 130 may generate a signal to inform the user 50 that the localization was successful (item 506). If the processing unit 130 can successfully locate the user 50 relative to the environment using the map, this means that the user 50 can use the map to place virtual content relative to the environment and / or restore content previously placed relative to the environment. In some embodiments, content that was in the space from a previous session will be presented within the screen of the image display device 101 for presentation to the user in response to successful location. If the location is for the purpose of allowing the user 50 to restore virtual content from a previous session, the processing unit 130 may also inform the user 50 that the content was successfully restored. For example, the processing unit 130 may generate a signal, operate a screen and / or a speaker, and provide a notification to the user 50.In some embodiments, the localization may be a coordinate system transformation between the tracking map coordinate system and the canonical map 220 coordinate system, such that after the coordinate system transformation is applied, the tracking and canonical maps are aligned with each other (e.g., are identical).

[0391] On the other hand, if the processing unit 130 is unable to locate the user 50 after a certain number of attempts or a predefined duration, the processing unit 130 may then ask the user 50 (through a user interface) whether to perform map generation and create a map (item 508). In some embodiments, the map may be a tracking map. The tracking map may be used as the canonical map 220 in a subsequent session. In some embodiments, a session may be a single user turning the device on and off once, such that two sessions may be two users each turning their respective devices on and off once, or one user turning the user's device on and off twice. The location may fail for some reason. For example, localization may fail if the user 50 is in a new space for which a map has not been previously built, if there is a lighting change, if there is a change in the environment (e.g., furniture has been moved), if there is a dynamic change (e.g., people are moving), if the user 50 is not in the same viewing state as for the previously built map, if the correct map cannot be identified, if the user's pose cannot be identified, or if any combination of the above. In item 508, if the user decides to generate a new map, the processing unit 130 then initiates a map creation session and generates the map (item 510). In the map creation session, the user 50 takes different head poses and places the camera of the image display device 101 in different viewing directions to view different parts of the environment. The camera captures images of the environment while the image display device 101 is in different head poses. The processing unit 130 then processes these images and creates a map for localization purposes. During or after the creation of the map, processing unit 130 may use embodiments of the techniques described herein to calculate values ​​of metrics to indicate the quality of the map.In some embodiments, the processing unit 130 may calculate a value of a metric to indicate the quality of the map when it is a tracking map and / or when it is a canonical map. If the value of the metric is above a certain threshold, the processing unit 130 may then determine that the quality of the map is sufficient to locate the user 50 (in which case the image display device 101 may successfully restore the content from the previous session). After the map is generated, the map may be stored in one or more non-transient media (e.g., a non-transient media of the image display device 101 and / or a map database 530 remote from the image display device 101). The value of the metric may also be stored in association with the created map. The processing unit 130 then utilizes the newly created map to locate the user 50 (item 512).

[0392] Returning to item 508, if the user 50 decides not to start a map creation session to create a new map, the processing unit 130 may then ask the user 50 (via the user interface) whether to restore content from a previous session without placing the content (item 520). If the user 50 decides to restore content without placing the content, the processing unit 130 may then provide a user interface to allow the user 50 to manually restore the content (item 528). On the other hand, if the user 50 decides not to restore the previous content from the previous session, the processing unit 130 may then start a new session (item 524). In the new session, the user 50 may operate the image display device 101 and retrieve another map from the non-transitory media 526 and / or create a new map. The retrieved map may then be used for user location (item 512). In some embodiments, the maps stored in the non-transitory media 526 have individual metric values ​​associated with them. In such a case, when processing unit 130 selects one of the maps for location location, processing unit 130 considers the metric value. For example, processing unit 130 may be configured to select a map having a metric value above a predefined threshold. In some embodiments, if image display device 101 is connected to the Internet, processing unit 130 may retrieve one or more maps from map database 530 for storage in non-transitory medium 526.

[0393] In some embodiments, in addition to storing a map for storage in non-transient media 526, non-transient media 526 may also store content associated with the map, an identification of the content associated with the map, metadata (e.g., the time the map was created, who created the map, etc.), metrics indicative of the quality of the map, or any combination of the foregoing.

[0394] As illustrated in the above embodiment, a metric to indicate the quality of the map is determined before the image display device 101 allows the user 50 to locate relative to the environment using the map. This is advantageous because it ensures that locating the user 50 will be successful. If the processing unit 130 can successfully locate the user 50 relative to the environment using the map, the user 50 can use the map to place virtual content relative to the environment and / or use the map to restore content (e.g., virtual content placed in the environment) from a previous session.

[0395] Also, in some embodiments, features related to determining a metric to indicate the quality of the map may be incorporated into a content sharing scenario. FIG. 11 illustrates a method 600 for sharing content between users of image display devices 101. The method implemented by a sending device starts at item 602. A first user (sender) of the sending device (e.g., the first image display device 101) selects an item for sharing with another user (item 604). As a non-limiting example, the item may be one or more objects, one or more applications, or any content that the user wants to share with another user for a map. As a more specific example, the user of the first image display device 101 may be in a room with a table and may select content (e.g., a photo of an apple) for placement on the table to virtually share with another user. In such a case, the processing unit 130 of the first image display device 101 first determines a map (e.g., a tracking map) of the environment using the techniques described herein or any known techniques for creating a map using camera images. The map allows the user to place the content relative to the real environment. The map may be stored in a non-transitory medium (e.g., the first image display device 101 and / or a map database). The processing unit 130 may also calculate a metric for the map and determine whether the map has a sufficiently good quality (item 606). The determination of the metric may be performed using an embodiment of the techniques described herein. If the value of the metric is above a certain threshold, the processing unit 130 may then determine that the quality of the map is sufficient to allow the user to share the content. In such a case, the processing unit 130 of the first image display device 101 may send information about the map to a recipient of the second image display device 101 (item 608).As non-limiting examples, the information about the map may be an identification of the map, a storage location of the map, an invitation to use the map, data about the map, the map itself, etc., or any combination of the foregoing.

[0396] On the other hand, if the value of the metric falls below a certain threshold, the processing unit 130 may then determine that the quality of the map is not sufficient. In such a case, the processing unit 130 may generate a notification signal to inform the user that the content cannot be placed using the map (item 630). In such a case, the user of the first image display device 101 may initiate a map generation session and attempt to create a better map so that the content can be shared using a better map. Alternatively, the user may operate the first image display device 101 and share the content with the recipient without using any map. In such a case, the recipient of the content may manually place the content. For example, after the recipient using the second image display device 101 receives the content, the recipient may manually place the content with respect to the environment in which the recipient is present. According to the above embodiment, if the recipient is present in the same room as the sender, the recipient of the content (e.g., an image of an apple) may place such content on a table in the room.

[0397] The method implemented by the receiving device starts at item 620. A first user (recipient) of the receiving device (e.g., the second image display device 101) receives a notification that another user wants to share content (item 622). The second image display device 101 may allow the user to use the sender's map and decide whether to retrieve the shared content. If the recipient decides to use the sender's map, the second image display device 101 may retrieve the map from a non-transient medium (e.g., a map database) based on the information transmitted by the first image display device 101 in item 608. Alternatively, the second image display device 101 may obtain the map transmitted by the first image display device 101. The second image display device 101 then checks whether the recipient is in the same map as the sender's (item 624). If the recipient is using the same map that is or was used by the sender, the second image display device 101 determines that the content sharing is successful (item 626). Because the recipient is using the same map associated with the sender, the recipient can retrieve the shared content with the content in the same location as selected by the sender when the sender virtually places the content. If the recipient is not using the same map as the sender, the second image display device 101 may determine that the content sharing has failed and may take other actions (such as any of those described with reference to FIG. 10) and attempt to retrieve a map for content sharing (item 628). If the other actions also fail, the content sharing is unsuccessful and the recipient may need to manually place the content, similar to that discussed with reference to item 630.

[0398] In some embodiments, a first user may share an item with a second user. In some embodiments, sharing may occur only when the first user and the second user are located in the same space (e.g., a shared coordinate system). In some embodiments, location may occur when the first user's tracking map is located in a canonical map and the second user's tracking map is located in the same canonical map, resulting in both systems accessing virtual content relative to the same coordinate system (e.g., the canonical map coordinate system). In some embodiments, location may occur when the first user's tracking map is located in a first canonical map and the second user's tracking map is located in a second canonical map, resulting in both canonical maps sharing the same coordinate system and both systems accessing virtual content relative to the same coordinate system. Any other method of location may be used as long as the first user and the second user are able to access a shared coordinate system through one or more maps. In some embodiments, a map (e.g., a tracking map, a canonical map) may only be used for location if a minimum quality metric threshold is met or exceeded. The map may be stored, for example, on the local user device or in a shared cloud.

[0399] It should be noted that content sharing may be performed in real time (e.g., while both the sender and the recipient are present in the same environment). Alternatively, content sharing may not be performed in real time. For example, a sender at a first image display device 101 may virtually place content in a room to share with a recipient. The recipient may not be present in the room at the time the sender places the content. In such a case, the first image display device 101 may send a message to the second image display device 101 to inform the recipient that the sender is not sharing content in the room. The recipient using the second image display device 101 may then enter the room at a later time and read the shared content.

[0400] As shown in the above examples, a metric for measuring the quality of a map is advantageous in content sharing to ensure that a map for content sharing is of sufficient quality before allowing the image display device 101 to share content using the map.

[0401] Map Sanitization

[0402] As discussed, a map for locating user 50 may be created using images captured by the camera system of image display device 101. In some embodiments, during the map creation process, processing unit 130 of image display device 101 may be configured to perform map sanitization to remove and / or mitigate the impact of undesirable data that may negatively contribute to the map being created. As non-limiting examples, undesirable data may include incorrect observations of reference points, poorly constrained images, undesirable map points, or any combination of the foregoing.

[0403] A poorly constrained image, if used to form a localization map, will cause errors in the map, which in turn will cause localization errors. An image may be poorly constrained due to the image not having enough views of the reference points and / or due to the image having noise that may be mischaracterized as a reference point. Once a poorly constrained image is incorporated into the map, subsequent images may only see the stereo points generated from this image instead of the rest of the "good" map, which may lead to drift.

[0404] FIG. 12 illustrates an example of a sequence of images acquired by the camera system of the image display device 101. A sequence of images is acquired by the camera system during the map creation process to create a localization map. In an example, some of the images are well constrained in that they form a well-connected portion of the map. In some embodiments, an image may be considered "well constrained" if it has a certain number of fiducial points that are also present in adjacent images, which number may be any user-defined threshold such as 2, 3, 4, 10, 20, 100, etc. The well-constrained portion of the map is followed by poorly constrained images (which may be due to fast head rotation). In an example, the small triangle represents the images acquired to form the map, and the large triangle represents the current image being considered. Each dot represents a fiducial point or point of interest. The left column shows the sequence of images that form the tracking map, while the right column shows the map after the current image (track image) from the left column has been assigned to a map segment. The different map segments are shown in different distinct colors / grayscales. As used herein, a "segment" of a map may refer to a portion of a map. In some cases, a portion of a map may be formed by two or more images that are logically related to each other.

[0405] In the example of FIG. 12, the first row shows three images acquired in a sequence that are well constrained. The three images are well constrained in the sense that each image captures multiple map points (fiducial points) that are also captured by one or more other images. Thus, the three images are well connected to each other based on the commonly observed map points. Note that the map points can be any captured feature of interest in the images that can be used for tracking purposes. For example, the map points can be features associated with corners of an object that can be used to identify the same object in different images. Since the first three images are well constrained, they are all assigned to the first segment of the map such that they form the first segment of the map. In the illustrated embodiment, the processing unit 130 may be configured to determine a quality score for each image, the quality score representing, indicating, or based on the number of captured map points in the image.

[0406] The second row in the illustrated diagram shows a fourth image being obtained. In the illustrated example, only few map points are seen in the fourth image and it is poorly constrained with respect to the first segment. The poorly constrained fourth image may have a corresponding quality score that is low due to the poorly constrained nature of the fourth image. Thus, the fourth image is assigned to a new segment (the second segment) due to the poor quality score resulting from the few map points observed. In the illustrated example, the fourth image is poorly constrained due to the fast rotation of the user's head, which results in blurring of the fourth image. Because the fourth image is blurred, only few map points can be detected. In other embodiments, the blurring in the image may be due to other factors different from head rotation.

[0407] The third row shows the fifth image being acquired. Again, only a few map points are seen in the fifth image, which is poorly constrained relative to the second segment. The poorly constrained fifth image may have a corresponding quality score that is low due to the poorly constrained nature of the fifth image. Thus, the fifth image is assigned to a new segment (the third segment). In an embodiment, the fifth image is also poorly constrained due to the continued rapid rotation of the user's head. In other embodiments, blurring in the images may be due to other factors.

[0408] The fourth row shows the sixth image being acquired. The sixth image is also poorly constrained with respect to the third segment and is therefore assigned to a new segment (the fourth segment). However, the rotation that contributes to motion blur has been stopped and therefore the fifth image contains more map points that can be observed by processing unit 130.

[0409] The fifth row shows the seventh image being acquired. The seventh image is well constrained with respect to the current segment, and therefore the seventh image is assigned to the same segment as the sixth image (the fourth segment).

[0410] Finally, the sixth row shows the eighth image being acquired. The eighth image is also well constrained with respect to the current segment (the fourth segment), and therefore the eighth image is assigned to the fourth segment. Thus, the last three images form part of the fourth segment of the map.

[0411] In the above embodiment, images and map points each have a segment indicator attached to them, which is assigned after the map points are identified and after the quality scores for the individual images are determined. Images that are well constrained will be assigned the current segment number by the processing 130 unit (as will all map points created from that image), while images that are poorly constrained and their map points will be assigned the next segment number by the processing unit 130.

[0412] As discussed, the degree to which an image is constrained is accomplished through a quality score, which is calculated by processing unit 130 based on the identified map points associated with the image. In some cases, the quality score may indicate the degree to which an image is constrained by the map points. In one implementation, the quality score may be calculated by processing unit 130 as J^TΣ^(-1)J, where J is the Jacobian of the map point measurements with respect to pose and Σ^(-1) is the information matrix of the measurements (assuming block diagonal due to measurement independence). In other embodiments, the quality score may be determined based on other factors and / or calculated using other algorithms.

[0413] In some embodiments, different map segments may be combined by the processing unit 130 to form a map. In some cases, "bad" segments (such as the second and third segments in the above example) may be removed and not used to form the map. In one implementation, bad segments may be identified based on the number of images assigned to the segment. If the number of images assigned to a segment falls below a certain value (e.g., 1, 2, 3, etc.), the segment may be considered "bad" and may be excluded from being used to form the map. In one implementation, the joining of map segments may be performed using a loop closure algorithm. In the loop closure algorithm, loop closure is performed based on images that are within a sufficiently large map segment. In accordance with the above example, loop closure is performed based on images in the first and fourth segments, while excluding the second and third segments due to the badly constrained images in these segments. Thus, badly constrained entities created between two well-connected map segments that are joined by loop closure may be deleted and removed so that they do not contribute to the final map.

[0414] In some embodiments, processing unit 130 may be configured to track well-constrained map segments and process them as if they were separate maps. Thus, the map segments themselves may be considered examples of maps. For example, processing unit 130 may be configured to perform local bundle adjustment on one or more map segments, perform global bundle adjustment on one or more map segments, calculate metrics for one or more map segments, perform tracking on points within the map segments, or any combination of the foregoing.

[0415] In some embodiments, the created map may be further adjusted by processing unit 130 during normal use of image display device 101. For example, in some embodiments, after a map is created, processing unit 130 may use such map to perform localization and / or other functions. However, during such use of image display device 101, the camera system continues to acquire images of the environment. Thus, in some embodiments, processing unit 130 may continue to periodically update the created map based on newly acquired images and attempt to improve the map. In one implementation, processing unit 130 may determine a metric based on the newly acquired images and may determine whether to update the map based on the new images. For example, processing unit 130 may determine a metric that would be for the map if a new image is used to update the map. If the metric is higher than that of the existing map, processing unit 130 may determine that an update of the map based on the new images is desirable. On the other hand, if the metric is lower than that of the existing map, processing unit 130 may determine that updating the map based on the new image is undesirable and will continue to use the existing map without updating it using the new image.

[0416] In the above embodiments, the map is described as being created for use in performing localization. In other embodiments, the created map may be utilized by processing unit 130 to perform other functions. For example, in other embodiments, the created map may be used by processing unit 130 as a frame of reference to place virtual content.

[0417] In some embodiments, multiple images from the camera system of the image display device 101 captured during a tracking mode to track the position of the user 50 may be used to form the tracking map. When performing localization, the processing unit 130 finds a six degree of freedom transformation between the tracking map and the canonical map to locate the user 50. Once the user 50 can be successfully localized relative to its environment using the map, the processing unit 130 may then enable the user to use the map to place virtual content relative to the environment, retrieve virtual content from a previous session, share virtual content with other users, etc.

[0418] 13 illustrates a method 900 for determining a map to enable an image display device to locate a user of the image display device. The method 900 includes performing a local optimization (item 902) and performing a global optimization (item 904) to determine the map. The determined map may then be used later by the image display device to locate a user of the image display device. In some embodiments, both the local and global optimization may be performed by the processing unit 130 of the image display device 101. In other embodiments, one or both of the local and global optimization may be performed by an external device, such as a server in a cloud network.

[0419] In some embodiments, the act of performing the local optimization in item 902 includes removing undesirable data so that they do not contribute to the map being created. As a non-limiting example, undesirable data may be images that are not well constrained to the map segment or any data associated with such images. For the global optimization, it is desirable to use good data. Thus, during the local optimization, processing unit 130 removes undesirable data.

[0420] In some embodiments, the act of performing a global optimization in item 904 includes performing a bundle adjustment in which one or more rays from the images are adjusted so that they have shared terms. Also, in some embodiments, the global optimization may be performed based on an image acquired by a camera system of the image display device 101, an orientation of an image plane associated with the image, and a 3D reference point. In further embodiments, the global optimization may also be performed based on the relative orientations of the cameras with respect to one another.

[0421] Optionally, in method 900, performing the local optimization includes, if the data is used to determine the map, performing sanitization by processing unit 130 to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0422] Optionally, in method 900, the data includes images from a camera system of image display device 101. In such cases, during local optimization, the processing unit performs sanitization by removing or not considering the images.

[0423] Optionally, method 900 is performed by an image display device 101 that includes multiple cameras. In such a case, the data includes a set of images generated by separate cameras, and processing unit 130 performs sanitization by removing or not considering the set of images.

[0424] Optionally, in method 900, the data includes an identification of reference points in an image from a camera system, and processing unit 130 performs sanitization by not taking into account the identification of the reference points.

[0425] Optionally, in method 900, the data represents rays or lines and reference points associated with an image from a camera system, and processing unit 130 performs sanitization by not considering the rays or lines associated with the image.

[0426] Optionally, in method 900, processing unit 130 performs bundle adjustment during global optimization to adjust one or more rays associated with one or more images from the camera system, and processing unit 130 performs sanitization and removes data before performing the bundle adjustment.

[0427] Optionally, in method 900, processing unit 130 performs a global optimization based on the images from the camera system and the three-dimensional reference points.

[0428] Optionally, in method 900, processing unit 130 performs a global optimization based on the relative orientation between the cameras of the camera system.

[0429] Optionally, in method 900, processing unit 130 determines a score for an image acquired from the camera system during local optimization. In some embodiments, the score may be a constraint score. Also, in some embodiments, the score indicates the degree to which the image is constrained. In some embodiments, the score may be determined by processing unit 130 based on a Jacobian of the fiducial point measurements. Also, in some embodiments, the score may be determined by processing unit 130 based on an information matrix, which is a diagonal matrix. Additionally, in some embodiments, the score may be determined by processing unit 130 based on the number of detected fiducial points in the image.

[0430] Optionally, in method 900, processing unit 130 may perform data sanitization based on the score.

[0431] Optionally, in method 900, processing unit 130 may remove image constraints or remove images when performing data sanitization.

[0432] Optionally, in method 900, the processing unit may remove the constraint on the image or remove the image when the score falls below a threshold.

[0433] Optionally, in method 900, processing unit 130 may determine the map by determining a number of map segments and connecting the map segments.

[0434] Optionally, in method 900, processing unit 130 may determine a first map segment of the map segments by obtaining images from a camera system and linking the images, the images being generated in a sequence by the camera system.

[0435] Optionally, in method 900, processing unit 130 determines an individual score for the image.

[0436] Optionally, in method 900, in response to the score of the additional image from the camera system satisfying the criterion, processing unit 130 acquires an additional image from the camera system, determines a score for the additional image, and starts a second of the map segments.

[0437] Optionally, in method 900, processing unit 130 may initiate a second map segment when the score indicates that the image having a degree of constraint for the first map segment falls below a threshold.

[0438] Methods implemented by a processing unit and / or an application within a processing unit

[0439] FIG. 14 illustrates a method 1000 according to some embodiments. The method 1000 may be implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit. In some embodiments, the method 1000 may be implemented by any of the image display devices 101 shown in FIGS. 1-4. For example, the method 1000 may be implemented by the processing unit 130 of the image display device 101. The method 1000 includes obtaining, by the processing unit, an output from the camera system (item 1002); determining, by the processing unit, a map based at least in part on the output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment (item 1004); and determining, by the processing unit, a metric indicative of the likelihood of success of locating the user using the map (item 1006).

[0440] In some embodiments, in method 1000, the processing unit may determine the metric by performing a calculation and obtaining the metric. In other embodiments, the processing unit may determine the metric by receiving the metric from another component or device and obtaining the metric. As a non-limiting example, the other component or device providing the metric may be a module within the image display device 101 or an external device in communication with the image display device, which may be worn by the user or may be physically separate from the user. For example, the external device may be a wireless transmitter, a computer, a handheld or body-worn device, a database, a server, a base station, etc.

[0441] Optionally, in method 1000, the act of determining the metric includes calculating, by a processing unit, the metric.

[0442] Optionally, in method 1000, the act of determining the metric includes receiving, by a processing unit, the metric.

[0443] Optionally, in method 1000, a metric is determined based on the simultaneous visibility of points of interest associated with different camera positions.

[0444] Optionally, in the method 1000, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0445] Optionally, in method 1000, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0446] Optionally, in method 1000, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0447] Optionally, in method 1000, the metric indicates the likelihood of success in locating a user in one or more viewing directions.

[0448] Optionally, in method 1000, a metric is determined based on the number of times the point of interest is detected from different camera positions.

[0449] Optionally, in method 1000, the metric is determined by the processing unit without determining any convex hull.

[0450] Optionally, in method 1000, the metric has a value that is based on the directionality.

[0451] Optionally, in method 1000, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0452] Optionally, in method 1000, the directivity includes a turning direction.

[0453] Optionally, in the method 1000, the directivity includes a tilt angle.

[0454] Optionally, in the method 1000, the directivity includes a roll angle.

[0455] Optionally, in method 1000, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0456] Optionally, in method 1000, the metric relates to one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment, and the metric has a value based on a position within the one of the plurality of cells.

[0457] Optionally, in method 1000, the metric relates to one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment, and the metric has a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0458] Optionally, in the method 1000, the metric relates to one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment.

[0459] Optionally, in method 1000, the act of determining the metric includes determining a total number of images from the camera system that are associated with one of the plurality of cells.

[0460] Optionally, in the method 1000, the total number of images is associated with a certain viewing direction for the cell.

[0461] Optionally, in method 1000, the total number of images is associated with a number of viewing directions for the cell.

[0462] Optionally, in method 1000, the camera system is configured to acquire a plurality of images, and a metric is determined for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0463] Optionally, in method 1000, the metric is determined by dividing the total score by the number of images in the subset of images to obtain an average score.

[0464] Optionally, in the method 1000, the average score is the metric.

[0465] Optionally, in the method 1000, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0466] Optionally, in method 1000, the individual scores are determined by accessing a co-visibility graph that associates reference points with multiple images.

[0467] Optionally, in the method 1000, the co-visibility graph indicates which of the reference points are visible among those in multiple images.

[0468] Optionally, in method 1000, each of the individual scores is determined by determining the number of fiducial points detected in a corresponding one of the images in the subset of images.

[0469] Optionally, the method 1000 further comprises determining an area score, which indicates the degree of coverage by the map.

[0470] Optionally, in method 1000, the area score is determined based on the spatial distribution of the data points of the map.

[0471] Optionally, in the method 1000, at least one of the cells has a footprint of 2m x 2m.

[0472] Optionally, in the method 1000, at least one of the cells also has a predetermined height.

[0473] Optionally, in method 1000, the metric is determined by acquiring a plurality of images from a camera system and determining a joint visibility value, each of the joint visibility values ​​indicating a number of reference points detected in a corresponding one of the plurality of images.

[0474] Optionally, in the method 1000, the camera system comprises multiple cameras.

[0475] Optionally, in method 1000, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in a first position.

[0476] Optionally, in method 1000, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0477] Optionally, in the method 1000, the plurality of cameras comprises a first forward-facing camera.

[0478] Optionally, in method 1000, the plurality of cameras comprises a second forward-facing camera.

[0479] Optionally, in method 1000, the plurality of cameras comprises a first side-facing camera.

[0480] Optionally, in method 1000, the plurality of cameras comprises a second side-facing camera.

[0481] Optionally, the method 1000 further comprises determining, by the processing unit, a desired viewing direction of the camera system to refine the value of the metric.

[0482] Optionally, method 1000 further includes generating a graphic based on the determined desired viewing direction, the graphic being configured to instruct the user to change the current viewing direction of the camera system to the desired viewing direction.

[0483] Optionally, the method 1000 further includes obtaining an image of the environment from the camera system after a desired viewing direction of the camera system is achieved.

[0484] Optionally, the method 1000 further comprises updating the map based on the image.

[0485] Optionally, the method 1000 further comprises updating the metrics based on the updated map.

[0486] Optionally, in method 1000, the metric is determined before the map is used to locate the user relative to the environment.

[0487] Optionally, in method 1000, the metric is determined before a device shares content with another device.

[0488] Optionally, in method 1000, the metric is determined during a map building session in which the processing unit determines the map.

[0489] Optionally, in method 1000, the metric is determined retrospectively by accessing a previously determined map from a non-transient medium.

[0490] Optionally, method 1000 further includes a step of performing sanitization, if the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0491] Optionally, in method 1000, the data includes images from a camera system and sanitization is performed by removing or not considering the images.

[0492] Optionally, in method 1000, the camera system comprises multiple cameras, the data comprises a set of images generated by individual cameras, and sanitization is performed by removing or not considering the set of images.

[0493] Optionally, in method 1000, the data comprises an identification of reference points in an image from a camera system, and sanitization is performed by a step that does not take into account the identification of the reference points.

[0494] Optionally, in method 1000, the data represents rays or lines associated with an image from a camera system and a reference point, and sanitization is performed by a step that does not take into account the rays or lines associated with the image.

[0495] Optionally, in method 1000, sanitization is performed as part of a local optimization.

[0496] Optionally, method 1000 further includes performing a bundle adjustment to adjust one or more light rays associated with one or more images from the camera system, the bundle adjustment being performed after sanitization has been performed to remove the data.

[0497] Optionally, in method 1000, bundle adjustment is performed as part of the global optimization.

[0498] Optionally, in method 1000, a global optimization is performed based on images from a camera system and three-dimensional reference points.

[0499] Optionally, in method 1000, the global optimization is also performed based on the relative orientation between the cameras of the camera system.

[0500] Optionally, the method 1000 further comprises determining, by the processing unit, a score for the image acquired from the camera system.

[0501] Optionally, in the method 1000, the score is a constraint score.

[0502] Optionally, in the method 1000, the score indicates the degree to which the image is constrained.

[0503] Optionally, in method 1000, the score is determined based on the Jacobian of the reference point measurements.

[0504] Optionally, in method 1000, the score is determined based on an information matrix, which is a diagonal matrix.

[0505] Optionally, in method 1000, the score is determined based on the number of fiducial points detected in the image.

[0506] Optionally, the method 1000 further comprises performing data sanitization based on the score.

[0507] Optionally, in method 1000, the act of performing data sanitization includes removing image constraints or removing the image.

[0508] Optionally, the method 1000 removes image constraints or images when the score falls below a threshold.

[0509] Optionally, in the method 1000, the map is determined by determining a number of map segments and connecting the map segments.

[0510] Optionally, in method 1000, the act of determining a plurality of map segments includes determining a first map segment of the map segments by acquiring images from a camera system and linking the images, the images being generated in sequence by the camera system.

[0511] Optionally, the method 1000 further comprises determining an individual score for the image.

[0512] Optionally, method 1000 further includes, in response to the score of the additional image from the camera system satisfying a criterion, acquiring an additional image from the camera system, determining a score for the additional image, and starting a second map segment of the map segment.

[0513] Optionally, in method 1000, a second map segment is initiated when the score indicates that the image having a degree of constraint for the first map segment falls below a threshold.

[0514] Optionally, in the method 1000, the output includes one or more images from a camera system.

[0515] FIG. 15 illustrates a method 1100 according to some embodiments. The method 1100 may be implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit. In some embodiments, the method 1100 may be implemented by any of the image display devices 101 shown in FIGS. 1-4. For example, the method 1100 may be implemented by the processing unit 130 of the image display device 101. The method 1100 includes obtaining, by the processing unit, an output from the camera system (item 1102); determining, by the processing unit, a map based at least in part on the output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment (item 1104); and determining, by the processing unit, a score for an image obtained from the camera system, the score indicating the degree to which the image is constrained to the map segments to form the map (item 1106).

[0516] In some embodiments, in method 1100, the processing unit may determine the score by performing the calculation and obtaining the score. In other embodiments, the processing unit may determine the score by receiving the score from another component or device and obtaining the score. As a non-limiting example, the other component or device that provides the metric may be a module within the image display device 101 or an external device that communicates with the image display device, which may be worn by the user or may be physically separate from the user. For example, the external device may be a wireless transmitter, a computer, a handheld or body-worn device, a database, a server, a base station, etc.

[0517] Optionally, in method 1100, the act of determining a score includes calculating, by a processing unit, the score.

[0518] Optionally, in method 1100, the act of determining the score includes receiving, by a processing unit, the score.

[0519] Optionally, in method 1100, the score is determined based on the Jacobian of the reference point measurements.

[0520] Optionally, in method 1100, the score is determined based on an information matrix, which is a diagonal matrix.

[0521] Optionally, in method 1100, the score is determined based on the number of reference points detected in the image.

[0522] Optionally, the method 1100 further comprises performing, by the processing unit, data sanitization based on the score.

[0523] Optionally, in method 1100, the act of performing data sanitization includes removing image constraints or removing the image.

[0524] Optionally, in method 1100, the act of performing data sanitization includes removing the constraint on the image or removing the image when the score falls below a threshold.

[0525] Optionally, in method 1100, sanitization is performed to remove or not consider data that would otherwise provide an undesirable contribution to the map if the data is used to determine the map.

[0526] Optionally, in method 1100, the data comprises images from a camera system and sanitization is performed to remove or not take into account the images.

[0527] Optionally, in method 1100, the camera system comprises multiple cameras, the data comprises a set of images generated by individual cameras, and sanitization is performed to remove or not consider the set of images.

[0528] Optionally, in method 1100, the data comprises an identification of reference points in an image from a camera system, and sanitization is performed without regard to the identification of the reference points.

[0529] Optionally, in method 1100, the data represents rays or lines associated with an image from a camera system and a reference point, and sanitization is performed such that it does not take into account the rays or lines associated with the image.

[0530] Optionally, in method 1100, sanitization is performed as part of a local optimization.

[0531] Optionally, method 1100 further includes performing a bundle adjustment to adjust one or more light rays associated with one or more images from the camera system, the bundle adjustment being performed after sanitization is performed and the image for which the score is determined is one of the one or more images or is different from the one or more images.

[0532] Optionally, in method 1100, bundle adjustment is performed as part of the global optimization.

[0533] Optionally, the method 1100 further includes performing, by the processing unit, a global optimization based on one or more images from the camera system and the three-dimensional reference points.

[0534] Optionally, in method 1100, the global optimization is also performed based on the relative orientation between the cameras of the camera system.

[0535] Optionally, in method 1100, the map is determined by determining a number of map segments, where the number of map segments constitute a map segment, and connecting the map segments, where a portion of the map constitutes one of the map segments.

[0536] Optionally, in method 1100, the camera system is configured to provide additional images, the additional images being generated by the camera system before the image for which the score is determined is generated, and the act of determining the map includes a step of determining a first map segment of the map segments by linking the additional images, the additional images being generated in sequence by the camera system.

[0537] Optionally, the method 1100 further comprises determining an individual score for the additional image.

[0538] Optionally, the method 1100 further includes initiating, by the processing unit, a second one of the map segments in response to the score of the image from the camera system satisfying a criterion.

[0539] Optionally, in method 1100, a second map segment is initiated when the score indicates that the image having a degree of constraint for the first map segment falls below a threshold.

[0540] Optionally, the method 1100 further includes determining a metric indicative of the likelihood of success in locating the user using the map.

[0541] Optionally, in the method 1100, a metric is determined based on the co-visibility of points of interest associated with different camera positions.

[0542] Optionally, in the method 1100, the camera positions include a first camera position of a camera of the camera system and a second camera position of a camera of the camera system.

[0543] Optionally, in the method 1100, the camera positions include a first camera position of a first camera of the camera system and a second camera position of a second camera of the camera system.

[0544] Optionally, in method 1100, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0545] Optionally, in method 1100, the metric indicates the likelihood of success in locating the user in one or more viewing directions.

[0546] Optionally, in the method 1100, a metric is determined based on the number of times the point of interest is detected from different camera positions.

[0547] Optionally, in method 1100, the metric is determined by the processing unit without determining any convex hull.

[0548] Optionally, in the method 1100, the metric has a value that is based on the directionality.

[0549] Optionally, in method 1100, the directivity is with respect to one or more vertical axes and / or one or more horizontal axes.

[0550] Optionally, in method 1100, the directivity includes a turning direction.

[0551] Optionally, in the method 1100, the directivity includes a tilt angle.

[0552] Optionally, in the method 1100, the directivity includes a roll angle.

[0553] Optionally, in method 1100, the metric has a first value associated with the first directivity and a second value associated with the second directivity.

[0554] Optionally, in method 1100, the metric relates to one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment, and the metric has a value based on a position within the one of the plurality of cells.

[0555] Optionally, in method 1100, the metric relates to one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment, and the metric has a first value associated with a first location within the one of the plurality of cells and a second value associated with a second location within the one of the plurality of cells.

[0556] Optionally, in the method 1100, the metric is for one of a plurality of cells, each cell representing a three-dimensional space of a portion of the environment.

[0557] Optionally, in the method 1100, the act of determining the metric includes determining a total number of images from the camera system that are associated with one of the plurality of cells.

[0558] Optionally, in the method 1100, the total number of images is associated with a certain viewing direction for the cell.

[0559] Optionally, in the method 1100, the total number of images is associated with a number of viewing directions for the cell.

[0560] Optionally, in method 1100, the camera system is configured to acquire a plurality of images, the plurality of images including an image for which a score is determined, and a metric is determined for one of the plurality of cells by identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0561] Optionally, the method 1100 further comprises dividing the total score by the number of images in the subset of images to obtain an average score.

[0562] Optionally, in the method 1100, the average score is the metric.

[0563] Optionally, in the method 1100, the average score represents the average expected number of simultaneous visibility points across a range of viewing directions for one of the multiple cells.

[0564] Optionally, in method 1100, the individual scores are determined by accessing a co-visibility graph that associates reference points with multiple images.

[0565] Optionally, in the method 1100, the co-visibility graph indicates which of the reference points are visible among those in multiple images.

[0566] Optionally, in method 1100, each of the individual scores is determined by determining the number of fiducial points detected in a corresponding one of the images in the subset of images.

[0567] Optionally, the method 1100 further comprises determining an area score, which indicates coverage by the map.

[0568] Optionally, in method 1100, the area score is based on the spatial distribution of the data points in the map.

[0569] Optionally, in the method 1100, at least one of the cells has a footprint of 2m x 2m.

[0570] Optionally, in the method 1100, at least one of the cells also has a predetermined height.

[0571] Optionally, in method 1100, the metric is determined by acquiring a plurality of images from a camera system, the plurality of images including the image for which the score is determined, and determining a co-visibility value, each of the co-visibility values ​​indicating the number of reference points detected within a corresponding one of the plurality of images.

[0572] Optionally, in the method 1100, the camera system comprises multiple cameras.

[0573] Optionally, in method 1100, the plurality of images comprises a first subset of images produced by the plurality of cameras when the camera system is in a first position.

[0574] Optionally, in method 1100, the plurality of images comprises a second subset of images produced by the plurality of cameras when the camera system is in a second position.

[0575] Optionally, in the method 1100, the multiple cameras comprises a first forward-facing camera.

[0576] Optionally, in the method 1100, the multiple cameras comprises a second forward-facing camera.

[0577] Optionally, in the method 1100, the plurality of cameras comprises a first side-facing camera.

[0578] Optionally, in the method 1100, the plurality of cameras comprises a second side-facing camera.

[0579] Optionally, the method 1100 further comprises determining, by the processing unit, a desired viewing direction of the camera system to refine the value of the metric.

[0580] Optionally, the method 1100 further includes generating a graphic based on the determined desired viewing direction, the graphic being configured to instruct the user to change the current viewing direction of the camera system to the desired viewing direction.

[0581] Optionally, the method 1100 further includes acquiring an additional image from the camera system after a desired viewing direction of the camera system is achieved.

[0582] Optionally, the method 1100 further includes updating the map based on the additional imagery.

[0583] Optionally, the method 1100 further includes updating the metrics based on the updated map.

[0584] Optionally, in method 1100, the metric is determined before the processing unit uses the map to locate the user relative to the environment.

[0585] Optionally, in method 1100, the metric is determined before a device shares content with another device.

[0586] Optionally, in method 1100, the metric is determined during a map building session in which the processing unit determines the map.

[0587] Optionally, in method 1100, the metric is determined retrospectively by accessing a previously determined map from a non-transient medium.

[0588] Special Treatment Systems

[0589] In some embodiments, the methods 500, 900, 1000, and / or 1100 described herein may be performed by a processing unit 130 executing an application or by an application. The application may contain a set of instructions. In one implementation, a specialized processing system may be provided having a non-transitory medium that stores a set of instructions for an application. Execution of the instructions by the processing unit 130 of the image display device 101 will cause the processing unit 130 to implement the features described herein.

[0590] In some embodiments, image display device 101 may also be considered a specialized processing system. In particular, image display device 101 is a specialized processing system in that it contains instructions stored in its non-transitory medium for execution by processing unit 130 to provide unique tangible effects in the real world. The features provided by image display device 101 (as a result of processing unit 130 executing the instructions) provide improvements in the art of localization map creation, image-based localization, and quality control for localization maps, as described herein.

[0591] As a non-limiting example, instructions may be provided that include determining a map based at least in part on output from a camera system; locating a user relative to the environment based on the map; determining a metric indicative of likelihood of success in locating the user using the map; determining a metric based on simultaneous visibility of points of interest associated with different camera positions; determining a number of reference points that can be used to locate the user relative to the environment; determining a metric indicative of likelihood of success in locating the user in one or more viewing directions; determining a metric based on a number of times points of interest are detected from different camera positions; determining a metric without determining any convex hull; and determining a metric based on directionality, where the directionality is relative to one or more vertical axes and / or one or more horizontal axes; determining a metric without determining any convex hull; and determining a metric based on directionality, where the directionality is relative to a rotation direction, a tilt angle, a roll angle. or any combination thereof; determining a metric for one of the plurality of cells, the cells each representing a three dimensional space of a portion of the environment, the metric having a first value associated with a first directionality and a second value associated with a second directionality; determining a metric for one of the plurality of cells, the cells each representing a three dimensional space of a portion of the environment, the metric having a first value associated with a first position within the one of the plurality of cells and a second value associated with a second position within the one of the plurality of cells; determining a metric for one of the plurality of cells, the cells each representing a three dimensional space of a portion of the environment; determining a total number of images from the camera system for one of the plurality of cells, the total number of images being associated with a view direction for the cell or being associated with a plurality of view directions for the cell; generating a control signal;determining a metric for one of the plurality of cells by causing a camera system to acquire a plurality of images, identifying a subset of images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score; determining an average score by dividing the total score by the number of images in the subset of images; determining the average score representing an average expected number of co-visibility points over the range of viewing directions for one of the plurality of cells; determining the individual scores by accessing a co-visibility graph that associates reference points with the plurality of images, the co-visibility graph indicating which of the reference points are visible among those in the plurality of images; determining each of the individual scores by determining a number of reference points detected in a corresponding one of the images in the subset of images; determining an area score indicative of coverage by the map; determining the area score based on a spatial distribution of the data points of the map; and from the camera system; determining a simultaneous visibility value, each simultaneous visibility value indicating a number of reference points detected in a corresponding one of the plurality of images; generating control signals and operating the plurality of cameras, the plurality of cameras comprising a first forward facing camera, a second forward facing camera, a first side facing camera, and a second side facing camera; determining a desired viewing direction of the camera system to refine the value of the metric; generating a graphic based on the determined desired viewing direction, the graphic being configured to instruct a user to change a current viewing direction of the camera system to the desired viewing direction; generating control signals and operating the camera system to obtain images of the environment after the desired viewing direction of the camera system is achieved; updating the map based on the images; and updating the metric based on the updated map; and prior to locating the user relative to the environment using the map.determining a metric for the map; determining a metric for the map before allowing a device to share content with another device; determining the metric during a map building session in which a processing unit determines the map; determining the metric retroactively by accessing a previously determined map from a non-transient medium; performing sanitization when data is used to determine the map, removing or not considering data that would otherwise provide an undesirable contribution to the map, the data comprising a set of images generated by individual cameras including images from a camera system, comprising identification of reference points within images from the camera system, or representing rays or rays and reference points associated with images from the camera system; performing the sanitization as part of a local optimization; performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system; performing the sanitization and removing the data. performing a bundle adjustment after determining the number of reference points detected in the image; performing a bundle adjustment as part of the global optimization; performing a global optimization based on images and 3D reference points from the camera system; performing a global optimization also based on a relative orientation between the cameras of the camera system; determining a score for the image acquired from the camera system, the score being a constraint score and / or indicating the degree to which the image is constrained; determining a score based on the Jacobian of the reference point measurements; determining a score based on an information matrix, which is a diagonal matrix; determining a score based on a number of reference points detected in the image; performing data sanitization based on the score; removing the constraints on the image or removing the image when performing data sanitization; removing the constraints on the image or removing the image when the score is below a threshold; determining a plurality of map segments; and connecting the map segments.The method may include instructions for determining a map segment, determining a first one of the map segments by acquiring images from a camera system and linking the images, the images being generated in a sequence by the camera system, determining an individual score for the images, and in response to the score of the additional image from the camera system satisfying a criterion, acquiring an additional image from the camera system, determining a score for the additional image, and initiating a second one of the map segments, and initiating the second map segment when the score indicates that the image having a degree of constraint for the first map segment is below a threshold.

[0592] FIG. 16 is a block diagram illustrating an embodiment of a specialized processing system 1600 that may be used to implement various features described herein. For example, in some embodiments, the processing system 1600 may be used to implement the image display device 101. The processing system 1600 includes a bus 1602 or other communication mechanism for communicating information and a processor 1604 coupled with the bus 1602 for processing information. The processor system 1600 also includes a main memory 1606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1602 for storing information and instructions to be executed by the processor 1604. The main memory 1606 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 1604. The processor system 1600 further includes a read only memory (ROM) 1608 or other static storage device coupled to the bus 1602 for storing static information and instructions for the processor 1604. A data storage device 1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to bus 1602 for storing information and instructions.

[0593] The processor system 1600 may be coupled to a display 1612, such as a screen, via the bus 1602 to display information to a user. In some cases, if the processing system 1600 is part of a device that includes a touch screen, the display 1612 may be a touch screen. An input device 1614, including alphanumeric and other keys, is coupled to the bus 1602 to communicate information and command selections to the processor 1604. Another type of user input device is a cursor control 1616, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 1604 and for controlling cursor movement on the display 1612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to define a position in a plane. In some cases, if the processing system 1600 is part of a device that includes a touch screen, the input device 1614 and cursor control may be a touch screen.

[0594] In some embodiments, the processor system 1600 can be used to perform various functions described herein. According to some embodiments, such use is provided by the processor system 1600 in response to the processor 1604 executing one or more sequences of one or more instructions contained in the main memory 1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into the main memory 1606 from another processor-readable medium, such as the storage device 1610. Execution of the sequences of instructions contained in the main memory 1606 causes the processor 1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in the main memory 1606. In alternative embodiments, hard-wired circuitry may be used in place of, or in combination with, software instructions to implement various embodiments described herein. Thus, the embodiments are not limited to any specific combination of hardware circuitry and software.

[0595] The term "processor-readable medium," as used herein, refers to any medium that participates in providing instructions to the processor 1604 for execution. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical, solid-state, or magnetic disks, such as the storage devices 1610. Non-volatile media may be considered an example of non-transitory media. Volatile media includes dynamic memory, such as the main memory 1606. Volatile media may be considered an example of non-transitory media. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 1602. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0596] Common forms of processor-readable media include, for example, a floppy disk, a hard disk, a magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, any other physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, a solid-state disk, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a processor can read.

[0597] Various forms of processor-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a network, such as the Internet. The processing system 1600 may receive data over a network line. The bus 1602 carries the data to the main memory 1606, from which the processor 1604 reads and executes the instructions. The instructions received by the main memory 1606 may optionally be stored on a storage device 1610 either before or after execution by the processor 1604.

[0598] The processing system 1600 also includes a communication interface 1618 coupled to the bus 1602. The communication interface 1618 provides a two-way data communication coupling to a network link 1620 that is connected to a local network 1622. For example, the communication interface 1618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 1618 sends and receives electrical, electromagnetic, or optical signals that carry data streams representing various types of information.

[0599] The network link 1620 typically provides data communication through one or more networks to other devices. For example, the network link 1620 may provide a connection through a local network 1622 to a host computer 1624 or a device 1626. Data streams transported via the network link 1620 may include electrical, electromagnetic, or optical signals. The signals through the various networks and the signals on the network link 1620 and through the communication interface 1618, which carry the data to and from the processing system 1600, are exemplary forms of carrier waves transporting information. The processing system 1600 can send messages and receive data, including program code, through the networks, the network link 1620, and the communication interface 1618.

[0600] Exemplary aspects of the present disclosure have been described above, together with details regarding material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the aforementioned reference patents and publications, and are generally known or can be understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional actions as generally or logically adopted.

[0601] In addition, although the present disclosure has been described with reference to several embodiments, optionally incorporating various features, the present disclosure is not limited to those described or illustrated as each variation of the disclosure is considered. Various modifications may be made to the present disclosure as described, and equivalents (whether listed herein or not included for purposes of some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of the range, and any other stated value or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0602] It is also contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. Furthermore, it is noted that any claim may be drafted to exclude any optional element. Thus, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0603] The scope of the present disclosure should not be limited to the examples provided and / or specification of the subject matter, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0604] The following are example features. One or more of the following features may be combined in one or more embodiments.

[0605] Feature 1. A device configured to be worn on a user's head includes a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit configured to determine a map based at least in part on output from the camera system, the map configured for use by the processing unit to locate the user relative to the environment, wherein the processing unit of the device is also configured to obtain a metric indicative of a likelihood of success of locating the user using the map, and the processing unit is configured to obtain the metric by calculating the metric or by receiving the metric.

[0606] Feature 2. In some embodiments, the processing unit is configured to determine a metric based on the co-visibility of points of interest associated with different camera positions.

[0607] Feature 3. In some embodiments, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0608] Feature 4. In some embodiments, a metric indicates the likelihood of success in locating a user in one or more viewing directions.

[0609] Feature 5. In some embodiments, the processing unit is configured to determine a metric based on the number of times the point of interest is detected from different camera positions.

[0610] Feature 6. In some embodiments, the processing unit is configured to determine the metric without determining any convex hull.

[0611] Feature 7. In some embodiments, the metric has a value that is based on directionality.

[0612] Feature 8. In some embodiments, the metric is for one of a plurality of cells, each cell representing a three dimensional space of a portion of the environment, and the metric has a value based on a position within the one of the plurality of cells.

[0613] Feature 9. In some embodiments, the metric is for one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment.

[0614] Feature 10. In some embodiments, the camera system is configured to acquire a plurality of images, and the processing unit is configured to determine a metric for one of the plurality of cells by identifying a subset of the images that belong to the same range of viewing directions, determining individual scores for the images in the subset of images, and summing the scores to obtain a total score.

[0615] Feature 11. In some embodiments, the processing unit is also configured to determine an average score by dividing the total score by the number of images in the subset of images.

[0616] Feature 12. In some embodiments, the processing unit is configured to determine the individual scores by accessing a co-visibility graph that associates reference points with multiple images.

[0617] Feature 13. In some embodiments, a co-visibility graph indicates which of the reference points are visible among those in multiple images.

[0618] Feature 14. In some embodiments, the processing unit is configured to determine each of the individual scores by determining a number of reference points detected in a corresponding one of the images in the subset of images.

[0619] Feature 15. In some embodiments, the processing unit is also configured to determine an area score, indicative of coverage by the map.

[0620] Feature 16. In some embodiments, the processing unit is configured to determine the metric by acquiring a plurality of images from a camera system and determining joint visibility values, each joint visibility value indicating a number of reference points detected in a corresponding one of the plurality of images.

[0621] Feature 17. In some embodiments, the processing unit is configured to determine a desired viewing direction of the camera system to refine the value of the metric.

[0622] Feature 18. In some embodiments, the processing unit is configured to generate a graphic based on the determined desired viewing direction, and the graphic is configured to instruct a user to change a current viewing direction of the camera system to the desired viewing direction.

[0623] Feature 19. In some embodiments, the processing unit is configured to update the metric after a desired viewing direction is achieved.

[0624] Feature 20. In some embodiments, the processing unit is configured to perform sanitization when the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0625] Feature 21. In some embodiments, the data includes images from a camera system, and the processing unit is configured to perform sanitization by (1) removing or not considering the images, (2) not considering identification of reference points in the images, and / or (3) not considering rays or lines associated with the images.

[0626] Feature 22. In some embodiments, the processing unit is configured to perform bundle adjustment and adjust one or more light rays associated with one or more images from the camera system, and the processing unit is configured to perform the bundle adjustment after performing sanitization and removing data.

[0627] Feature 23. In some embodiments, the processing unit is configured to perform the optimization based on images from the camera system, the three-dimensional reference points, and a relative orientation between the cameras of the camera system.

[0628] Feature 24. In some embodiments, the processing unit is configured to determine a score for an image obtained from the camera system.

[0629] Feature 25. In some embodiments the score indicates the degree to which the image is constrained.

[0630] Feature 26. In some embodiments, the processing unit is configured to determine the score based on a Jacobian of the reference point measurements.

[0631] Feature 27. In some embodiments, the processing unit is configured to perform data sanitization based on the score, and the processing unit is configured to remove the image constraint or remove the image when performing the data sanitization.

[0632] Feature 28. In some embodiments, the processing unit is configured to determine the map by determining a plurality of map segments and connecting the map segments.

[0633] Feature 29. In some embodiments, the processing unit is configured to determine a first map segment of the map segments by acquiring images from a camera system and linking the images, the images being generated in a sequence by the camera system.

[0634] Feature 30. In some embodiments, the processing unit is configured to obtain an additional image from the camera system, determine a score for the additional image, and initiate a second one of the map segments in response to the score of the additional image from the camera system satisfying a criterion.

[0635] Feature 31. In some embodiments, the processing unit is configured to initiate a second map segment when the score indicates that an image having a degree of constraint on the first map segment is below a threshold.

[0636] Feature 32. A method performed by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method including the steps of: acquiring, by the processing unit, an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and acquiring, by the processing unit, a metric indicative of a likelihood of success in locating the user using the map, the act of acquiring including, by the processing unit, calculating the metric or receiving the metric.

[0637] Feature 33. In some embodiments, a metric is determined based on the simultaneous visibility of points of interest associated with different camera positions.

[0638] Feature 34. In some embodiments, the metric indicates the number of reference points that can be used to locate the user relative to the environment.

[0639] Feature 35. In some embodiments a metric indicates the likelihood of success in locating a user in one or more viewing directions.

[0640] Feature 36. In some embodiments, a metric is determined based on the number of times the point of interest is detected from different camera positions.

[0641] Feature 37. In some embodiments, the metric is determined by the processing unit without determining any convex hull.

[0642] Feature 38. In some embodiments, the metric has a value that is based on directionality.

[0643] Feature 39. In some embodiments, the metric is for one of a plurality of cells, each cell representing a three dimensional space of a portion of the environment, and the metric has a value based on a position within the one of the plurality of cells.

[0644] Feature 40. In some embodiments, the metric is for one of a plurality of cells, each of which represents a three-dimensional space of a portion of the environment.

[0645] Feature 41. In some embodiments, the camera system is configured to acquire a plurality of images, and a metric is determined for one of the plurality of cells by identifying a subset of the images that belong to the same range of viewing directions, determining individual scores for images in the subset of images, and summing the scores to obtain a total score.

[0646] Feature 42. In some embodiments, the metric is determined by dividing the total score by the number of images in the subset of images and taking the average score.

[0647] Feature 43. In some embodiments, the individual scores are determined by accessing a co-visibility graph that associates reference points with multiple images.

[0648] Feature 44. In some embodiments, a co-visibility graph indicates which of the reference points are visible among those in multiple images.

[0649] Feature 45. In some embodiments, each of the individual scores is determined by determining the number of fiducial points detected in a corresponding one of the images in the subset of images.

[0650] Feature 46. In some embodiments, the method also includes determining an area score, indicative of coverage by the map.

[0651] Feature 47. In some embodiments, the metric is determined by acquiring a plurality of images from a camera system and determining co-visibility values, each of the co-visibility values ​​indicating a number of reference points detected in a corresponding one of the plurality of images.

[0652] Feature 48. In some embodiments, the method also includes determining, by the processing unit, a desired viewing direction of the camera system to refine the value of the metric.

[0653] Feature 49. In some embodiments, the method also includes generating a graphic based on the determined desired viewing direction, the graphic configured to instruct a user to change a current viewing direction of the camera system to the desired viewing direction.

[0654] Feature 50. In some embodiments, the method also includes updating the metric after the desired viewing direction is achieved.

[0655] Feature 51. In some embodiments, the method also includes performing sanitization, if the data is used to determine the map, to remove or not consider data that would otherwise provide an undesirable contribution to the map.

[0656] Feature 52. In some embodiments, the data includes images from a camera system, and sanitization is performed by (1) removing or not considering the image, not considering identification of reference points in the image, and / or (3) not considering rays or lines associated with the image.

[0657] Feature 53. In some embodiments, the method also includes performing a bundle adjustment to adjust one or more light rays associated with one or more images from the camera system, the bundle adjustment being performed after sanitization is performed to remove the data.

[0658] Feature 54. In some embodiments, the processing unit is configured to perform the optimization based on images from the camera system, the three-dimensional reference points, and a relative orientation between the cameras of the camera system.

[0659] Feature 55. In some embodiments, the method also includes determining, by the processing unit, a score for an image obtained from the camera system.

[0660] Feature 56. In some embodiments the score indicates the degree to which the image is constrained.

[0661] Feature 57. In some embodiments, the score is determined based on the Jacobian of the reference point measurements.

[0662] Feature 58. In some embodiments, the method also includes performing data sanitization based on the score, where the data sanitization is performed to remove the image constraint or to remove the image.

[0663] Feature 59. In some embodiments, the map is determined by determining a plurality of map segments and connecting the map segments.

[0664] Feature 60. In some embodiments, the act of determining the plurality of map segments includes determining a first map segment of the map segments by obtaining images from a camera system and linking the images, the images being generated in a sequence by the camera system.

[0665] Feature 61. In some embodiments, the method also includes, in response to the score of the additional image from the camera system satisfying a criterion, acquiring an additional image from the camera system, determining a score for the additional image, and initiating a second one of the map segments.

[0666] Feature 62. In some embodiments, a second map segment is initiated when the score indicates that the image having a degree of constraint on the first map segment falls below a threshold.

[0667] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A method implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising: the processing unit acquiring an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and determining a score for an image acquired from the camera system, the score indicating the degree to which the image is constrained to the map segments to form the map; A method comprising:

2. The method of claim 1 , wherein the score is determined based on a number of fiducial points detected in the image.

3. The method of claim 1 , further comprising the processing unit performing data sanitization based on the score.

4. The method of claim 3 , wherein the act of performing data sanitization includes removing a constraint on the image or removing the image.

5. The method of claim 4 , wherein the act of performing data sanitization includes removing the constraints on the image or removing the image when the score falls below a threshold.

6. 4. The method of claim 3, wherein the data sanitization is performed to remove or not consider the data that would otherwise provide an undesirable contribution to the map if the data is used to determine the map.

7. 7. The method of claim 6, wherein the camera system comprises a plurality of cameras, the data comprises a set of images produced by each camera, and the data sanitization is performed to remove or not consider the set of images.

8. performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, the bundle adjustment being performed after the data sanitization is performed, and the image for which the score is determined is one of the one or more images or is different from the one or more images; The method of claim 3 , wherein the bundle adjustment is performed as part of a global optimization.

9. the processing unit further performing the global optimization based on the one or more images from the camera system and the three-dimensional reference points; The method of claim 8 , wherein the global optimization is also performed based on a relative orientation between cameras of the camera system.

10. The map may include: determining a plurality of map segments, the plurality of map segments including the map segment; connecting said plurality of map segments; and the portion of the map includes one of the plurality of map segments.

11. the camera system is configured to provide an additional image, the additional image being generated by the camera system before the image for which the score is determined is generated; 2. The method of claim 1, wherein the act of determining the map includes determining a first one of the map segments by linking the additional images, the additional images being generated in sequence by the camera system.

12. determining a score for each of said additional images; the processing unit initiating a second one of the map segments in response to the score of the image from the camera system satisfying a criterion; The method of claim 11 further comprising:

13. The map may include: determining a plurality of map segments, the plurality of map segments including the map segment; connecting said plurality of map segments; and the portion of the map includes one of the plurality of map segments.

14. the camera system is configured to provide an additional image, the additional image being generated by the camera system before the image for which the score is determined is generated; 14. The method of claim 13, wherein the act of determining the map includes determining a first one of the map segments by linking the additional images, the additional images being generated in sequence by the camera system.

15. determining a score for each of said additional images; the processing unit initiating a second one of the map segments in response to the score of the image from the camera system satisfying a criterion; The method of claim 14 further comprising:

16. The method of claim 1 , further comprising determining a metric indicative of a likelihood of success in locating the user using the map.

17. The method of claim 16 , wherein the metric is determined based on the co-visibility of points of interest associated with different camera positions.

18. the metric is for one of a plurality of cells, each of the plurality of cells representing a three-dimensional space of a portion of the environment; the camera system is configured to capture a plurality of images, the plurality of images including the image for which the score is determined; The metric is: identifying a subset of said images that belong to the same range of viewing directions; determining a respective score for images in the subset of images; Adding up the individual scores to obtain a total score. The method of claim 16 , wherein the determination for one of the plurality of cells is made by:

19. 1. A method implemented by a device configured to be worn on a user's head, the device having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising: the processing unit acquiring an output from the camera system; determining, by the processing unit, a map based at least in part on the output from the camera system, the map being configured for use by the processing unit to locate the user relative to the environment; and obtaining a score for an image obtained from the camera system, the score indicating the degree to which the image is constrained to a map segment for forming the map; wherein the act of obtaining the score includes calculating the score or receiving the score.

20. 1. A device configured to be worn on a user's head, comprising: a screen configured to present a graphic to the user; a camera system configured to view an environment in which the user is located; a processing unit configured to determine a map based at least in part on output from the camera system; and the map is configured for use by the processing unit to locate the user relative to the environment; The apparatus, wherein the processing unit is configured to determine a score for an image acquired from the camera system, the score indicating the degree to which the image is constrained relative to a map segment to form the map.

Images