Wearable system with controller position identification using headset camera and controller reference

Abstract

A wearable system incorporating headset and controller position identification using headset camera and controller criteria, and a method for its operation are disclosed. The wearable system may include a headset and a controller. The wearable system may alternately perform headset tracking and controller tracking by repeatedly capturing images using the headset camera of the headset in a headset tracking frame and a controller tracking frame. The wearable system may cause the headset camera to capture a first exposure image having an exposure exceeding a threshold, and cause the headset camera to capture a second exposure image having an exposure less than the threshold. The wearable system may determine a reference interval, and in the reference interval, the reference of the controller flashes at a reference frequency and a reference period. The wearable system may cause the reference to flash in the reference interval according to the reference frequency and the reference period.

Description

Technical Field

[0001]

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 345,159, filed May 24, 2022, entitled "WEARABLE SYSTEM WITH CONTROLLER LOCALIZATION USING HEADSET CAMERAS AND CONTROLLER FIDUCIALS", and U.S. Provisional Patent Application No. 63 / 345,162, filed May 24, 2022, entitled "WEARABLE SYSTEM WITH HEADSET AND CONTROLLER INSIDE - OUT TRACKING", the entire disclosure of which is incorporated herein by reference for all purposes.

Background Art

[0002]

[0002] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a way that they appear to be real or can be perceived as real. Virtual reality, i.e., "VR" scenarios, typically involve presenting digital or virtual image information that is opaque to other actual real - world visual inputs, and augmented reality, i.e., "AR" scenarios, typically involve presenting digital or virtual image information as an extension of the visualization of the actual world around the user.

[0003]

[0003] Despite these advances in display technologies, improved methods, systems, and devices related to augmented reality systems are needed in the art.

Summary of the Invention

[0004]

[0004] This disclosure generally relates to techniques for improving the performance and user experience of optical systems. More particularly, embodiments of the present disclosure provide methods for operating an augmented reality (AR), virtual reality (VR), or mixed reality (MR) wearable system in which a handheld device is employed to assist in the operation of the wearable system. While a portion of the present disclosure is described with reference to an AR system, the present disclosure is applicable to a variety of applications.

Means for Solving the Problems

[0005]

[0005] An overview of various embodiments of the present invention is provided below as a list of examples. As used below, any reference to a series of examples should be understood disjunctively as a reference to each of those examples (e.g., "Examples 1-4" should be understood as "Example 1, 2, 3, or 4").

[0006]

[0006] Example 1 is a method of operating a wearable system having a headset and a controller, the method comprising alternately performing headset tracking and controller tracking by repeatedly capturing images using a headset camera of the headset in a headset tracking frame and a controller tracking frame respectively; in each of the headset tracking frames, causing the headset camera to capture a first exposure image among images having an exposure exceeding a threshold, the first exposure image being associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration; in each of the controller tracking frames, causing the headset camera to capture a second exposure image among images having an exposure less than the threshold, the second exposure image being associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, the second exposure duration being shorter than the first exposure duration; determining a reference interval defined by a reference start time and a reference end time, in the reference interval, a set of references of the controller flashes a plurality of times at a reference frequency and a reference period, the reference interval being determined such that the second exposure interval at least partially overlaps the reference interval; and flashing the set of references a plurality of times in the reference interval according to the reference frequency and the reference period.

[0007]

[0007] Example 2 is the method of Example 1, wherein the wearable system comprises a headset comprising a headset camera and a headset inertial measurement unit, and a controller comprising a set of references arranged in a known geometric shape, one or more controller cameras, and a controller inertial measurement unit, and the wearable system is configured to determine the position or orientation of the headset or the controller based on data captured by the headset camera, one or more controller cameras, the headset inertial measurement unit, or the controller inertial measurement unit.

[0008]

[0008] Example 3 is the method of Example 2, including determining a first posture of the headset relative to a reference frame based on data captured by a headset camera of the headset or a headset inertial measurement unit of the headset for operating a wearable system, flushing a set of references of a controller, capturing a headset image using the headset camera for a second posture of the controller relative to the headset, identifying a set of references within the headset image, and determining the second posture of the controller relative to the headset based on the set of references identified within the headset image and a known geometric shape.

[0009]

[0009] Example 4 is the method of Example 1, where a reference interval is determined such that a second exposure interval is centered with the reference interval.

[0010]

[0010] Example 5 is the method of Example 1, where a first time duration of a certain headset tracking frame among headset tracking frames is equal to a second time duration of a certain controller tracking frame among controller tracking frames.

[0011]

[0011] Example 6 is the method of Example 1, where a first exposure duration includes at least 1 millisecond.

[0012] Example 7 is a method of operating a wearable system comprising a headset, the method comprising capturing a set of images using a headset camera of the headset, identifying a plurality of criteria within the set of images being repeatedly flashed, determining that at least some of the plurality of criteria include a first set of criteria belonging to a first controller and a second set of criteria belonging to a second controller different from the first controller, determining that the flashing of the first set of criteria is at least partially temporally aligned with the flashing of the second set of criteria, and causing a modification of a period, frequency, or offset associated with at least one of the first set of criteria or the second set of criteria to shift the flashing of the first set of criteria and the second set of criteria.

[0013]

[0013] Example 8 is the method of Example 7, wherein causing the modification comprises causing the first controller to modify a first period, a first frequency, or a first offset associated with the first set of criteria to shift the flashing of the first set of criteria and the second set of criteria.

[0014]

[0014] Example 9 is the method of Example 8, further comprising causing the second controller to modify a second period, a second frequency, or a second offset associated with the second set of criteria to shift the flashing of the first set of criteria and the second set of criteria.

[0015]

[0015] Example 10 is the method of Example 7, wherein causing the modification comprises causing the first controller to modify the flashing of the first set of criteria encoded by the first coding.

[0016]

[0016] Example 11 is the method of Example 10, further comprising causing the second controller to modify the flashing of the second set of criteria encoded by the second coding.

[0017]

[0017] Example 12 further includes performing the method of method 7 by causing a head-mounted camera to capture an image from a set of images having an exposure less than a threshold value for controller tracking of a first controller, where the image is associated with an exposure interval defined by an exposure start time, an exposure end time, and an exposure duration; determining a reference interval defined by a reference start time and a reference end time, where a first set of first references of the first controller flashes a plurality of times at a reference frequency and a reference period during the reference interval, and the reference interval is determined such that the exposure interval at least partially overlaps the reference interval; and causing the first set of references to flash a plurality of times during the reference interval according to the reference frequency and the reference period.

[0018]

[0018] Example 13 is a method of example 7, where the first controller comprises a first set of references arranged in a known geometric shape, the second controller comprises a second set of references arranged in a known geometric shape, and the method further includes causing a first controller to flash a first subset of the first set of references and causing a second controller to flash a second subset of the second set of references, where the first subset and the second subset are asymmetric with respect to each other.

[0019]

[0019] Example 14 is a method of operating a wearable system comprising a head-mounted device and a controller having a display, the method including causing the controller to display a set of references on the display according to a set of pixel positions; using a head-mounted camera of the head-mounted device to capture a set of images; identifying the set of references within the set of images; and determining a position and / or orientation of the controller with respect to the head-mounted device based on the identified set of references within the set of images.

[0020] Example 15 is the method of Example 14, further including flashing a set of references according to a period and a frequency.

[0021] Example 16 is the method of Example 15, further including causing a controller to modify a set of a period, a frequency, or a pixel position.

[0022] Example 17 is the method of Example 14, further including identifying a set of second references belonging to a second controller within a set of images, and in response to identifying the set of second references, causing the controller to modify a set of a period, a frequency, or a pixel position.

[0023] Example 18 is the method of Example 17, further including identifying a first geometric shape of the set of second references, and causing the controller to display the set of references in a second geometric shape different from the first geometric shape.

[0024] Example 19 is the method of Example 14, further including synchronizing the display of the set of references with at least one exposure interval of a headset camera.

[0025]

[0025] Example 20 includes at least one exposure interval including a first exposure interval in a headset tracking frame and a second exposure interval in a controller tracking frame, where the headset tracking frame causes the headset camera to capture a first exposure image from a set of images having an exposure exceeding a threshold, the first exposure image being associated with the first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration, and the controller tracking frame causes the headset camera to capture a second exposure image from a set of images having an exposure less than the threshold, the second exposure image being associated with the second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, the second exposure duration being shorter than the first exposure duration, and determining a reference interval defined by a reference start time and a reference end time, where a set of references flashes a plurality of times at a reference frequency and a reference period during the reference interval, and the reference interval is determined such that the second exposure interval at least partially overlaps with the reference interval, and is a method of Example 19.

[0026]

[0026] Example 21 further includes causing the controller to display one or more buttons configured to receive user input on a display according to a second set of pixel positions, and is a method of Example 14.

[0027]

[0027] Example 22 is a method of Example 14 where the controller includes a mobile device.

[0028]

[0028] Example 23 is a modular controller for use in a wearable system, the modular controller comprising one or more of a set of components including a visual inertial odometry (VIO) module, a constellation module, a main printed circuit board (PCB), a battery, a wireless communication engine, a user input including at least one of (i) a trigger, (ii) a bumper, or (iii) a touchpad, a tactile engine, and / or a user indicator, and one or more of the set of components being removably or addable independently while maintaining at least some of the functions of the modular controller.

[0029]

[0029] Example 24 is the modular controller of Example 23, wherein the modular controller is powered and communicates by a universal serial bus (USB) connection.

[0030]

[0030] Example 25 is the modular controller of Example 23, wherein the modular controller has a minimum size of 84 mm in length, 64 mm in width, and 18 mm in thickness.

[0031]

[0031] Example 26 is the modular controller of Example 23, wherein the modular controller has a minimum size of 64 mm in diameter and 18 mm in thickness.

[0032]

[0032] Example 27 is the modular controller of Example 23, wherein the modular controller has a minimum size of 50 mm in diameter and 15 mm in thickness.

[0033]

[0033] Example 28 is the modular controller of Example 23, wherein the modular controller is integrated into a drone controllable by an application on a wearable system, and the wearable system is configured to identify a set of references of the constellation module to locate the drone.

[0034]

[0034] Example 29 is a method of operating a wearable system having a headset and a controller, the method comprising flashing a set of reference of the controller, the set of reference being arranged in a known geometric shape including a plurality of groups of reference that are rotationally symmetric with respect to each other, the quantity of each of the plurality of groups of reference being equal to a predetermined number, the predetermined number being at least three, flashing, causing an image to be captured by a headset camera of the headset, identifying a set of objects in the image corresponding to the reference, repeatedly selecting a subset of the set of objects based on the known geometric shape, each of the subsets having a number equal to the predetermined number, selecting, and calculating the posture of the controller by associating the subset with the plurality of groups of reference, calculating a statistical value of the associated subsets based on the compatibility of the postures of the plurality of groups of reference, and associating the set of objects with the set of reference by finding a correct association between the set of objects and the set of reference based on the calculated statistical value.

[0035]

[0035] Example 30 is the method of Example 29, wherein calculating the posture includes inputting a subset of the set of objects into a three-point perspective algorithm configured to output the posture of the controller.

[0036]

[0036] Example 31 is the method of Example 29, wherein the known geometric shape includes a first gap between a first pair of adjacent references of the set of reference that is larger than other gaps between other pairs of adjacent references of the set of reference.

[0037]

[0037] Example 32 is the method of Example 29, wherein the set of objects corresponds to a set of reference projected in the image and / or one or more light sources.

[0038] Example 33 is the method of Example 29, which includes calculating statistical values of associated subsets and determining the number of sets of objects aligned with a set of criteria for each posture.

[0039] Example 34 is the method of Example 29, further including identifying a second set of criteria belonging to a second controller in an image and, in response to identifying the second set of criteria, causing the controller to modify the period, frequency, or offset of the flash of the set of criteria to shift the set of criteria from the second set of criteria.

[0040] Example 35 is a method of operating a wearable system having a headset and a controller, the method including flashing a set of criteria of the controller, where the set of criteria is arranged in a known geometric shape, causing the headset camera of the headset to capture an image, identifying a set of objects in the image corresponding to the criteria, capturing rotational measurement values using the controller inertial measurement unit of the controller, repeatedly selecting a subset of the set of objects based on the known geometric shape and associating the subset with a plurality of groups of criteria to calculate the posture of the controller, calculating statistical values of the associated subsets based on the compatibility of the postures of the plurality of groups of criteria and based on the rotational measurement values, and associating the set of objects with the set of criteria by finding the correct association between the set of objects and the set of criteria based on the calculated statistical values.

[0041] Example 36 is the method of Example 35, where calculating the posture includes inputting a subset of the set of objects into a two-point perspective algorithm configured to output the posture of the controller.

[0042]

[0042] Example 37 is the method of Example 36, where the two-point perspective algorithm includes a gravity-based two-point perspective algorithm.

[0043]

[0043] Example 38 is the method of Example 35, where the set of objects corresponds to a set of references projected into the image and / or one or more light sources.

[0044]

[0044] Example 39 is the method of Example 35, where calculating the statistical value of the associated subset includes determining the number of sets of objects aligned with the set of references for each pose.

[0045]

[0045] Example 40 further includes identifying a second set of references belonging to a second controller in the image and, in response to identifying the second set of references, causing the controller to modify the period, frequency, or offset of the flash of the set of references to shift the set of references from the second set of references.

[0046]

[0046] Example 41 is the method of Example 35, where the wearable system includes a headset with a headset camera and a headset inertial measurement unit, and a controller with a set of references arranged in a known geometric shape, one or more controller cameras, and a controller inertial measurement unit, and the wearable system is configured to determine the position or orientation of the headset or the controller based on data captured by the headset camera, one or more controller cameras, the headset inertial measurement unit, or the controller inertial measurement unit.

[0047]

[0047] Example 42 is a method of operating a wearable system having a headset and a controller, the method comprising: flashing a set of references of the controller, the set of references being arranged in a known geometric shape; causing the headset camera of the headset to capture an image; identifying a set of objects in the image corresponding to the references; identifying the position of a hand in the image using hand-tracking data; determining a region of interest in the image based on the position of the hand in the image; excluding a first subset of the set of objects outside the region of interest; and associating a second subset of the set of objects inside the region of interest with the set of references, the first subset and the second subset being mutually exclusive, thereby associating the set of objects with the set of references.

[0048]

[0048] Example 43 is the method of Example 42, further comprising: identifying the orientation of a hand in the image using hand-tracking data; and determining a region of interest in the image based on the orientation of the hand in the image.

[0049]

[0049] Example 44 is the method of Example 43, wherein determining a region of interest in the image based on the orientation of the hand in the image includes tilting the region of interest in a direction in which the controller is held according to the orientation.

[0050]

[0050] Example 45 is a method of operating a wearable system having a headset and a controller, the method comprising: flashing a set of references of the controller, the set of references being arranged in a known geometric shape; causing the headset camera of the headset to capture an image; identifying the position of a hand in the image using hand-tracking data; determining a region of interest in the image based on the position of the hand in the image; identifying a set of objects in the region of interest in the image corresponding to the references; and associating the set of objects in the region of interest with the set of references.

[0051]

[0051] Example 46 is the method of Example 45, further including identifying the orientation of a hand in an image using hand-tracking data and determining a region of interest based on the orientation of the hand in the image.

[0052]

[0052] Example 47 is the method of Example 45, wherein determining a region of interest based on the orientation of a hand in an image includes tilting the region of interest in a direction in which the controller is held according to the orientation.

[0053]

[0053] Example 48 is a method of operating a wearable system having a headset and a controller, the method including maintaining a calibration profile that models the physical relationship between a first headset camera of the headset and a second headset camera of the headset; flushing a set of references of the controller, the set of references being arranged in a known geometric shape; capturing a first image with the first headset camera and a second image with the second headset camera; identifying the set of references in the first image and the second image; and detecting a calibration level of the calibration profile or modifying the calibration profile, or both, based on the identified set of references in the first image and the second image and based on the known geometric shape.

[0054]

[0054] Example 49 is the method of Example 48, wherein the calibration profile includes a translation parameter corresponding to the relative distance between the first headset camera and the second headset camera.

[0055]

[0055] Example 50 is the method of Example 49, wherein the calibration profile further includes a rotation parameter corresponding to the relative angular orientation between the first headset camera and the second headset camera.

[0056]

[0056] Example 51 is the method of Example 50, where each of the translation parameter and the rotation parameter includes a single quantity, a one-dimensional matrix, a multi-dimensional matrix, an array, or a vector.

[0057]

[0057] Example 52 further includes determining a center point between a first headset camera and a second headset camera, where a first distance between the first headset camera and the center point and a second distance between the second headset camera and the center point are equal to the translation parameter, which is the method of Example 49.

[0058]

[0058] Example 53 is the method of Example 48, which includes determining a calibration level by generating an epipolar line based on a first image, projecting the epipolar line onto a second image using a calibration profile, and determining the calibration level based on a deviation of a set of references from the epipolar line in the second image.

[0059]

[0059] Example 54 is the method of Example 53, where the deviation corresponds to a calibration error between a first headset camera and a second headset camera, and the method further includes adjusting the first headset camera and / or the second headset camera based on the deviation.

[0060]

[0060] Example 55 is a wearable system configured to execute the method of any one of Examples 1 to 54.

[0061]

[0061] Example 56 is a non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to execute the operations of the method of any one of Examples 1 to 54.

Brief Description of the Drawings

[0062]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 12C

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 14A

Figure 14B

Figure 14C

Figure 15A

Figure 15B

Figure 15C

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 20A

Figure 20B

Figure 21A

Figure 21B

Figure 22A

Figure 22B

Figure 23A

Figure 23B

Figure 24

Figure 25

Figure 26A

Figure 26B

Figure 26C

Figure 27A

Figure 27B

Figure 28

Figure 29

Figure 30A

Figure 30B

Figure 31A

Figure 31B

Figure 31C

Figure 32A

Figure 32B

Figure 33

Figure 34

Figure 35A

Figure 35B

Figure 35C

Figure 35D

Figure 35E

Figure 36A

Figure 36B

Figure 36C

Figure 36D

Figure 36E

Figure 37A

Figure 37B

Figure 37C

Figure 37D

Figure 37E

Figure 37F

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

DETAILED DESCRIPTION OF THE INVENTION

[0063]

[0112] Some aspects of the present disclosure relate to the localization (e.g., position, orientation, and / or distance) of a handheld device, such as a controller, with respect to a wearable device, such as an augmented reality (AR), virtual reality (VR), or mixed reality (MR) headset. In some cases, 6 degrees of freedom (6DOF) pose tracking of the headset can be performed using one or more headset sensors, such as one or more headset inertial measurement units (IMUs) and one or more headset cameras, in a technique sometimes referred to as “inside-out tracking of the headset.” For each image captured by the headset camera, features can be identified within the image, the pixel positions of the identified features can be compared to the pixel positions of the same features in other images, and it becomes possible to calculate the 6DOF pose of the headset for each image.

[0064]

[0113] At the same time, 6DOF pose tracking of the controller can be performed using a combination of headset sensors and controller sensors or components, using one or both of two separate techniques. The first technique is referred to as “inside-out tracking of the controller” and is 6DOF pose tracking of the controller based on images of the real-world environment captured by a camera on the controller. For each captured image, features can be identified, the pixel positions of the identified features can be compared to the pixel positions of the same features in other images, and it becomes possible to calculate the 6DOF pose of the controller for each image. The second technique is referred to as “constellation tracking” and is 6DOF pose tracking of the controller based on captured images of a reference (e.g., a light-emitting diode (LED)) fixed to the controller by a camera on the headset. The reference can be programmed to flash (i.e., emit light) while the headset camera is exposed, such that each captured image includes the flashed reference, which can then be identified by an image processing routine. The pixel position of the identified reference within the image can be determined, and the identified reference may be associated with the known geometric shape of the reference, whereby the 6DOF pose of the controller can be determined.

[0065]

[0114] During operation of the wearable system, one or both of these controller pose tracking techniques may be used. For example, if controller inside-out tracking is not available (e.g., the image of the environment does not contain a sufficient number of features due to, for example, low illumination conditions), the wearable system may rely on constellation tracking. Conversely, if constellation tracking is not available (e.g., the reference of the controller is not within the field of view of the headset camera), the wearable system may rely on controller inside-out tracking. Further, if both tracking techniques are available, the tracking data generated by the two techniques may be fused with each other.

[0066]

[0115] In conventional VR or AR systems, 6DOF pose tracking of peripheral devices is achieved by incorporating a series of electromagnetic sensors and emitters strategically positioned on the user's AR headset, remote device, and / or other auxiliary devices (e.g., totems, haptic devices, gaming devices, etc.). Typically, an electromagnetic tracking system includes at least one electromagnetic field emitter and at least one electromagnetic field sensor. Since the emitted electromagnetic field has a known distribution, the detected electromagnetic field can be analyzed to determine the position and / or orientation of the peripheral device. Such a system provides a simple solution to the positioning problem, but additional solutions are needed to provide higher accuracy positioning. Embodiments of the present disclosure can replace or complement the electromagnetic tracking system.

[0067]

[0116] When employed in an AR system, the 6DOF pose tracking information of a handheld device can facilitate the operation of the AR system. For example, the AR system can generate virtual content that represents or interacts with a controller that feels comfortable to the user. For example, during a game where multiple users play with virtual balls and virtual bats, the AR system can generate virtual content for a virtual bat that is accurately positioned and oriented by the controller of the user holding the bat.

[0068]

[0117] Figure 1 shows an AR scene 100 as seen through a wearable AR device according to some embodiments of the present disclosure. AR scene 100 depicts a setting 106 such as a real-world park where a user of AR technology views various real-world objects 130 such as people, trees, buildings, and a real-world concrete platform 120 in the background. In addition to these items, the user of AR technology also "sees" various virtual objects 102 such as an image 102-2 of a robot standing on the real-world concrete platform 120 and a character 102-1 of a comic-like avatar flying nearby, even though these elements (character 102-1 and image 102-2) do not exist in the real world. Due to the extreme complexity of the human visual and nervous systems, it is difficult to generate VR or AR technology that facilitates the presentation of virtual image elements that are rich in a comfortable and natural feeling among other virtual or real-world image elements.

[0069]

[0118] Figure 2A shows an AR device 200A having a single fixed focal plane according to some embodiments of the present disclosure. During operation, the projector 214 of the AR device 200A may project virtual image light 222 (i.e., light associated with virtual content) onto the eyepiece lens 202-1, whereby a light irradiation field (i.e., an angular representation of the virtual content) may be projected onto the user's retina such that the user perceives the corresponding virtual content to be positioned at a location within the user's environment. For example, the virtual image light 222 outcoupled by the eyepiece lens 202-1 may cause the user to perceive that the character 102-1 is positioned on the first virtual depth plane 210-1 and that the image 102-2 is positioned on the second virtual depth plane 210-2. The user perceives the virtual content together with world light 232 corresponding to one or more world objects 230 such as the platform 120. In some embodiments, the AR device 200A includes a first lens assembly 205-1 positioned on the user side of the eyepiece lens 202-1 (the side closest to the user's eye of the eyepiece lens 202-1) and a second lens assembly 205-2 positioned on the world side of the eyepiece lens 202-1. Each of the lens assemblies 205-1, 205-2 may be configured to apply a refractive power to the passing light.

[0070]

[0119] Figure 2B shows an AR device 200B having two fixed focal planes according to some embodiments of the present disclosure. During operation, the projector 214 may project virtual image light 222 onto the first eyepiece lens 202-1 and the second eyepiece lens 202-2, whereby a light irradiation field may be projected onto the user's retina such that the user perceives the corresponding virtual content to be positioned at a location within the user's environment. For example, the virtual image light 222 outcoupled by the first eyepiece lens 202-1 may cause the user to perceive that the character 102-1 is positioned on the first virtual depth plane 210-1, and the virtual image light 222 outcoupled by the second eyepiece lens 202-2 may cause the user to perceive that the image 102-2 is positioned on the second virtual depth plane 210-2.

[0071]

[0120] Figure 3 shows a schematic diagram of an exemplary wearable system 300 according to some embodiments of the present disclosure. The wearable system 300 can include a wearable device 301 and at least one remote device 303 that is remote from the wearable device 301 (e.g., separate hardware but communicatively coupled). While the wearable device 301 is worn by a user (generally as a headset), the remote device 303 can be held by the user (e.g., as a handheld controller), or fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or removably attached to the user in other ways (e.g., backpack-type configuration, belt-coupled configuration, etc.) and can be worn in various configurations.

[0072]

[0121] The wearable device 301 can include a left eyepiece 302A and a left lens assembly 305A that are arranged in an adjacent configuration to form a left optical stack. The left lens assembly 305A can include a focusing lens on the user side of the left optical stack and a correction lens on the world side of the left optical stack. The wearable device 301 can include a right eyepiece 302B and a right lens assembly 305B that are arranged in an adjacent configuration to form a right optical stack. The right lens assembly 305B can include a focusing lens on the user side of the right optical stack and a correction lens on the world side of the right optical stack.

[0073]

[0122] In some embodiments, the wearable device 301 includes one or more sensors including, but not limited to, a left front-facing world camera 306A attached directly or near the left eyepiece 302A, a right front-facing world camera 306B attached directly or near the right eyepiece 302B, a left-side-facing world camera 306C attached directly or near the left eyepiece 302A, a right-side-facing world camera 306D attached directly or near the right eyepiece 302B, a left-eye tracking camera 326A directed at the left eye, a right-eye tracking camera 326B directed at the right eye, and a depth sensor 328 attached between the eyepieces 302. The wearable device 301 may include one or more image projection devices such as a left projector 314A optically linked to the left eyepiece 302A and a right projector 314B optically linked to the right eyepiece 302B.

[0074]

[0123] The wearable system 300 may include a processing module 350 for collecting, processing, and / or controlling data within the system. The components of the processing module 350 may be distributed between the wearable device 301 and the remote device 303. For example, the processing module 350 may include a local processing module 352 on the wearable portion of the wearable system 300 and a remote processing module 356 physically separated from the local processing module 352 and communicatively linked to the local processing module. Each of the local processing module 352 and the remote processing module 356 may include one or more processing units (e.g., a central processing device (CPU), a graphics processing unit (GPU), etc.) and one or more storage devices such as non-volatile memory (e.g., flash memory).

[0075]

[0124] The processing module 350 can collect data captured by various sensors of the wearable system 300, such as the camera 306, the gaze tracking camera 326, the depth sensor 328, the remote sensor 330, the ambient light sensor, the microphone, the inertial measurement unit (IMU), the accelerometer, the compass, the global navigation satellite system (GNSS) unit, the wireless device, and / or the gyroscope. For example, the processing module 350 can receive the image 320 from the camera 306. Specifically, the processing module 350 can receive the left front image 320A from the left front world camera 306A facing the left front, the right front image 320B from the right front world camera 306B facing the right front, the left side image 320C from the left side world camera 306C facing the left side, and the right side image 320D from the right side world camera 306D facing the right side. In some embodiments, the image 320 can include a single image, a pair of images, a video including a stream of images, and a video including a stream of pairs of images. The image 320 may be periodically generated and transmitted to the processing module 350 while the power of the wearable system 300 is on, or may be generated in response to commands transmitted by the processing module 350 to one or more of the cameras.

[0076]

[0125] The camera 306 may be configured at various positions and orientations along the outer surface of the wearable device 301 to capture an image of the user's surroundings. In some cases, the cameras 306A, 306B may be positioned to capture images that substantially overlap the FOVs of the user's left and right eyes, respectively. Thus, the positioning of the camera 306 may be near the user's eyes, but not so close as to obscure the user's FOV. Alternatively or additionally, the cameras 306A, 306B may be positioned to align with the input coupling positions of the virtual light 322A, 322B, respectively. The cameras 306C, 306D may be positioned to capture images of the user's side, for example, the user's peripheral vision or outside the user's peripheral vision. The images 320C, 320D captured using the cameras 306C, 306D do not necessarily have to overlap with the images 320A, 320B captured using the cameras 306A, 306B.

[0077]

[0126] In some embodiments, the processing module 350 may receive ambient light information from an ambient light sensor. The ambient light information may indicate a range of luminance values or spatially resolved luminance values. The depth sensor 328 may capture a depth image 332 in a direction facing the front of the wearable device 301. Each value of the depth image 332 may correspond to the distance between the depth sensor 328 and the closest detected object in a particular direction. As another example, the processing module 350 may receive gaze tracking data 334 from a gaze tracking camera 326 that may include left and right eye images. As another example, the processing module 350 may receive image luminance values projected from one or both of the projectors 314. The remote sensor 330 located within the remote device 303 may include any of the above-described sensors having similar functions.

[0078]

[0127] Virtual content is delivered to a user of the wearable system 300 using the projectors 314 and the eyepieces 302, along with other components within the optical stack. For example, the eyepieces 302A, 302B may each comprise a transparent or translucent waveguide configured to guide and out-couple light generated by the projectors 314A, 314B, respectively. Specifically, the processing module 350 may cause the left projector 314A to output left virtual image light 322A onto the left eyepiece 302A, and the right projector 314B to output right virtual image light 322B onto the right eyepiece 302B. In some embodiments, the projector 314 may include a microelectromechanical systems (MEMS) spatial light modulator (SLM) scanning device. In some embodiments, each of the eyepieces 302A, 302B may comprise a plurality of waveguides corresponding to different colors. In some embodiments, the lens assemblies 305A, 305B may be coupled and / or integrated with the eyepieces 302A, 302B. For example, the lens assemblies 305A, 305B may be incorporated into a multi-layer eyepiece, or may form one or more layers that constitute one of the eyepieces 302A, 302B.

[0079]

[0128] FIG. 4 shows an example of how a visual tracking system can be incorporated into an AR system having a wearable device 401 (e.g., a headset) and a handheld device 404 (e.g., a controller). In some embodiments, the handheld device 404 is a handheld controller that enables a user to provide input to the AR system. For example, the handheld device 404 can be a totem used in a gaming scenario. The handheld device 404 may be a haptic device and may include one or more haptic surfaces that utilize various sensor types. During operation of the AR system, the user may hold the handheld device 404 in their left or right hand by actively gripping the handheld device 404 and / or by firmly securing an attachment mechanism (e.g., a wraparound strap) to the user's hand.

[0080]

[0129] The handheld device 404 may include one or more fiducials 422 positioned along one or more outer surfaces of the handheld device 404 such that the fiducials can be within the field of view of an imaging device external to the handheld device 404. The fiducials 422 may have known geometric relationships to each other such that the imaging device can determine its position and / or orientation relative to the handheld device 404 by capturing an image of one or more of the fiducials 422. The fiducials 422 may be dynamic, static, powered, or unpowered, and in some embodiments, may be distinguishable from each other. For example, a first fiducial may be an LED having a first wavelength, and a second fiducial may be an LED having a second wavelength. Alternatively or additionally, different fiducials may have different brightnesses and / or may pulsate at different frequencies (e.g., a first fiducial may pulsate at 100 Hz and a second fiducial may pulsate at 150 Hz).

[0081]

[0130] The handheld device 404 may include one or more imaging devices (referred to herein as controller cameras 426) positioned such that when the handheld device 404 is held by a user, the wearable device 401 and / or some feature around the handheld device 404 is within the field of view of the imaging device. For example, a front controller camera 426A may be positioned such that its field of view faces away from the user and towards one or more features around the handheld device 404, and a rear controller camera 426B may be positioned such that its field of view faces towards the wearable device 401. The controller camera 426 may include one or more front-facing imaging devices and / or one or more rear-facing imaging devices to create a desired cumulative field of view. In some embodiments, the controller camera 426 may capture still or moving images.

[0082]

[0131] The handheld device 404 may include an IMU (referred to herein as the controller IMU 424), and the controller IMU 424 is rigidly and firmly fixed within the handheld device 404 such that the rotational and linear motions of the handheld device 404 are similarly experienced by the controller IMU 424. In some cases, the controller IMU 424 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, the controller IMU 424 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, the controller IMU 424 may output one or more of linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, the controller IMU 424 may output raw data from which any of the processed data in the above forms may be calculated.

[0083]

[0132] The handheld device 404 may include a rechargeable and / or replaceable battery 428 or other power source that powers the reference 422, the controller camera 426, the controller IMU 424, and any other components of the handheld device 404. Although not shown in FIG. 4, the handheld device 404 may include circuitry to enable wireless communication with the wearable device 401 and / or the remote device 440. For example, when using the controller camera 426 and the controller IMU 424 to detect or capture data, the handheld device 404 may transmit the raw data or the processed data to the wearable device 401 and / or the remote device 440.

[0084]

[0133] Wearable device 401 may include one or more imaging devices (referred to herein as headset cameras 410) positioned such that when handheld device 404 is held by a user, handheld device 404 including reference 422 is within the field of view of the imaging device. For example, one or more headset cameras 410 may be positioned on wearable device 401 above, below, and / or to the sides of the optical see-through component of wearable device 401 and facing forward. In one embodiment, two headset cameras 410 may be positioned on both sides of the optical see-through component of wearable device 401. In some embodiments, headset camera 410 may capture still or moving images.

[0085]

[0134] Wearable device 401 may include a headset IMU 408, which is rigidly and firmly fixed within wearable device 401 such that rotational and linear motion of wearable device 401 is similarly experienced by headset IMU 408. Optionally, headset IMU 408 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, headset IMU 408 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, headset IMU 408 may output one or more of linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, headset IMU 408 may output raw data from which any of the processed data in the above forms may be calculated.

[0086]

[0135] In some embodiments, the AR system may include a remote device 440 that includes a computing device (e.g., one or more processors and associated memory) for performing localization of the handheld device 404 relative to the wearable device 401. Alternatively or additionally, the computing device may be present in the wearable device 401 itself, or even in the handheld device 404. The computing device may receive raw or processed data (via wired and / or wireless connections) from each of the headset IMU 408, the headset camera 410, the controller IMU 424, and the controller camera 426, and may calculate the geospatial location of the handheld device 404 (relative to the geospatial location of the wearable device 401) and the orientation of the handheld device 404 (relative to the orientation of the wearable device 401). Further, the computing device may include a mapping database 442 (e.g., a navigable world model, a coordinate space, etc.) for detecting poses and determining coordinates of real and virtual objects, and in one or more embodiments, may also be connected to cloud resources and a navigable world model. In some embodiments, images captured using the headset camera 410 and / or the controller camera 426 may be used to construct a navigable world model. For example, features may be detected within the captured images, or otherwise collected data (e.g., sparse points) may be used to construct a navigable world model or an environmental map.

[0087]

[0136] FIG. 5 shows a diagram of a localization task executed by an AR system in which the position and orientation of the handheld device 504 are determined with respect to the wearable device 501. In the illustrated figure, the wearable device 501 has a geospatial position (the “wearable position”) defined as (XWP, YWP, ZWP) with respect to a world reference and an orientation (the “wearable orientation”) defined as (XWO, YWO, ZWO) with respect to the world reference. In some cases, the geospatial position of the wearable device 501 is represented by values of longitude, latitude, and altitude, and the orientation of the wearable device 501 is represented by values of pitch angle, yaw angle, and roll angle.

[0088]

[0137] As shown, the handheld device 504 has a geospatial position (the “handheld position”) defined as (X’HP, Y’HP, Z’HP) with respect to the geospatial position (XWP, YWP, ZWP) of the wearable device 501 and an orientation (the “handheld orientation”) defined as (X’HO, Y’HO, Z’HO) with respect to the orientation (XWO, YWO, ZWO) of the wearable device 501. In some cases, the geospatial position of the handheld device 504 is represented by X, Y, and Z Cartesian values, and the orientation of the handheld device 504 is represented by values of pitch angle, yaw angle, and roll angle. As one specific example, when the handheld device 504 is held by a user, the geospatial position of the handheld device 504 can be equal to (0.7 m, -0.5 m, 0.1 m), and the orientation of the handheld device 504 can be equal to (10.2°, -46.2°, 15.2°).

[0089]

[0138] FIG. 6 shows a perspective view of the controller 604 of the AR system. In some embodiments, the controller 604 includes fiducials 622 positioned along one or more outer surfaces of the controller 604 such that the fiducials can be within the field of view of an imaging device external to the controller 604. For example, the controller 604 can include nine fiducials positioned on the top surface 606 of the controller 604 and two fiducials positioned on both side surfaces of the controller 604. The two fiducials on the side surfaces of the controller 604 may be positioned proximate to the controller camera 626. In other examples, the controller 604 can include more or fewer fiducials 622. The fiducials 622 can have known relationships to each other such that the imaging device can determine the position and / or orientation of the controller 604 by capturing an image of one or more of the fiducials 622. As shown, the fiducials 622 on the top surface 606 are arranged in a circular pattern, but other configurations are possible.

[0090]

[0139] In some cases, 6DOF pose tracking of the headset (e.g., wearable devices 401, 501, 601) can be performed using images captured by the headset camera in combination with the headset IMU. This technique is referred to as headset inside-out tracking. For each captured image, features can be identified within the image, the pixel positions of the identified features can be compared to the pixel positions of the same features in other images, and it becomes possible to calculate the 6DOF pose of the headset for each image.

[0091]

[0140] In some cases, the 6DOF pose tracking of the controller 604, which may be an example of the handheld devices 404 and 504, can be performed using one or both of two distinct techniques. The first technique is (1) controller inside-out tracking, which is the 6DOF pose tracking of the controller 604 based on an image of the real-world environment captured by the controller camera 626 on the controller 604. Similar to headset inside-out tracking, for each captured image, features can be identified, the pixel positions of the identified features can be compared to the pixel positions of the same features in other images, and it becomes possible to calculate the 6DOF pose of the controller 604 for each image. The second technique is (2) constellation tracking, which is the 6DOF pose tracking of the controller 604 based on a captured image of a reference 622 (e.g., an LED) fixed to the controller 604 by a camera on the headset. The 6DOF pose of the controller 604 can be calculated from an image of the controller 604 captured by the headset camera. The image can be any single frame in which at least three of the references 622 are visible. During operation of the wearable system, one or both of these techniques can be used. For example, if controller inside-out tracking is not available (e.g., the image of the environment does not contain a sufficient number of features due to, e.g., low-light conditions), the wearable system can rely on constellation tracking. Conversely, if constellation tracking is not available (e.g., the references of the controller are not within the field of view of the headset camera), the wearable system can rely on controller inside-out tracking. Further, if both tracking techniques are available, the tracking data generated by the two techniques may be fused with each other.

[0092]

[0141] The images captured by the headset are used for both headset inside-out tracking and constellation tracking, so there may be a problem that the reference flashed during headset inside-out tracking appears as a distinguishable feature, resulting in the reference 622 being visible in the images used for headset tracking. To solve this, different images may be used for headset inside-out tracking (or simply "headset tracking") and constellation tracking, and the reference 622 may be controlled to flash while the images for constellation tracking are being captured. Further, to prevent image blurring, reduce power consumption, and enable easy identification of the reference 622, the flash "on" time can be shortened during constellation tracking. To reduce power consumption, the exposure interval of the camera is also shortened during constellation tracking. To ensure that the images for constellation tracking contain the flashed reference, the reference 622 may be controlled to flash multiple times surrounding the exposure interval of the camera. For example, the reference interval may be calculated to be centered with the exposure interval of the camera.

[0093]

[0142] The reference 622 may be dynamic, static, powered, or unpowered, and in some embodiments, may be distinguishable from each other. For example, the first reference may be an LED having a first wavelength, and the second reference may be an LED having a second wavelength. Alternatively or additionally, different references may have different brightnesses and / or may pulsate at different frequencies (e.g., the first reference may pulsate at 100 Hz, and the second reference may pulsate at 150 Hz). The reference 622 may normally flash at a first frequency, but may flash at a second frequency when inside-out tracking is lost. For example, the reference 622 may normally flash at 2 Hz, but may flash at 30 Hz when inside-out tracking is lost.

[0094]

[0143] When the controller 604 is held by a user, the controller 604 includes a controller camera 626 positioned such that some feature around the headset and / or the controller 604 is within the field of view of the controller camera 626. For example, the controller 604 may include a front controller camera positioned such that its field of view faces away from the user and toward one or more features around the controller 604, and a rear controller camera positioned such that its field of view faces toward the headset. The controller camera 626 may include one or more front-facing imaging devices and / or one or more rear-facing imaging devices to create a desired cumulative field of view. In some embodiments, the controller camera 626 may capture still or moving images.

[0095]

[0144] FIG. 7A shows an example of intervals of camera exposure and reference flash in the nominal mode. The nominal mode may be used for lighting conditions (e.g., about 1 ms or more) where the camera exposure of a world camera (e.g., one or more cameras of a headset) remains relatively large. The reference flash may be synchronized with the world camera exposure such that the reference flash occurs during the world camera exposure to ensure that the reference flash is captured by the world camera. As shown, the world camera exposure may be greater than 1 ms and the reference flash may be 0.01 ms, although other timings are possible. The reference flash may occur at a known offset relative to the world camera exposure. For example, the reference flash may occur near the start of the world camera exposure, at the center of the world camera exposure, or near the end of the world camera exposure (e.g., as shown in FIG. 7A). Thus, when the world camera exposure is turned on for a given interval, the reference can flash during the same given interval, such that the flash always occurs during the world camera exposure and the controller can be continuously tracked.

[0096]

[0145] FIG. 7B shows an example of intervals of camera exposure and reference flash in a high ambient light mode. The high ambient light mode can be used in lighting conditions where the camera exposure of the world camera is small (e.g., less than 1 ms) due to bright ambient light. Since it can be difficult to synchronize the reference pulse with a short camera exposure, instead of synchronizing the flash with the world camera exposure, the reference can flash at a rate whose period is equal to the world camera exposure time. As a result, the world camera is guaranteed to capture the reference flash. For example, if the world camera exposure is 0.5 ms, the reference can flash for 0.01 ms every 0.5 ms, ensuring that the flash occurs during the world camera exposure and that the reference flash is captured by the world camera. The reference flash is shown as occurring at the center of the world camera exposure, but alternatively, the reference flash may occur near the start or near the end of the world camera exposure.

[0097]

[0146] FIG. 8 shows an example of intervals of camera exposure and reference flash for headset tracking and constellation tracking. To track the headset and the controller, the wearable system can alternately repeat a headset tracking frame and a constellation tracking frame. Since each frame can be about 16 ms (16.6 ms at 60 Hz), the length of a pair of a headset tracking frame and a constellation tracking frame can be 33 ms. This process can start from a headset tracking frame in which the headset captures an image of the world to determine the pose of the headset. The world camera exposure of the headset camera can be at least 1 ms. As shown, the world camera exposure for headset tracking is shown as lasting over the 16 ms duration of the headset tracking frame. Since the headset camera does not localize the controller with respect to the headset during headset tracking, the reference for the controller does not flash during the headset tracking frame.

[0098]

[0147] After the headset tracking frame, a constellation tracking frame may occur, in which the headset captures a reference image of the controller to determine the posture of the controller relative to the headset. The world camera exposure of the headset camera can be less than 1 ms. As shown in the figure, the world camera exposure for headset tracking is shown as lasting only during a portion of the 16 ms duration of the constellation tracking frame. Since the world camera exposure is less than 1 ms, the reference can flash at a period equal to the duration of the world camera exposure to ensure that the headset camera captures a reference image that flashes during the constellation tracking frame. After the constellation tracking frame, the wearable system can return to headset tracking with another headset tracking frame. The alternation between the headset tracking frame and the constellation tracking frame can continue until the power of the wearable system is turned off.

[0099]

[0148] As shown in FIG. 8, for each of the constellation tracking frames, the wearable system may determine a reference interval 820 defined by a reference start time 822 and a reference end time 824, during which a set of references for the controller flashes a plurality of times at a reference frequency (having an associated reference period 826). The duration of each flash, called the flash pulse width, may be adjustable by the wearable system in some examples. The wearable system may determine the reference interval such that the world camera exposure interval during each constellation tracking frame at least partially overlaps the reference interval.

[0100]

[0149] Figure 9 shows an example of the synchronization between the camera exposure and the reference flash interval for bright illumination conditions. The reference between the headset camera and the controller can be synchronized via Bluetooth such that the reference flash occurs during the duration of the camera exposure. Therefore, the headset can send a signal to the controller indicating the time or interval when the world camera exposure is turned on so that the controller can flash the reference at the same time or interval. However, due to Bluetooth communication, along with other factors such as internal clock drift in the headset and the controller, there can be added uncertainty in the time synchronization. In some examples, the uncertainty can exceed 200 μs.

[0101]

[0150] Under normal and low illumination conditions (e.g., indoors), as shown by camera exposure 912A and flash 914A, the world camera exposure is significantly larger than the synchronization uncertainty and the reference pulse width. Therefore, the synchronization algorithm can use Bluetooth to synchronize camera exposure 912A and flash 914A by placing flash 914A at the center of camera exposure 912A. However, in a bright light environment (e.g., sunny outdoor environment), the values of the camera exposure and the synchronization error interval become close, and synchronization fails. The reference pulse can be increased until it equals the uncertainty interval to repair the failure, but at the expense of the reference appearing brighter on the image because the total integration time of the reference pulse increases. Instead, synchronization can be repaired while maintaining an equivalent reference luminance within the camera image by generating a reference flash train having the same pulse width and a period equal to the camera exposure. Therefore, the flash train width can equal the uncertainty window.

[0102]

[0151] Camera exposures 912B - 912D and their respective flashes 914B - 914D indicate reference flashes having a reference period (i.e., the period between consecutive reference flashes) equal to the duration of the world camera exposure (i.e., the "exposure duration"). The exposure duration of camera exposure 912C is shorter than that of camera exposure 912B, and the exposure duration of camera exposure 912D is shorter than that of camera exposure 912C. Flash 914B occurs near the start of camera exposure 912B, flash 914C occurs near the end of camera exposure 912C, and flash 914D is centered with camera exposure 912D. In all cases, the total integrated reference pulses can be substantially the same as in a normal luminance environment.

[0103]

[0152] Figure 10 shows an example of synchronizing a reference flash with a low - exposure interval 1030. The low - exposure interval 1030 is the time interval during which the world camera exposure of the headset camera is turned on for constellation tracking under bright illumination conditions. To synchronize the reference flash with the low - exposure interval 1030, the headset determines a low - exposure offset 1032. The low - exposure offset 1032 can be the time between the reference time and the start, end, or center of the low - exposure interval 1030. As shown in Figure 10, the first exemplary low - exposure offset 1032 is the time between the start of the world camera exposure within the headset tracking frame and the center of the low - exposure interval 1030 within the constellation tracking frame that occurs immediately after the headset tracking frame. The second exemplary low - exposure offset 1032 is the time between the start of the constellation tracking frame and the center of the low - exposure interval 1030 within the constellation tracking frame.

[0104]

[0153] The headset can send an indication of the low exposure offset 1032 and the exposure duration of the low exposure interval 1030 to the controller so that the controller can determine a reference interval 1020 in which a plurality of reference flashes occur. The reference interval 1020 may be centered with the low exposure interval 1030 to increase the likelihood that at least one reference flash overlaps with the low exposure interval 1030. As shown in FIG. 10, when five reference flashes occur in the constellation tracking frame, the reference interval 1020 can be set such that the center of the third reference flash is aligned with the center of the low exposure interval 1030 based on the low exposure offset 1032.

[0105]

[0154] FIG. 11 shows an exemplary method 1100 for synchronizing headset camera exposure and reference flashes according to some embodiments of the present invention. One or more steps of method 1100 may be executed in an order different from the illustrated embodiment, and one or more steps of method 1100 may be omitted during the execution of method 1100. Further, two or more steps of method 1100 may be executed simultaneously or in parallel with each other.

[0106]

[0155] In step 1102, cause the headset camera to capture a first exposure image having an exposure exceeding a threshold in the headset tracking frame. The first exposure image is captured in the headset tracking frame. The wearable system can perform headset tracking and controller tracking alternately. The headset camera of the headset repeatedly captures images in the headset tracking frame and the controller tracking frame. Since the threshold can be 1 ms, the first exposure image can be associated with an exposure exceeding 1 ms. The first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration.

[0107]

[0156] In step 1104, during the controller tracking frame, the headset camera is caused to capture a second exposure image having an exposure less than the threshold value. The second exposure image is captured during the controller tracking frame. Since the threshold value can be 1 ms, the second exposure image can be associated with an exposure less than 1 ms. The second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration. The second exposure duration is shorter than the first exposure duration.

[0108]

[0157] In step 1106, a reference interval during the controller tracking frame is determined. The reference interval is an interval in which a set of references flashes a plurality of times. The reference interval is defined by a reference start time and a reference end time while the set of references flashes at a reference frequency and a reference period. The reference interval is determined such that the second exposure interval at least partially overlaps with the reference interval.

[0109]

[0158] In step 1108, during the controller tracking frame, the set of references is caused to flash a plurality of times during the reference interval. The set of references flashes according to the reference frequency and the reference period. Accordingly, the set of references can be captured within the second exposure image during the controller tracking frame. In addition, the posture of the controller can be determined based on the set of references captured within the second exposure image.

[0110]

[0159] During constellation tracking, problems may occur if references flashing from multiple controllers are visible within the same headset image. For example, it may be difficult for the wearable system to determine which reference belongs to which controller, and thus constellation tracking may become invalid if not resolved. The two controllers may both be held by the user of the wearable system or by two different users of two different wearable systems.

[0111]

[0160] In some cases, the wearable system may perform some steps to perform one or more of the ambiguity removal techniques for several controllers. FIGS. 12 to 15 show exemplary techniques for removing ambiguity of a plurality of controllers. Optionally, a set of preliminary steps may be performed before performing any of these techniques. For example, the wearable system may use a headset camera to capture an image, identify a set of references in the image, and determine that at least one of the identified set of references belongs to the first controller and at least one of the identified set of references belongs to the second controller. The wearable system may determine that the set of references belongs to the first controller and the second controller based on the determined configuration of the references in the image, the number of references in the image, or other techniques.

[0112]

[0161] FIGS. 12A to 12C show ambiguity removal using independently controlled groups of references. In FIG. 12A, the controller 1204 is shown as having 11 references 1222, 9 of which are positioned on the upper surface of the controller 1204 and 2 of which are positioned on both side surfaces of the controller 1204. The references 1222 can be LEDs arranged in a plurality of groups that can be independently controlled. The groups can be asymmetric so that even if the pattern is translated, rotated, or inverted, one group does not look the same as another group. The groups play an important role in removing the ambiguity of a plurality of devices that can be recognized by a single wearable device.

[0113]

[0162] Figure 12B shows a group of reference for two controllers. For example, the first controller 1204A can be associated with the first group of reference 1222, and the second controller 1204B can be associated with the second group of reference 1222. The first group includes reference 1222A, reference 1222D, reference 1222F, and reference 1222H, while the second group includes reference 1222B, reference 1222C, reference 1222E, reference 1222G, and reference 1222I. Therefore, the first group is asymmetric with respect to the second group. In some examples, the two groups of reference may include at least one common reference, and in other examples, the two groups of reference may be mutually exclusive.

[0114]

[0163] For the disambiguation of the first controller 1204A and the second controller 1204B, as shown in Figure 12C, the wearable system can flash the first group of reference 1222 with the first controller 1204A and flash the second group of reference 1222 with the second controller 1204B. The headset can capture the flash images of the first group and the second group and detect which group of reference is depicted in the image. Even if both groups flash simultaneously (as shown in Figure 12C) and are captured in the image, the wearable system can differentiate between the first controller 1204A and the second controller 1204B based on the pattern of the group of reference. The wearable system can remember or access the mapping between the controllers 1204A - 1204B and the group of reference to determine which controller corresponds to a particular group of reference detected in the image. Each instance of the reference 1222 of the first controller 1204A and the second controller 1204B that flashes in Figure 12C corresponds to one image capture. That is, the world camera exposure 912 of the headset camera can occur during each flash.

[0115]

[0164] Figures 13A - 13D illustrate ambiguity removal using offset reference flashes. In Figure 13A, the first controller 1304A and the second controller 1304B are shown as each having a reference 1322 positioned on its top surface. Instead of separating the references 1322 into groups specific to each of the controllers 1304A - 1304B, the controllers 1304A - 1304B can be flashed at different periods, frequencies, or offsets relative to each other. For example, as shown in Figure 13B, the flash of the reference 1322A of the first controller 1304A can be offset with respect to the flash of the reference 1322B of the second controller 1304B such that pairs of references from different controllers do not flash simultaneously. In one example, the offset can be 2 ms, and the wearable system can flash the reference 1322B of the second controller 1304B at 0 ms, 4 ms, and 8 ms, and the wearable system can flash the reference 1322A of the first controller 1304A at 2 ms, 6 ms, and 10 ms. Each instance of the references 1322A - 1322B of the first controller 1304A and the second controller 1304B that are flashing in Figure 13B corresponds to one image capture. That is, the world camera exposure 912 of the headset camera can occur during each flash.

[0116]

[0165] FIG. 13C shows an example of correcting the period or frequency of the flash of reference 1322A of the first controller 1304A with respect to the flash of reference 1322B of the second controller 1304B. As an example, the wearable system may flash the reference 1322 of the second controller 1304B every 4 ms (e.g., at 0 ms, 4 ms, and 8 ms), but the wearable system may flash the reference 1322 of the first controller 1304A every 6 ms (e.g., at 0 ms, 6 ms, and 12 ms). In addition, FIG. 13D shows an example of correcting the offset and period of the flash of reference 1322A of the first controller 1304A with respect to the flash of reference 1322B of the second controller 1304B, which is in fact a combination of FIGS. 13B to 13C. In some cases, the period, frequency, or offset of the second controller 1304B may be corrected in addition to or instead of the correction example of the first controller 1304A. In any case, by knowing the pattern in which the controller's reference flashes and when the controller's image is captured, the wearable system can distinguish between multiple controllers that may be depicted in the image. Each instance of the references 1322A to 1322B of the first controller 1304A and the second controller 1304B that flash in FIGS. 13B to 13D corresponds to one image capture. That is, the world camera exposure 912 of the headset camera may occur during each flash.

[0117]

[0166] Figures 14A - 14C illustrate the ambiguity removal using the reference flash coding of the controller. In Figure 14A, the first controller 1404A and the second controller 1404B are shown as having a reference 1422 positioned on their respective upper surfaces. The wearable system can introduce random codes into the flash so that the reference 1422 of each of the controllers 1404A - 1404B can be uniquely identified. Each controller can have a unique coding. That is, the flash of the first controller 1404A can be coded with the first coding, and the flash of the second controller 1404B can be coded with the second coding. In some examples, the coding can indicate which flashes are omitted during the flash sequence, making the flash no longer periodic. The coding can be associated with a binary pattern (e.g., 110110110).

[0118]

[0167] The coding may be applied to only one of the controllers 1404A, as shown in Figure 14B for the first controller 1404A, and the coding has a binary pattern of 110110110. Thus, the flash of the reference 1422B of the second controller 1404B is periodic, but the flash of the reference 1422A of the first controller 1404A is not periodic. As shown, the first controller 1404A and the second controller 1404B flash simultaneously in two time periods, and then the second controller 1404B flashes during a third time period when the first controller 1404A does not flash.

[0119]

[0168] Figure 14C shows the introduction of coding to each of the controllers 1404A - 1404B. When each flash period corresponds to 1 ms, the first coding (having the binary pattern 110110110) of the first controller 1404A causes the reference 1422A to flash at 0 ms, 1 ms, 3 ms, 4 ms, 6 ms, and 7 ms. The second coding (having the binary pattern 101101011) of the second controller 1404B can cause the reference 1422B to flash at 0 ms, 2 ms, 3 ms, 5 ms, 7 ms, and 8 ms. Regardless of whether coding is introduced to one or both of the controllers 1404A - 1404B, by knowing the pattern at which the controller's reference flashes and the time at which the controller's image is captured, the wearable system can distinguish between multiple controllers that can be depicted in the image. In some examples, one or both of the first coding or the second coding can be a random code. Each instance of the references 1422A - 1422B of the first controller 1404A and the second controller 1404B that flash in FIGS. 14B - 14C corresponds to one image capture. That is, the world camera exposure 912 of the headset camera can occur during each flash.

[0120]

[0169] Figures 15A - 15C show ambiguity removal using independently controlled groups of references and flash coding. In FIG. 15A, the first controller 1504A and the second controller 1504B are shown as each having a reference 1522 positioned on the upper surface. The reference 1522 can be arranged in a plurality of groups that can be independently controlled, as described in FIG. 12B. The groups can be asymmetric such that even if the pattern is translated, rotated, or inverted, one group does not look the same as another group. The groups play an important role in ambiguity removal for a plurality of devices that can be recognized by a single wearable device.

[0121]

[0170] Figure 15B shows a group of reference of two controllers. For example, the first controller 1504A may be associated with the first group of reference 1522, and the second controller 1504B may be associated with the second group of reference 1522. The first group includes reference 1522A, reference 1522D, reference 1522F, and reference 1522H, while the second group includes reference 1522B, reference 1522C, reference 1522E, reference 1522G, and reference 1522I. Therefore, the first group is asymmetric with respect to the second group. In addition to the two groups, the reference 1522 of the controllers 1504A-1504B may also be coded with different flash codings as described in FIGS. 14B-14C and shown in FIG. 15C. Therefore, the combination of the group and different flash codings may enable the wearable system to distinguish the controllers 1504A-1504B. Each instance of the reference 1522 of the first controller 1504A and the second controller 1504B to be flashed in FIG. 15C corresponds to one image capture. That is, the world camera exposure 912 of the headset camera may occur during each flash.

[0122]

[0171] Figure 16 shows an exemplary method 1600 for removing reference flash ambiguity of a controller (e.g., controller 604) according to some embodiments of the present invention. One or more steps of method 1600 may be executed in an order different from the illustrated embodiments, and one or more steps of method 1600 may be omitted during the execution of method 1600. Further, two or more steps of method 1600 may be executed simultaneously or in parallel with each other.

[0123]

[0172] In step 1602, a set of images is captured using the headset camera of the headset. The images may show one or more controllers within the field of view of the headset camera.

[0124]

[0173] In step 1604, a reference within the set of repeatedly flashed images is identified. The reference can flash at a reference frequency and a reference period based on the exposure of the headset camera during a reference interval. The identified reference can belong to one or more controllers.

[0125]

[0174] In step 1606, it is determined that at least some of the references include a first set of references belonging to a first controller and a second set of references belonging to a second controller. Based on the number or positioning of the references, the wearable system can determine that some references belong to the first controller and some references belong to the second controller.

[0126]

[0175] In step 1608, it is determined that the flashing of the first set of references is at least partially temporally aligned with the flashing of the second set of references. The wearable system can determine that they are temporally aligned because the reference intervals of the first set of references are the same as those of the second set of references. Or, the wearable system can determine that both the first set of references and the second set of references appear within each image of the set of images and are thus temporally aligned. If the references are flashing at the same frequency and period, the wearable system may not be able to accurately track the postures of the first and second controllers.

[0127]

[0176] In step 1610, a modification is made to a period, frequency, or offset associated with at least one of the first set of criteria or the second set of criteria such that the flash of the first set of criteria is shifted from the flash of the second set of criteria. The wearable system may cause a first subset of the first set of criteria to be flashed by a first controller and a second subset of the second set of criteria to be flashed by a second controller. The first subset may be asymmetric with respect to the second subset. Additionally or alternatively, the flash of the first set of criteria and / or the second set of criteria may be coded by coding.

[0128]

[0177] FIG. 17 shows an example of a mobile device that may be used as a handheld device of a wearable system. The mobile device may be a mobile phone, a tablet, or other device having a display. Alternatively, in some examples, a sticker pattern positioned on a known concave surface may be suitable. Similar to the handheld device 404 of FIG. 4, the mobile device 1704 includes a camera and an IMU. Mobile phones already have a camera and an IMU. In some examples, software such as ARCore and ARKit can use the camera and IMU of the mobile device 1704 for SLAM tracking. The mobile device 1704 also includes a user interface 1706 that can display a reference 1722 according to a set of pixel positions of the display of the mobile device 1704 for 6DOF pose tracking of the mobile device 1704. The user interface 1706 can also display buttons 1724 that can receive user input according to other pixel positions, which provide all of the functions of a controller together with the camera, IMU, and reference 1722.

[0129]

[0178] By displaying the reference 1722 on the user interface 1706, each mobile device can be enabled to have a unique configuration of the reference. That is, since the reference 1722 is not restricted to be circular, the user interface 1706 can display the reference 1722 in a unique pattern that can be used to distinguish the mobile device 1704 from other controllers. For example, the reference 1722 may be displayed in a square shape on the user interface 1706, and another mobile device may display the reference in a star shape.

[0130]

[0179] For using the mobile device 1704 as a controller, the reference 1722 may be displayed as bright dots on a dark digital screen (e.g., the user interface 1706) or the background of a sticker. The headset of the wearable system can capture an image including the mobile device 1704 and process the image using an algorithm that detects the pattern of the reference 1722 to identify the mobile device 1704 as a controller. The reference 1722 may be a persistent display to limit the dynamics within the FOV to slow motion. The wearable system may flash the reference 1722 according to a period and a frequency. In some embodiments, the persistent display may be suitable for tasks such as writing on a virtual whiteboard.

[0131]

[0180] In some embodiments, the wearable system may cause the mobile device 1704 to modify a set of periods, frequencies, or pixel positions at which the reference 1722 is displayed. For example, the reference 1722 may be displayed over a shorter known period associated with the headset camera, whereby motion blur may be limited, enabling tracking at higher dynamics. Alternatively, if the wearable system determines that the image of the mobile device 1704 is also depicted by another controller, the wearable system may cause the mobile device 1704 to modify a set of periods, frequencies, or pixel positions to distinguish the reference 1722 of the mobile device 1704 from the references of other controllers. The wearable system may synchronize the display of the reference 1722 with the exposure interval of the headset camera so that the reference 1722 is visible within the image captured by the headset camera. The exposure interval may be determined as described with respect to FIGS. 8-11. The IMU of the mobile device 1704 may be further used to refine the determined position and orientation of the mobile device 1704, and may extend tracking slightly more than when all of the reference 1722 is visible within the camera FOV. The camera of the mobile device 1704 may be used to perform visual-inertial odometry (VIO) to provide tracking outside the FOV of the headset camera and to expand the dynamic range of tracking.

[0132]

[0181] FIG. 18 shows an exemplary method 1800 of using a device as a controller and displaying a reference on the display of the device, according to some embodiments of the present invention. One or more steps of method 1800 may be performed in an order different from that shown in the illustrated embodiment, and one or more steps of method 1800 may be omitted during the execution of method 1800. Further, two or more steps of method 1800 may be performed simultaneously or in parallel with each other.

[0133]

[0182] In step 1802, the controller causes a reference set to be displayed on the display. The reference set is displayed according to a set of pixel positions. The controller can also display on the display one or more buttons configured to receive user input according to another set of pixel positions.

[0134]

[0183] In step 1804, a set of images is captured using the headset camera of the headset. If the controller is within the field of view of the headset camera, the image can show the controller.

[0135]

[0184] In step 1806, the reference set is identified within the set of images. References to other controllers depicted within the set of images can also be identified.

[0136]

[0185] In step 1808, the position and / or orientation of the controller relative to the headset is determined based on the identified reference set. By knowing the position and orientation of the headset, the wearable system can determine the posture of the controller using the identified reference. Additionally, the wearable system may modify the set of periods, frequencies, or pixel positions to resolve ambiguities with references to another controller of the reference set.

[0137]

[0186] Figures 19A - 19B are exemplary internal perspective views of a controller of a wearable system. The optical 6DOF platform of controller 1904 may include a VIO sensor module 1912, a constellation module 1914, a main printed circuit board (PCB) 1916, a wireless communication engine 1918, a battery 1920, and a tactile engine 1922. Other components such as triggers, bumpers, touch pads, and input buttons may be additionally included. The VIO sensor module 1912 may include one or more cameras and an IMU on a rigid submount. The constellation module 1914 may include a reference array (e.g., LEDs). The main PCB 1916 includes via-in pads (VIPs) for the controller 1904 and LED drive electronics. The wireless communication engine 1918 includes components for communicating with a headset or other devices of the wearable system. The battery 1920 powers the controller 1904, and the tactile engine 1922 provides vibrations and other sensory outputs in response to inputs received by the controller 1904.

[0138]

[0187] In some cases, only the constellation module 1914, the main PCB 1916, and the input buttons may be required to provide 6DOF tracking. Other components may be optional depending on the application. Thus, the controller 1904 may be modular. The modules may be used to control and / or track external devices, an example of which is a drone. In another example (e.g., for pure tracking), one of these modules (or a complete controller) may be attached to a firearm for military training or law enforcement training, thereby enabling the direction in which a rifle is pointed to be tracked and / or "shown" to the user via a headset. Such embodiments may be useful in combat or training.

[0139]

[0188] Figures 20A - 20B show perspective views of exemplary modules for maximum reuse. Module 2000 includes a VIO sensor module 1912, a constellation module 1914, a main PCB 1916, a wireless communication engine 1918, and a battery 1920. Module 2000 may also include an input button and an LED user indicator. However, module 2000 may not have other user inputs such as a touchpad and a trigger. Additionally, module 2000 may not have a tactile engine. In one example, module 2000 may have an approximate size of 84 mm in length, 64 mm in width, and 18 mm in thickness, and may be powered and communicate via a Universal Serial Bus (USB).

[0140]

[0189] Figures 21A - 21B are perspective views of an exemplary hybrid module. Hybrid module 2100 includes a VIO sensor module 1912, a constellation module 1914, a main PCB 1916, and a wireless communication engine 1918. Hybrid module 2100 may also include an input button and an LED user indication. However, hybrid module 2100 may not have other user inputs such as a touchpad and a trigger. Additionally, hybrid module 2100 may not have a battery and a tactile engine. Therefore, hybrid module 2100 may be powered by a USB connection. In one example, the approximate size of hybrid module 2100 may be 64 mm in diameter and 18 mm in thickness.

[0141]

[0190] Figures 22A - 22B are perspective views of another exemplary module. Module 2200 includes a constellation module 1914, a main PCB 1916, a wireless communication engine 1918, and a battery 1920. Module 2200 may also include an input button and an LED user indication. However, module 2200 may not have other user inputs such as a touchpad and a trigger. Additionally, module 2200 may not have a VIO sensor module and a tactile engine. Thus, module 2200 can function only for constellation tracking when it is within the FOV of the headset camera, can be limited to tracking at 30 frames per second, and the battery life of module 2200 can be extended. In one example, the approximate size of module 2200 can be 64 mm in diameter and 18 mm in thickness.

[0142]

[0191] Other modules are also possible. For example, the smallest possible module can include a VIO sensor module, a constellation module, a main PCB, an input button, and a user indication. Thus, the smallest module may not have a wireless communication engine, a battery, other user inputs, and a tactile engine. As a result, the module can be powered and communicate via a Universal Serial Bus (USB). In one example, the approximate size of the module can be 50 mm in diameter and 15 mm in thickness.

[0143]

[0192] Due to the modular nature of the controller, as shown in FIGS. 23A - 23B, it may be possible to apply the controller to drone and unmanned aerial vehicle (UAV) applications. Modules such as the hybrid module 2100 of FIGS. 21A - 21B can be connected to the drone 2330. Alternatively, the hardware of the hybrid module 2100 may be integrated into the drone 2330 to enable a larger baseline between constellation criteria 2322. For example, as shown in FIG. 23B, the criteria 2322 can be spaced further apart from each other when they are incorporated into the drone 2330 than when the criteria 2322 are part of a hybrid module 2100 connected to the drone 2330 as shown in FIG. 23A.

[0144]

[0193] The drone 2330 can be controlled by an application on the wearable system 2300. Thus, when the user's device (e.g., headset 2302) views the criteria 2322 on the drone 2330, the drone 2330 is accurately located. Then, the drone 2330 can fly out of the user's line of sight and rely on VIO for navigation and to communicate its position and altitude to the user. When the drone 2330 returns to the line of sight, the drone is accurately located again and any errors in the path can be refined. A drone with a controller module can be useful for packaging and goods delivery applications, survey mapping applications, and remote inspection of construction sites or other hazardous areas. In some cases, the drone can be equipped with other sensors including GPS, radar, etc.

[0145]

[0194] FIG. 24 shows a headset 2401 of a wearable system that includes an image sensor 2426 capable of sensing a constellation of passive or active references included in a controller 2404 of the wearable system. The constellation can be a reference 2422. The headset 2401 may also include an algorithm for detecting the reference. However, the algorithm can be affected by a high rate of outlier detection. For example, the algorithm can detect other lights in proximity to the controller 2404 and may incorrectly associate those lights with the controller 2404 (as shown by the incorrect associations in FIG. 24). A good association is when the algorithm detects only the reference 2422 of the controller 2404 for tracking.

[0146]

[0195] To associate the references of the controller, the headset 2401 can be aimed at finding an association between an array of references and their corresponding detections, and the number of detections is usually greater than the number of references. This association problem can be solved by brute force by adopting a minimum pose estimator called a voting matrix and perspective-three-point (P3P) method and iteratively voting for correct reprojections after a correspondence trial. This problem is proportional to the factorial of the number of references and detections.

Number

[0147]

[0196] FIG. 25 shows a table indicating the iterations required to test all P3P combinations. The columns represent the number of detections and the rows represent the number of references. The scaling of this problem has a significant impact on the bandwidth of the algorithm that depends on this sensing modality.

[0148]

[0197] Figure 26A shows an exemplary set of fiducials arranged in multiple symmetry groups for fiducial association. Since the constellation is rotationally symmetric up to the gap, detection of the constellation sufficient to calculate the median distance between fiducials can be satisfied. Due to the rotational symmetry, Equation 1 can be simplified as follows.

Number

[0149]

[0198] Group 2634 may include three fiducials that can match any three detections. The wearable system can calculate the pose using the P3P algorithm. Next, the remaining fiducials can be projected using the pose and counted by comparing the inliers to a fixed reprojection tolerance. The group with the maximum amount of inlier association can be selected and the association made accurate up to the axis of rotational symmetry (e.g., the gap). The wearable system can project the pattern onto a plane and rotate the pattern around the axis of symmetry to find the best match. In other words, the gap within the constellation can be found. This process can function by having rotational symmetry that is not perfect at the gap, i.e., the gap is smaller or larger than the median of the adjacent marker distances.

[0150]

[0199] Figure 26B shows a wearable system including a headset 2601 having an imaging device 2626 that captures an image of a controller 2604 having several active (i.e., emitting) fiducials 2622. The image can be analyzed to identify the number of detections corresponding to a particular range of values of pixels that are thought to correspond to the active fiducials. In some cases, one or more of the detections can be caused by an interference light source 2610 that projects light onto the imaging device 2626. Such a light source 2610 can be from an internal or external illumination source and can be direct light or reflected light (e.g., light reflected from one or more surfaces before reaching the imaging device 2626). Thus, the image can include detections corresponding to the fiducials 2622 of the controller 2604 and detections caused by other light sources, and thus, the wearable system may need to determine which detections are associated with the fiducials 2622 of the controller 2604 so that the controller 2604 can be accurately tracked.

[0151]

[0200] Figure 26C shows a general association problem using the P3P algorithm. The algorithm can receive an image 2620 captured by the imaging device 2626 of the headset 2601 and a model of the constellation of the controller 2604 being tracked. The output of the algorithm can be the 6DOF pose of the controller 2604.

[0152]

[0201] Figure 27A shows a failure case of associating detections to determine the pose of the controller 2704. The wearable system receives an image 2720A of the field of view of a headset camera and identifies a set 2724A - 2724C of objects within the image 2720A. The set of objects 2724A - 2724C can include detection of projections of a reference and / or projections of other light sources. The wearable system selects a subset of the set of objects within the image 2720A. The subset can include a predetermined number (e.g., at least three) of objects. The wearable system can also select a subset of the set of objects from a model 2742A of the controller 2704 that includes the reference. Thus, the selected objects 2724A - 2724C within the image 2720A can be two - dimensional points, and the selected objects within the model 2742A can be three - dimensional points. In Figure 27A, the predetermined number of the subset is three.

[0153]

[0202] A subset of the objects can be input into a P3P algorithm that outputs the 6DOF pose of the controller 2704. The pose of the controller 2704 is calculated by associating the subset with a group of references that are rotationally symmetric to each other and each contain a predetermined number of references. The wearable system can calculate statistical values of the associated subsets based on the compatibility of the poses of the group of references. The statistical values can include, for example, errors associated with fitting the group of references to the subset of the objects. In some examples, the wearable system can project the pose onto the image 2720A and verify the alignment of the subset of the objects 2724A - 2724C with the remaining points in the image 2720A. The wearable system can determine the number of sets of objects 2724A - 2724C that are aligned with the set of pose references. FIG. 27A can be a failure case because only three objects are determined to match the pose calculated by the algorithm. Since the objects 2724A - 2724C are the objects initially selected within the image 2720A, they match the pose. However, since only the object 2724C corresponds to the reference 2722 of the controller 2704, the six references of the projected pose do not match. As a result, the wearable system can select different subsets of the set of objects, calculate a new pose, and improve the association between the pose and the references.

[0154]

[0203] FIG. 27B shows a successful case of associating detections to determine the pose of controller 2704. The wearable system receives an image 2720B of the field of view of the headset camera and identifies a set 2724D - 2724F of objects within image 2720B. The set of objects 2724D - 2724F may include detection of a reference projection and / or projections of other light sources. In FIG. 27B, objects 2724D - 2724F each correspond to a reference of controller 2704. The wearable system selects a subset of the set of objects 2724D - 2724F within image 2720B. The subset may include a predetermined number (e.g., at least three) of objects. The wearable system may also select a subset of the set of objects from a model 2742B of controller 2704 that includes the reference. Thus, the selected objects 2724D - 2724F within image 2720B may be two - dimensional points, and the selected objects within model 2742B may be three - dimensional points. In FIG. 27B, the predetermined number of the subset is three.

[0155]

[0204] A subset of the objects can be input into the P3P algorithm that outputs the 6DOF pose of the controller 2704. The pose of the controller 2704 is calculated by associating the subset with a group of references that are rotationally symmetric to each other and each contain a predetermined number of references. The wearable system can calculate the statistical value of the associated subset based on the compatibility of the poses of the group of references. To do this, the wearable system can project the pose into the image 2720B and verify the alignment of the subset of the objects with respect to the remaining points in the image 2720B. The wearable system can determine the number of sets of objects 2724D to 2724FC that are aligned with the set of pose references. FIG. 27B can be a successful case because all nine of the references of the controller 2704 are determined to match the pose calculated by the algorithm. Therefore, the wearable system can determine that the association between the set of objects and the set of references is correct based on the statistical value, and thus the calculated pose is accurate with respect to the controller 2704.

[0156]

[0205] FIG. 28 shows another example of associating the references of the controller with the objects in the image. The wearable system can create a circular reference pattern such that the distances between the triplets of the references 2822 are the same. This means that the triangles constructed by each triplet are the same up to rotation around the center of the circle. The gap between two of the adjacent references is larger than the gap between the other pairs of adjacent references. Triangles of nL / 3 can be constructed from the triplets of the references 2822.

[0157]

[0206] The wearable system attempts to associate the objects detected in the image with the references 2822 of the controller. To do this, the wearable system selects a subset of the objects depicted in the image that form a triplet. The subset of the objects is input into the P3P algorithm that calculates the pose as follows.

Number

[0158]

[0207] The posture can be rotated around the center of the triplet to determine how all the other possible triangles fit into the set of objects. This process can be repeated for multiple subsets of the objects and multiple postures. The posture that matches the most criteria can be determined to be the posture of the controller. Referring to FIG. 28, the first posture 2844A can be determined to match all but one of the criteria, the second posture 2844B can be determined not to match multiple criteria, and the third posture 2844C can be determined to match all the criteria. Boxes 2846A-2846B indicate the misalignment of the criteria in the first posture 2844A and the second posture 2844B, respectively. Since the third posture 2844C matches the most criteria, the wearable system can determine that it is the most accurate posture of the controller.

[0159]

[0208] FIG. 29 shows an exemplary method 2900 for criterion association using rotational symmetry according to some embodiments of the present invention. One or more steps of method 2900 may be performed in an order different from that shown in the illustrated embodiment, and one or more steps of method 2900 may be omitted during the execution of method 2900. Further, two or more steps of method 2900 may be performed simultaneously or in parallel with each other.

[0160]

[0209] In step 2902, a set of criteria of the controller is flushed. The criteria are arranged in a known geometric shape (e.g., a circle) that includes a group of multiple criteria that are rotationally symmetric with respect to each other. The number of each group of multiple criteria can be made equal to a predetermined number (e.g., at least three).

[0161]

[0210] In step 2904, the headset camera of the headset is caused to capture an image. The image can show the controller if the controller is within the field of view of the headset camera when the image is captured.

[0162]

[0211] In step 2906, a set of objects in the image corresponding to the reference is identified. The set of objects can correspond to a set of references of the controller and / or one or more light sources projected into the image.

[0163]

[0212] In step 2908, the set of objects is associated with the set of references based on a known geometric shape. A subset of the set of objects can be repeatedly selected, and by associating the subset with a group of multiple references, the posture of the controller can be calculated. Each of the subsets can have a number equal to a predetermined number. Statistical values can be calculated for the subsets associated based on the compatibility of the postures of the groups of multiple references, and the correct association between the set of objects and the set of references can be found based on the statistical values. A subset of the set of objects can be input into a P3P algorithm configured to output the posture of the controller.

[0164]

[0213] FIG. 30A shows a headset 3001 of a wearable system that includes an image sensor 3026 capable of sensing a constellation of passive or active references included in a controller 3004 of the wearable system. The constellation can be a reference 3022. The headset 3001 can execute an algorithm for detecting the reference. However, the algorithm can be affected by a high rate of detection of outliers. For example, the algorithm can detect other lights in the vicinity of the controller 3004 and incorrectly associate them with the controller 3004 (as shown by the incorrect association in FIG. 30A). A good association is the case where the algorithm detects only the reference 3022 of the controller 3004 for tracking.

[0165]

[0214] To associate the controller's references, the headset 3001 may aim to find the association between the array of references and their corresponding detections, and the number of detections is usually greater than the number of references. This association problem can be solved by brute force by adopting voting matrices and a minimum pose estimator called P3P and repeatedly voting for correct reprojections after a correspondence trial. This problem is proportional to the factorial of the number of references and detections, as shown by Equation 1.

[0166]

[0215] FIG. 30B shows a table indicating the iterations required to test all P2P combinations. If the controller 3004 includes an IMU that can determine the rotational measurements of the controller 3004 with respect to the headset 3001, a P2P algorithm may be used instead of the P3P algorithm. The P2P algorithm may be a standard P2P algorithm or a gravity-based P2P algorithm. The columns represent the number of detections and the rows represent the number of references. As shown by comparing the table of FIG. 30B with the table of FIG. 25, the number of iterations required to test all combinations can be significantly reduced by utilizing the IMU measurements made by the controller 3004.

[0167]

[0216] FIG. 31A shows an example of a projected reference of the controller 3104. Since the controller 3104 includes the IMU 3112B, the reference 3122 is projected in a known orientation. Thus, Equation 1 can be simplified as follows.

Equation

[0168]

[0217] Figure 31B shows a wearable system including a headset 3102 having an IMU 3112A and an imaging device 3126 that captures an image of a controller 3104 having several active (i.e., emitting) fiducials 3122. The image can be analyzed to identify the number of detections corresponding to a particular range of values of pixels that are thought to correspond to the active fiducials. In some cases, one or more of the detections can be caused by an interference light source 3110 that projects light onto the imaging device 3126. Such a light source 3110 can be from an internal or external illumination source and can be direct light or reflected light (e.g., light reflected from one or more surfaces before reaching the imaging device 3126). Thus, the image can include detections corresponding to the fiducials 3122 of the controller 3104 and detections caused by other light sources, and thus the wearable system may need to determine which detections are associated with the fiducials 3122 of the controller 3104 so that the controller 3104 can be accurately tracked. The headset 3101 can include an IMU 3112A and the controller 3104 can include an IMU 3112B, whereby the rotation of the controller 3104 relative to the headset 3101 can be determined.

[0169]

[0218] Figure 31C shows a typical association problem using a P2P algorithm. The algorithm can receive an image 3120 captured by the imaging device 3126 of the headset 3102 and a model of the constellation of the controller 3104 being tracked. The output of the algorithm can be the 6DOF pose of the controller 3104.

[0170]

[0219] Figure 32A shows a failure case of associating detections to determine the pose of the controller 3204. The wearable system receives an image 3220A of the field of view of the headset camera and identifies a set 3224A - 3224B of objects within the image 3220A. The set of objects 3224A - 3224B can include detection of reference projections and / or projections of other light sources. Object 3224A is the reference of the controller 3204, and object 3224B is the detection of another light source projection. The wearable system selects a subset of the set of objects within the image 3220A. The subset can include a predetermined number (e.g., at least two) of objects. The wearable system can also select a subset of the set of objects from a model 3242A of the controller 3204 that includes the reference. Thus, the selected objects within the image 3220A can be two - dimensional points, and the selected objects within the model 3242A can be three - dimensional points. In Figure 32A, the predetermined number of the subset is three.

[0171]

[0220] A subset of objects can be input into a P3P algorithm that outputs the 6DOF pose of the controller 3204, along with the rotational measurements (e.g., 3DOF orientation) of the controller 3204 determined from the IMU. The pose of the controller 3204 is calculated by associating the subset based on the known geometric shape of the reference arrangement. The wearable system can repeatedly select a subset of the set of objects and calculate the pose of the controller 3204 by associating the subset with a group of references. The wearable system calculates statistical values of the associated subsets based on the compatibility of the poses of the group of references and based on the rotational measurements. The statistical values can include, for example, errors related to fitting the group of references to the subset of objects. In some examples, the wearable system can project the pose into the image 3220A and verify the alignment of the subset of objects 3224A - 3224B with respect to the remaining points in the image 3220A. The wearable system can determine the number of sets of objects aligned with the set of pose references. FIG. 32A can be a failure case because only three references of the controller 3204 are determined to match the pose calculated by the algorithm. Objects 3224A - 3224C are determined to match the pose, while the other six objects are determined not to match. As a result, the wearable system can select different subsets of the set of objects, calculate a new pose, and improve the association between the pose and the references.

[0172]

[0221] FIG. 32B shows a successful case of associating detections to determine the pose of the controller 3204. The wearable system receives an image 3220B of the field of view of the headset camera and identifies a set 3224D - 3224E of objects within the image 3220B. The set of objects 3224D - 3224E can include the detection of the projection of the reference and / or the projection of other light sources. Each of the objects 3224D - 3224E corresponds to a reference of the controller 3204. The wearable system selects a subset of the set of objects within the image 3220B. The subset can include a predetermined number (e.g., at least two) of objects. The wearable system can also select a subset of the set of objects from a model 3242B of the controller 3204 that includes the reference. Thus, the selected objects within the image 3220B can be two - dimensional points, and the selected objects within the model 3242B can be three - dimensional points. In FIG. 32B, the predetermined number of the subset is three.

[0173]

[0222] A subset of the objects can be input into a P2P algorithm that outputs the 6DOF pose of the controller 3204, along with the rotational measurements (e.g., 3DOF directions) of the controller 3204 determined from the IMU. The pose of the controller 3204 is calculated by associating the subset with respect to a reference based on the known geometric shape of the reference arrangement. The wearable system can repeatedly select a subset of the set of objects and calculate the pose of the controller 3204 by associating the subset with a group of references. The wearable system calculates statistical values of the associated subsets based on the compatibility of the poses of the group of references and based on the rotational measurements. To do this, the wearable system can project the pose into the image 3220A and verify the alignment of the subset of objects with respect to the remaining points in the image 3220A. The wearable system can determine the number of sets of objects aligned with the set of pose references. FIG. 32B can be a success case because all nine of the references of the controller 3204 were determined to match the pose calculated by the algorithm. Therefore, the wearable system can determine that the association between the set of objects and the set of references is correct based on the statistical values, and thus the calculated pose is accurate for the controller 3204.

[0174]

[0223] FIG. 33 shows an exemplary method 3300 for reference association using IMU measurements according to some embodiments of the present invention. One or more steps of method 3300 may be performed in an order different from that shown in the illustrated embodiment, and one or more steps of method 3300 may be omitted during the execution of method 3300. Further, two or more steps of method 3300 may be performed simultaneously or in parallel with each other.

[0175]

[0224] In step 3302, flush the set of references of the controller. The set of references is arranged in a known geometric shape. The set of references can be flushed at a reference frequency and a reference period.

[0176]

[0225] In step 3304, the headset camera of the headset is caused to capture an image. The image can show the controller if the controller is within the field of view of the headset camera when the image is captured.

[0177]

[0226] In step 3306, a set of objects in the image corresponding to the reference is identified. The set of objects can correspond to a set of references of the controller and / or one or more light sources projected into the image.

[0178]

[0227] In step 3308, a rotational measurement value is captured using the controller inertial measurement unit of the controller. The rotational measurement value can correspond to the position and orientation of the controller relative to the headset.

[0179]

[0228] In step 3310, the set of objects is associated with the set of references based on a known geometric shape. A subset of the set of objects can be repeatedly selected, and by associating the subset with a plurality of groups of references, the posture of the controller can be calculated. Statistical values can be calculated for the associated subsets based on the compatibility of the postures of the plurality of groups of references, and the correct association between the set of objects and the set of references can be found based on the statistical values. A subset of the set of objects can be input into a P2P algorithm configured to output the posture of the controller.

[0180]

[0229] There are several advantages to reducing the image search area during constellation tracking. First, portions of the image that include the criteria of other controllers can be avoided, thereby eliminating the need to perform ambiguity removal methods for one or more of the plurality of controllers. Second, reducing the image search area can reduce the likelihood of false positives when identifying criteria (e.g., due to LEDs on other objects or “LED-like” features) that can be much more problematic for the 6DOF pose tracking algorithm than false negatives. Third, reducing the image search area shortens the search time and enables the 6DOF pose of the controller to be calculated more quickly.

[0181]

[0230] Figure 34 shows an example of a hand-tracking region of interest. In some embodiments, hand-tracking data is used to reduce the image search area by reducing the size of the image and / or the searched area of the image based on the pose of the hand. Alternatively or additionally, the criteria may still be identified across the entire image, and then the identified criteria outside the region of interest may be excluded. In one example, the camera 3426 of the headset 3401 may capture an image 3420 that includes the criteria 3422 of the controller 3404. The headset 3401 can detect a hand 3406 in the image and generate hand-tracking data regarding the hand 3406. The hand-tracking data may be used to identify the position and / or orientation of the user's hand 3406 within the image 3420. The region of interest 3408 within the image 3420 may be determined based on the position and / or orientation of the hand 3406. As an example, the region of interest 3408 may be a predetermined area around the hand 3406 (e.g., a box starting two inches above the hand). Next, the region of interest 3408 is searched and one or more criteria 3422 may be identified. The identified criteria 3422 may be associated with the criteria 3422 of the controller and lead to the calculation of the 6DOF pose of the controller 3404.

[0182]

[0231] Often, hand tracking has already been performed while the controller 3404 is in the user's hand 3406, and thus it is almost cost-free to utilize this hand tracking data during constellation tracking. Alternatively or additionally, in some embodiments, hand tracking can be made to execute in response to the generation of a command to shrink the image search area.

[0183]

[0232] Figures 35A - 35E show exemplary steps of performing hand tracking using the hand position to shrink the image search space. In Figure 35A, an image 3520 is captured by the headset camera. The image 3520 can depict the reference constellation of the user's hand 3506 and the controller 3504. Thus, the headset can generate hand tracking data that can be used to determine the position and / or orientation of the hand 3506 within the image. In Figure 35B, the hand tracking data is used to determine the position and / or orientation of the hand 3506. The position may be represented as the distance of the hand 3506 from the headset in each direction, and the orientation may be represented as the rotation of the hand 3506 in each direction with respect to the headset.

[0184]

[0233] In Figure 35C, the headset can determine a region of interest 3508 based on the position of the hand 3506. As an example, a reference point may be determined, and the region of interest 3508 may be a circular area of a specific radius around the reference point. For example, the reference point may be the point where the hand 3506 is determined to intersect the controller 3504. A pre-defined radius can be established around the reference point, and the area within the radius can be made the region of interest 3508. In one example, the radius can be 8 inches.

[0185]

[0234] In FIG. 35D, an object within the region of interest 3508 is identified. The object can be a light projection corresponding to the criteria of the controller 3504, or another light source projected within the image. For example, in the region of interest 3508, nine objects each corresponding to the criteria of the controller 3504 can be identified. In FIG. 35E, the wearable system determines the 6DOF pose of the controller 3504 based on the identified objects. The wearable system can perform a process similar to the process described in FIG. 29 that uses the P3P algorithm, or the process described in FIG. 33 that uses the P2P algorithm and the IMU data of the controller 3504, to determine the 6DOF pose of the controller 3504.

[0186]

[0235] FIGS. 36A-36E are diagrams showing exemplary steps for performing hand tracking using the position and orientation of a hand to reduce an image search space. In FIG. 36A, an image 3620 is captured by a headset camera. The image 3620 can depict the user's hand 3606 and the constellation of the criteria of the controller 3604. Thus, the headset can generate hand tracking data that can be used to determine the position and / or orientation of the hand 3606 within the image 3620. In FIG. 36B, the hand tracking data is used to determine the position and / or orientation of the hand 3606. The position may be represented as the distance of the hand 3606 from the headset in each direction, and the orientation may be represented as the rotation of the hand 3606 in each direction with respect to the headset.

[0187]

[0236] In FIG. 36C, the headset can determine the region of interest 3608 based on the position and orientation of the hand 3606. As an example, a reference point may be determined, and the region of interest 3608 may be a circular area with a specific radius around the reference point. For example, the reference point may be the point determined when the hand 3606 intersects the controller 3604. An ellipse with a predefined radius can be established around the reference point, and the area within the ellipse can be used as the region of interest 3608. The orientation of the hand 3606 can be used to tilt the region of interest 3608 in the direction in which the controller 3604 is held.

[0188]

[0237] In FIG. 36D, the objects within the region of interest 3608 are identified. The objects can be light projections corresponding to the reference of the controller 3604, or other light sources projected into the image. For example, in the region of interest 3608, nine objects each corresponding to the reference of the controller 3604 can be identified. In FIG. 36E, the wearable system determines the 6DOF pose of the controller 3604 based on the identified objects. The wearable system can execute a process similar to the process described in FIG. 29 using the P3P algorithm, or the process described in FIG. 33 using the P2P algorithm and the IMU data of the controller 3604 to determine the 6DOF pose of the controller 3604.

[0189]

[0238] Figures 37A - 37F illustrate exemplary steps for performing hand tracking to reduce the image search space. In Figure 37A, an image 3720 is captured by a headset camera. The image 3720 can depict the user's hand 3706 and at least one constellation of the reference of at least one controller. As shown, the image 3720 can depict a first constellation of the reference of controller 3704A and a second constellation of the reference of controller 3704B. The headset can generate hand tracking data that can be used to determine the position and / or orientation of the hand 3706 within the image 3720. In Figure 37B, an object is identified within the image 3720. The object can be a light projection corresponding to the reference of controller 3704A and / or another light source projected within the image 3720 (e.g., the reference of controller 3704B).

[0190]

[0239] In Figure 37C, the hand tracking data is used to determine the position and / or orientation of the hand 3706. The position may be represented as the distance of the hand 3706 from the headset in each direction, and the orientation may be represented as the rotation of the hand 3706 in each direction relative to the headset. In Figure 37D, the headset can determine a region of interest 3708 based on the position of the hand 3706. As an example, a reference point may be determined, and the region of interest 3708 may be a circular area of a specific radius around the reference point. For example, the reference point may be the point at which the hand 3706 is determined to intersect the controller 3704. A pre - defined radius can be established around the reference point, and the area within the radius can be made the region of interest 3708. In one example, the radius can be 8 inches.

[0191]

[0240] In FIG. 37E, the wearable system excludes the identified objects outside the region of interest 3708. Thus, since the reference of the controller 3704A is within the region of interest 3708 and the reference of the controller 3704B is outside the region of interest 3708, the reference of the controller 3704B can be excluded from the constellation pose tracking. In FIG. 37F, the wearable system determines the 6DOF pose of the controller 3704A based on the identified object. The wearable system can execute a process similar to the process described in FIG. 29 using the P3P algorithm, or the process described in FIG. 33 using the P2P algorithm and the IMU data of the controller 3704A to determine the 6DOF pose of the controller 3704A.

[0192]

[0241] FIG. 38 shows an exemplary method 3800 of using hand tracking to reduce the constellation search area according to some embodiments of the present invention. One or more steps of the method 3800 may be executed in an order different from that shown in the illustrated embodiment, and one or more steps of the method 3800 may be omitted during the execution of the method 3800. Further, two or more steps of the method 3800 may be executed simultaneously or in parallel with each other.

[0193]

[0242] In step 3802, a set of references of the controller is flashed. The set of references is arranged in a known geometric shape. The set of references can be flashed at a reference frequency and a reference period.

[0194]

[0243] In step 3804, the headset camera of the headset is caused to capture an image. The image can show the controller if the controller is within the field of view of the headset camera when the image is captured.

[0195]

[0244] In step 3806, a set of objects within the image corresponding to the reference is identified. The set of objects can correspond to a set of criteria of the controller, other sets of criteria of other controllers within the image, and / or one or more light sources projected within the image.

[0196]

[0245] In step 3808, hand-tracking data is used to identify the position of the hand within the image. The hand can be detected within the image, and the hand-tracking data can identify the position of the hand based on the detection. The hand-tracking data may also be used to identify the orientation of the hand within the image.

[0197]

[0246] In step 3810, the set of objects is associated with the set of criteria. In one example, the region of interest can be determined within the image based on the position and / or orientation of the hand within the image. A first subset of the set of objects outside the region of interest can be excluded, and a second subset of the set of objects inside the region of interest can be associated with the set of criteria. The first subset and the second subset can be mutually exclusive. In another example, the region of interest may be determined based on the position and / or orientation of the hand within the image, the set of objects within the region of interest within the image corresponding to the reference can be identified, and the set of objects within the region of interest can be associated with the set of criteria.

[0198]

[0247] A wearable device (e.g., a headset within any of the previous figures) can be calibrated by highly sophisticated equipment while in the factory, but during use, it may deform due to heat, use, and various forms of wear and tear and cracking, making the factory calibration inaccurate. One possible solution is for the user to repeatedly return the wearable device to the factory for recalibration. To avoid the obvious cost of such a solution, some embodiments enable accurate and robust runtime calibration while the wearable device is in use, eliminating the need for factory recalibration. In an embodiment, the current calibration level of the wearable device can be predicted, and calibration can be performed based on the predicted calibration level.

[0199]

[0248] During operation, the wearable device can correctly analyze the captured images by considering the differences in the spacing and orientation between front-facing cameras (e.g., the left front-facing world camera 306A and the right front-facing world camera 306B in FIG. 3) using one or more parameters from the calibration profile. The calibration profile may also be further used when generating virtual light to account for the differences in the spacing and orientation between the eyepieces so that the user can comfortably view the virtual elements in proper alignment. To achieve this, the wearable device may repeatedly access the calibration profile to ensure that the parameters being used reflect the most up-to-date and accurate parameters available. In some cases, the wearable device may read the parameters from the calibration profile immediately after the calibration process is executed.

[0200]

[0249] FIG. 39 shows an example of a calibration profile 3900 of a wearable device. In some embodiments, the calibration profile 3900 is maintained by the wearable device to model the physical spatial relationship between cameras 3906 facing the left and right fronts. The calibration profile 3900 may include a translation parameter T corresponding to the relative distance between the left-front-facing camera 3906A and the right-front-facing camera 3906B, and a rotation parameter R corresponding to the relative angular direction between the left-front-facing camera 3906A and the right-front-facing camera 3906B. Each of the translation parameter T and the rotation parameter R can take a wide range of data types. For example, the translation parameter T may be a single quantity (e.g., 0.1 meter), a one-dimensional matrix (e.g., [0.1;0;0] meters), a multi-dimensional matrix (e.g., [[0.1;0;0][0;0;0][0;0;0]] meters), an array, a vector, or any other possible representation of a single or multiple quantities. Similarly, the rotation parameter R may be a single quantity (e.g., 0.5 degrees), a one-dimensional matrix (e.g., [0.5;0;0] degrees), a multi-dimensional matrix (e.g., [[0.5;0;0][0;0;0][0;0;0]] degrees), an array, a vector, or any other possible representation of a single or multiple quantities.

[0201]

[0250] The calibration profile 3900 can be represented as each of the front-facing cameras 3906 occupying a single point using a pinhole camera model. The center point 3950 between the left front-facing camera 3906A and the right front-facing camera 3906B may be used to track the position of the wearable device within the environment relative to the origin of the world and may also be used as a baseline for translational and rotational adjustments. In some embodiments, the relative distance between the left front-facing camera 3906A and the right front-facing camera 3906B and the center point 3950 may be equal to the translational parameter T, where the translational parameter T represents a 3×1 matrix corresponding to a three-dimensional (3D) vector (e.g., [0.1 0.2 0.1] meters). In some embodiments, the relative angular orientation between the left front-facing camera 3906A and the right front-facing camera 3906B and the center point 3950 may be equal to the rotational parameter R, where the rotational parameter R represents a 3×3 matrix. Thus, the transformation between the right front-facing camera 3906B and the center point 3950 can be modeled by the transformation [T|R], and the transformation between the left front-facing camera 3906A and the center point 3950 can be modeled by the transformation [T|R]-1.

[0202]

[0251] The calibration level associated with the calibration profile 3900 may be determined periodically. Based on the calibration level, the wearable device may cause one of several types of calibration to occur. To determine the calibration level associated with the calibration profile 3900, a deformation amount D of the wearable device may be determined, where the deformation amount D is inversely proportional to the calibration level. FIG. 40 shows various examples of the deformation amount. In some cases, the calibration level is determined by identifying the same reference in two images 3920 captured by the left front-facing camera 3906A and the right front-facing camera 3906B. After determining that both images 3920 contain at least two of the same references, the epipolar line 4052 may be generated based on one image (e.g., the left image 3920A) and projected onto the other image (e.g., the right image 3920B). The epipolar line 4052 may be projected onto the right image 3920B using the most updated version of the calibration profile 3900. The deviation of the position of the reference from the epipolar line 4052 indicates the calibration error between the left front-facing camera 3906A and the right front-facing camera 3906B. Thus, the cameras can be recalibrated using the comparison between the reference and the epipolar line 4052.

[0203]

[0252] As shown, the wearable device may be determined to have low deformation when the detected reference projected from the left image 3920A onto the right image 3920B is substantially centered with the epipolar line 4052A (e.g., the deformation amount D is less than 20). The wearable device may be determined to have moderate deformation when the detected reference projected from the left image 3920A onto the right image 3920B is slightly displaced from the epipolar line 4052B (e.g., the deformation amount D is 20 to 80). When one of the detected references is substantially centered with the epipolar line 4052B and the other detected reference is not aligned with the epipolar line 4052B, the detected reference may be slightly displaced. The wearable device may be determined to have high deformation when the detected reference projected from the left image onto the right image is substantially displaced from the epipolar line 4052C (e.g., the deformation amount D is greater than 80). When both of the detected references are not aligned with the epipolar line 4052C, the detected reference may be significantly displaced.

[0204]

[0253] FIG. 41 shows the calibration of a headset 4102 or other wearable device using an image 4120 that includes a reference positioned on a controller 4101. The headset 4102 may include a stereo camera 4106 that includes a left front-facing camera 4106A and a right front-facing camera 4106B. The headset 4102 may be prone to deformation because the cameras 4106 are far apart and triangulation is susceptible to the effects of external camera calibration. Given the positions of features in the environment and their projected two-dimensional positions within the cameras, it may be possible to recover the extrinsic properties of the stereo system. The quality of the calibration may depend on the internal camera calibration, the external constellation calibration, and the constellation detection on the image. The quality of the calibration may be improved using several image pairs 4120 in the techniques described in FIGS. 39-40.

[0205]

[0254] In some examples, the controller 4101 may be calibrated with a headset that includes a wearable stereo camera. To restore the position of the controller using a reference constellation, it may be important to have a good reference calibration for the controller's rig. Given the 2D positions of the calibrated headset and the projected constellations on the headset camera, it may be possible to triangulate each reference and restore the relative positions between them. The quality of the calibration may depend on the internal calibration of the camera, the external calibration of the constellation, and the detection of the constellation on the image. The quality of the calibration may be improved using several image pairs in the techniques described in FIGS. 39-40.

[0206]

[0255] FIG. 42 shows an exemplary method 4200 for using a reference for headset camera calibration, according to some embodiments of the present invention. One or more steps of method 4200 may be performed in an order different from that shown in the illustrated embodiment, and one or more steps of method 4200 may be omitted during the execution of method 4200. Further, two or more steps of method 4200 may be performed simultaneously or in parallel with each other.

[0207]

[0256] In step 4202, a calibration profile is maintained that models the physical relationship between a first headset camera and a second headset camera. The first headset camera may be a left-facing front camera, and the second headset camera may be a right-facing front camera. The calibration profile may include a translation parameter corresponding to the relative distance between the first headset camera and the second headset camera, and a rotation parameter corresponding to the relative angular orientation between the first headset camera and the second headset camera.

[0208]

[0257] In step 4204, a set of references for the controller is flashed.

[0209]

[0258] In step 4206, cause the first headset camera to capture a first image and cause the second headset camera to capture a second image.

[0210]

[0259] In step 4208, a set of reference is identified within the first image and the second image. After determining that both images include at least two of the same reference, an epipolar line can be generated based on one image (e.g., the first image of the first image) and projected onto another image (e.g., the second image of the second image).

[0211]

[0260] In step 4210, a calibration level of the calibration profile is detected based on the set of identified reference within the first image and the second image and a known geometric shape. The calibration level can be determined based on the deviation of the set of reference from the epipolar line within the second image. A higher deviation can correspond to a higher amount of deformation and a lower calibration level.

[0212]

[0261] In step 4212, the calibration profile is corrected based on the set of identified reference within the first image and the second image and a known geometric shape. The calibration profile can be corrected such that the set of identified reference within the first image is aligned with the set of identified reference within the second image.

[0213]

[0262] FIG. 43 shows a simplified computer system 4300 according to one embodiment described herein. A computer system 4300 as shown in FIG. 43 may be incorporated into a device described herein. FIG. 43 provides a schematic diagram of one embodiment of a computer system 4300 that can execute some or all of the steps of a method provided by various embodiments. Note that FIG. 43 is only meant to provide a generalized illustration of various components, any or all of which may be appropriately utilized. Thus, FIG. 43 broadly shows how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0214]

[0263] The computer system 4300 is shown as including hardware elements that may be electrically coupled via bus 4305 or otherwise communicate as needed. The hardware elements may include, but are not limited to, one or more processors 4310, including one or more general-purpose processors and / or one or more dedicated processors such as digital signal processing chips and / or graphics acceleration processors, and one or more input devices 4315 that may include, but are not limited to, a mouse, keyboard, and / or camera, and one or more output devices 4320 that may include, but are not limited to, a display device and / or a printer.

[0215]

[0264] The computer system 4300 can further include, but is not limited to, one or more non-transitory storage devices 4325 that can include local and / or network-accessible storage and / or, but is not limited to, a disk drive, an array of drives, an optical storage device, and / or a solid state storage device such as random access memory (“RAM”) and / or read-only memory (“ROM”), and can communicate therewith. Such storage devices can be configured to implement any suitable data store including, but not limited to, various file systems and / or database structures, etc.

[0216]

[0265] The computer system 4300 can also include, but is not limited to, a communication subsystem 4319 that can include, but is not limited to, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset such as a Bluetooth (trademark) device, an 802.11 device, a WiFi device, a WiMax device, cellular communication equipment, etc. The communication subsystem 4319 can include, by way of example, one or more input and / or output communication interfaces for permitting data exchange with a network such as a network of other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device can communicate images and / or other information via the communication subsystem 4319. In other embodiments, a portable electronic device, such as a first electronic device, can be incorporated into the computer system 4300, such as an electronic device as an input device 4315. In some embodiments, the computer system 4300 will further include a working memory 4335 that can include a RAM or ROM device as described above.

[0217]

[0266] Computer system 4300 may also include software elements shown as currently located within working memory 4335, including other code such as operating system 4340, device drivers, executable libraries, and / or one or more application programs 4345. The software elements may include computer programs provided by various embodiments, and / or may implement methods provided by other embodiments, and / or may be designed to configure systems provided by other embodiments, as described herein. By way of example only, one or more of the procedures described with respect to the above methods may be implemented as code and / or instructions executable by a computer and / or a processor within a computer, in which case, in an aspect, such code and / or instructions may be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the described methods.

[0218]

[0267] These sets of instructions and / or code may be stored on a non-transitory computer-readable storage medium such as the aforementioned storage device 4325. In some cases, the storage medium may be incorporated into a computer system such as computer system 4300. In other embodiments, the storage medium may be separate from a computer system, such as a removable medium like a compact disk, so that the storage medium can be used to program, configure, and / or adapt a general-purpose computer using the instructions / code stored thereon. These instructions may take the form of executable code executable by computer system 4300, and / or may take the form of source and / or installable code that takes the form of executable code when compiled and / or installed on computer system 4300 using any of, for example, various commonly available compilers, installation programs, compression / decompression utilities, etc.

[0219]

[0268] It will be apparent to those skilled in the art that substantial modifications can be made in accordance with specific requirements. For example, customized hardware may be used and / or certain elements may be implemented in software, including portable software such as applets, hardware, or both. Further, connections to other computing devices, such as network input / output devices, may be employed.

[0220]

[0269] As described above, in one aspect, some embodiments may employ a computer system, such as computer system 4300, to execute methods according to various embodiments of the present technology. According to a set of embodiments, some or all of the steps of such methods may be executed by computer system 4300 in response to a processor 4310 executing one or more sequences of one or more instructions that may be incorporated into other code, such as an operating system 4340 and / or an application program 4345 included in working memory 4335. Such instructions may be read into working memory 4335 from another computer-readable medium, such as one or more of storage devices 4325. By way of example only, execution of a sequence of instructions included in working memory 4335 may cause processor 4310 to perform one or more steps of the methods described herein. Additionally or alternatively, some of the methods described herein may be executed via dedicated hardware.

[0221]

[0270] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a specific manner. In one embodiment implemented using computer system 4300, various computer-readable media may participate in providing instructions / code to processor 4310 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, the computer-readable medium is a physical and / or tangible storage medium. Such media may take the form of non-volatile media or volatile media. Non-volatile media includes, for example, optical and / or magnetic disks such as storage device 4325. Volatile media includes, but is not limited to, dynamic memory such as working memory 4335.

[0222]

[0271] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0223]

[0272] Various forms of computer-readable media may participate in carrying one or more sequences of one or more instructions to processor 4310 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer can load the instructions into its dynamic memory and transmit the instructions as a signal via a transmission medium that can be received and / or executed by computer system 4300.

[0224]

[0273] The communication subsystem 4319 and / or its components generally receive signals, and then the bus 4305 can convey the signals, and / or data, instructions, etc. carried by the signals, to the working memory 4335, from where the processor 4310 reads and executes the instructions. The instructions received by the working memory 4335 may optionally be stored in the non-transitory storage device 4325 either before or after execution by the processor 4310.

[0225]

[0274] The methods, systems, and devices described above are examples. In various configurations, various procedures or components may be omitted, replaced, or added as needed. For example, in alternative configurations, the method may be executed in an order different from the described order, and / or various steps may be added, omitted, and / or combined. Also, the features described with respect to a particular configuration may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology is evolving, and thus many of the elements are examples and do not limit the scope of the present disclosure or the claims.

[0226]

[0275] Specific details are given in the description to provide a complete understanding of the exemplary configurations including the implementation forms. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the configurations. This description provides only exemplary configurations and does not limit the claims, applicability, or configurations. Rather, the foregoing description of the configurations provides a possible explanation to those skilled in the art for implementing the described techniques. Various changes may be made to the functions and arrangements of the elements without departing from the spirit or scope of the present disclosure.

[0227]

[0276] Also, the architecture can be described as a process depicted as a schematic flowchart or block diagram. Each can describe the operations as sequential processes, but many of the operations can be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. The process may have additional steps not included in the figures. Further, examples of the method may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a non-transitory computer-readable medium such as a storage medium. The processor can execute the described tasks.

[0228]

[0277] Although some exemplary architectures have been described, various modifications, alternative architectures, and equivalents may be used without departing from the spirit of the present disclosure. For example, the above elements may be components of a larger system, where other rules may take precedence or the application of the present technology may be modified otherwise. Also, several steps may be performed before, during, or after considering the above elements. Accordingly, the above description does not limit the scope of the claims.

[0229]

[0278] As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to "a user" includes a plurality of such users, and a reference to "a processor" includes references to one or more processors and their equivalents known to those skilled in the art.

[0230]

[0279] Also, the terms "comprise", "comprising", "contains", "containing", "include", "including", and "includes", as used in this specification and the following claims, are intended to specify the presence of the stated features, integers, components, or steps, but do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0231]

[0280] Also, the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes may be suggested to those skilled in the art in light of them, and it is also understood that they should be within the spirit and scope of this application and the appended claims.

Claims

1. A method for operating a wearable system having a headset and a controller, wherein the method is The process involves alternating between performing headset tracking and controller tracking by repeatedly capturing images using the headset camera of the headset during headset tracking frames and controller tracking frames, respectively. Within each of the aforementioned headset tracking frames, The headset camera is to capture a first exposure image from among the images having exposure exceeding a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration. Within each of the controller tracking frames, The headset camera is to capture a second exposed image from among the images having an exposure below the threshold, wherein the second exposed image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, and the second exposure duration is shorter than the first exposure duration. Determining a reference interval defined by a reference start time and a reference end time, wherein during the reference interval, the set of references of the controller flashes multiple times at a reference frequency and a reference period, and the reference interval is determined such that the second exposure interval at least partially overlaps with the reference interval. The set of references is flashed multiple times during the reference interval according to the reference frequency and the reference period, Methods that include...

2. The aforementioned wearable system The headset camera and Equipped with a headset inertial measurement unit. The aforementioned headset, A set of the aforementioned references arranged in a known geometric shape, One or more controller cameras, and Equipped with a controller inertia measurement unit. The controller comprises, The wearable system is configured to determine the position or orientation of the headset or controller based on data captured by the headset camera, one or more controller cameras, the headset inertia measurement unit, or the controller inertia measurement unit. The method according to claim 1.

3. Operating the aforementioned wearable system Based on data captured by the headset camera or the headset inertial measurement unit of the headset, a first orientation of the headset relative to a reference frame is determined. To flash the set of references of the controller, The second orientation of the controller relative to the headset is as follows: The headset camera is used to capture images of the headset, Identifying the set of criteria in the headset image, Determining the second orientation of the controller relative to the headset based on the set of criteria identified in the headset image and the known geometric shape, The method according to claim 2, including the method described in claim 2.

4. The method according to claim 1, wherein the reference interval is determined such that the second exposure interval is centered with the reference interval.

5. The method according to claim 1, wherein the first time length of one headset tracking frame among the headset tracking frames is equal to the second time length of one controller tracking frame among the controller tracking frames.

6. The method according to claim 1, wherein the first exposure duration includes at least 1 millisecond.

7. The method according to claim 1, wherein the headset comprises a plurality of cameras arranged to provide overlapping fields of view.

8. The method according to claim 1, further comprising storing the first exposure image and the second exposure image for subsequent analysis.

9. The method according to claim 1, wherein the set of criteria includes a first criterion that emits light at a first wavelength and a second criterion that emits light at a second wavelength.

Images