Wearable system for controller positioning using head mounted device camera and

Abstract

The invention discloses a wearable system and a method of operating the same. The wearable system combines head-mounted device camera and controller positioning using head-mounted device camera and controller reference points. A wearable system may include a head-mounted device camera and a controller. The wearable system may repeatedly capture images using a head-mounted device camera of the head-mounted device during the head-mounted device tracking frame and the controller tracking frame, thereby alternately performing head-mounted device tracking and controller tracking. The wearable system may cause the head-mounted device camera to capture a first exposure image having an exposure above a threshold, and cause the head-mounted device camera to capture a second exposure image having an exposure below the threshold. The wearable system may determine a reference interval during which a reference point of the controller flickers at a reference frequency and a reference period. The wearable system may cause the fiducial to flicker within a fiducial interval according to a fiducial frequency and a fiducial period.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 345,159, filed on May 24, 2022, entitled “Wearable System with Head-Mounted Device Camera and Controller Fiducials for Controller Localization,” and to U.S. Provisional Patent Application No. 63 / 345,162, filed on May 24, 2022, entitled “Wearable System with Head-Mounted Device and Controller Inside-Out Tracking,” the entire disclosures of which are incorporated herein by reference for all purposes. Background Art

[0003] Modern computing and display technologies have facilitated the development of so-called "virtual reality" or "augmented reality" experience systems in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears real or may be perceived as real. Virtual reality or "VR" scenarios generally involve presenting digital or virtual image information without transparency to other actual real-world visual input; augmented reality or "AR" scenarios generally involve presenting digital or virtual image information as an enhancement to the user's visualization of the real world around him or her.

[0004] Despite these advances in display technology, there remains a need in the art for improved methods, systems, and devices related to augmented reality systems. Summary of the invention

[0005] The present disclosure generally relates to techniques for improving the performance and user experience of optical systems. More specifically, embodiments of the present disclosure provide methods for operating an augmented reality (AR), virtual reality (VR), or mixed reality (MR) wearable system, wherein a handheld device is employed to assist in the operation of the wearable system. Although portions of the present disclosure are described with reference to an AR system, the present disclosure is applicable to a variety of applications.

[0006] Various embodiments of the present invention are summarized below in the form of an example list. As used below, any reference to a series of examples should be understood as a separate reference to each of the examples (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3 or 4").

[0007] Example 1 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: during a head-mounted device tracking frame and a controller tracking frame, respectively, by repeatedly capturing images using a head-mounted device camera of the head-mounted device to alternately perform head-mounted device tracking and controller tracking; during each of the head-mounted device tracking frames: causing the head-mounted device camera to capture a first exposure image having an exposure higher than a threshold in the image, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration; during each of the controller tracking frames: causing the head-mounted device camera to capture a second exposure image in the image having an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration; determining a benchmark interval defined by a benchmark start time and a benchmark end time, during which a benchmark point set of the controller will flash a plurality of times at a benchmark frequency and a benchmark period, wherein the benchmark interval is determined so that the second exposure interval at least partially overlaps with the benchmark interval; and causing the benchmark point set to flash a plurality of times during the benchmark interval according to the benchmark frequency and the benchmark period.

[0008] Example 2 is a method according to Example 1, wherein the wearable system includes: the head-mounted device, including: the head-mounted device camera; and the head-mounted device inertial measurement unit; and the controller, including: the set of reference points arranged in a known geometric shape; one or more controller cameras; and the controller inertial measurement unit; wherein the wearable system is configured to: determine the position or orientation of the head-mounted device or the controller based on data captured by the head-mounted device camera, the one or more controller cameras, the head-mounted device inertial measurement unit, or the controller inertial measurement unit.

[0009] Example 3 is a method according to Example 2, wherein operating the wearable system includes: determining a first posture of the head-mounted device relative to a reference frame based on data captured by the head-mounted device camera of the head-mounted device or the head-mounted device inertial measurement unit of the head-mounted device; causing the reference point set of the controller to flash; determining a second posture of the controller relative to the head-mounted device by: capturing a head-mounted device image using the head-mounted device camera; identifying the reference point set in the head-mounted device image; and determining the second posture of the controller relative to the head-mounted device based on the reference point set identified in the head-mounted device image and the known geometric shape.

[0010] Example 4 is a method according to Example 1, wherein the reference interval is determined so that the second exposure interval is centered on the reference interval.

[0011] Example 5 is a method according to Example 1, wherein a first time length of a head mounted device tracking frame in the head mounted device tracking frame is equal to a second time length of a controller tracking frame in the controller tracking frame.

[0012] Example 6 is a method according to Example 1, wherein the first exposure duration comprises at least one millisecond.

[0013] Example 7 is a method of operating a wearable system including a head-mounted device, the method comprising: capturing a set of images using a head-mounted device camera of the head-mounted device; identifying multiple reference points that repeatedly flash in the image set; determining that at least some of the multiple reference points include a first set of reference points belonging to a first controller and a second set of reference points belonging to a second controller different from the first controller; determining that the flashing of the first set of reference points is at least partially aligned in time with the flashing of the second set of reference points; and modifying a period, frequency, or offset associated with at least one of the first set of reference points or the second set of reference points so that the flashing of the first set of reference points is not aligned with the flashing of the second set of reference points.

[0014] Example 8 is a method according to Example 7, wherein causing the modification includes: causing the first controller to modify a first period, a first frequency, or a first offset associated with the first set of reference points so that the flickering of the first set of reference points is not aligned with the flickering of the second set of reference points.

[0015] Example 9 is a method according to Example 8, wherein causing the modification further includes: causing the second controller to modify a second period, a second frequency, or a second offset associated with the second set of reference points so that the flickering of the first set of reference points is not aligned with the flickering of the second set of reference points.

[0016] Example 10 is a method according to Example 7, wherein causing the modifying comprises causing the first controller to modify flashing of the first set of reference points to be encoded using a first encoding.

[0017] Example 11 is a method according to Example 10, wherein causing the modifying further comprises causing the second controller to modify the flashing of the second set of reference points to be encoded using a second encoding.

[0018] Example 12 is a method according to Example 7, further comprising: performing controller tracking on the first controller in the following manner: causing the head-mounted device camera to capture an image in the image set with an exposure below a threshold, wherein the image is associated with an exposure interval defined by an exposure start time, an exposure end time, and an exposure duration; determining a baseline interval defined by a baseline start time and a baseline end time, during which the first baseline point set of the first controller will flash multiple times at a baseline frequency and a baseline period, wherein the baseline interval is determined so that the exposure interval at least partially overlaps with the baseline interval; and causing the first baseline point set to flash multiple times during the baseline interval according to the baseline frequency and the baseline period.

[0019] Example 13 is a method according to Example 7, wherein the first controller includes the first set of reference points arranged in a known geometric shape, and the second controller includes the second set of reference points arranged in the known geometric shape, and wherein the method further includes: causing a first subset of the first set of reference points to flash at the first controller; and causing a second subset of the second set of reference points to flash at the second controller, wherein the first subset and the second subset are asymmetrical to each other.

[0020] Example 14 is a method of operating a wearable system, the wearable system comprising a head-mounted device and a controller having a display, the method comprising: causing the controller to display a set of reference points on the display based on a set of pixel positions; capturing a set of images using a head-mounted device camera of the head-mounted device; identifying the set of reference points in the set of images; and determining a position and / or orientation of the controller relative to the head-mounted device based on the set of reference points identified in the set of images.

[0021] Example 15 is a method according to Example 14, further comprising: causing the set of reference points to flash according to a period and a frequency.

[0022] Example 16 is a method according to Example 15, further comprising: causing the controller to modify the period, the frequency, or the set of pixel positions.

[0023] Example 17 is a method according to Example 14, further comprising: identifying a second set of reference points belonging to a second controller in the image set; and causing the controller to modify the period, the frequency, or the set of pixel positions in response to identifying the second set of reference points.

[0024] Example 18 is a method according to Example 17, further comprising: identifying a first geometric shape of the second set of reference points; and causing the controller to display the set of reference points in a second geometric shape different from the first geometric shape.

[0025] Example 19 is a method according to Example 14, further comprising: synchronizing the display of the set of reference points with at least one exposure interval of the head-mounted device camera.

[0026] Example 20 is a method according to Example 19, wherein the at least one exposure interval includes a first exposure interval during a head-mounted device tracking frame and a second exposure interval during a controller tracking frame, wherein the head-mounted device tracking frame includes: causing the head-mounted device camera to capture a first exposure image in the image set with an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration; and wherein the controller tracking frame includes: causing the head-mounted device camera to capture a second exposure image in the image set with an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration; and determining a baseline interval defined by a baseline start time and a baseline end time, during which the baseline point set will flash multiple times at a baseline frequency and a baseline period, wherein the baseline interval is determined so that the second exposure interval at least partially overlaps with the baseline interval.

[0027] Example 21 is a method according to Example 14, further comprising causing the controller to display one or more buttons configured to receive user input on the display according to the second set of pixel positions.

[0028] Example 22 is a method according to Example 14, wherein the controller comprises a mobile device.

[0029] Example 23 is a modular controller for a wearable system, the modular controller comprising one or more of a set of components, the set of components comprising: a visual inertial odometry (VIO) module; a constellation module; a main printed circuit board (PCB); a battery; a wireless communication engine; a user input comprising at least one of the following: (i) a trigger, (ii) a bumper, or (iii) a touchpad; a haptic engine; and / or a user indicator; wherein one or more of the set of components can be independently removed or added while maintaining at least some functionality of the modular controller.

[0030] Example 24 is a modular controller according to Example 23, wherein the modular controller is powered and communicates via a universal serial bus (USB) connection.

[0031] Example 25 is a modular controller according to Example 23, wherein the minimum dimensions of the modular controller are 84 mm in length, 64 mm in width, and 18 mm in thickness.

[0032] Example 26 is a modular controller according to Example 23, wherein the minimum dimensions of the modular controller are 64 mm in diameter and 18 mm in thickness.

[0033] Example 27 is a modular controller according to Example 23, wherein the minimum dimensions of the modular controller are 50 mm in diameter and 15 mm in thickness.

[0034] Example 28 is a modular controller according to Example 23, wherein the modular controller is integrated into a drone that can be controlled by an application on the wearable system, and wherein the wearable system is configured to identify a set of reference points of the constellation module to locate the drone.

[0035] Example 29 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: causing a set of reference points of the controller to flash, the set of reference points being arranged in a known geometric shape and comprising multiple groups of reference points that are rotationally symmetric relative to each other, wherein the number of each of the multiple groups of reference points is equal to a predetermined number, wherein the predetermined number is at least three; causing a head-mounted device camera of the head-mounted device to capture an image; identifying a set of objects in the image corresponding to the reference points; associating the set of objects with the set of reference points based on the known geometric shape in the following manner: repeatedly selecting subsets of the set of objects, wherein the number of each subset is equal to the predetermined number, and calculating the posture of the controller by associating the subset with the multiple groups of reference points; calculating statistics of the associated subsets based on the compatibility of the postures of the multiple groups of reference points; and finding the correct association between the set of objects and the set of reference points based on the calculated statistics.

[0036] Example 30 is a method according to Example 29, wherein calculating the pose comprises: inputting the subset of the set of objects into a perspective three-point algorithm configured to output the pose of the controller.

[0037] Example 31 is a method according to Example 29, wherein the known geometric shape includes a first gap between a first pair of adjacent reference points in the set of reference points, and the first gap is larger than other gaps between other pairs of adjacent reference points in the set of reference points.

[0038] Example 32 is a method according to Example 29, wherein the set of objects corresponds to the set of reference points and / or one or more light sources projected in the image.

[0039] Example 33 is a method according to Example 29, wherein calculating the statistics of the associated subset includes: determining, for each of the postures, a number of the set of objects aligned with the set of reference points.

[0040] Example 34 is a method according to Example 29, further comprising: identifying a second set of reference points in the image belonging to a second controller; and in response to identifying the second set of reference points, causing the controller to modify the period, frequency or offset of the flashing of the set of reference points so that the set of reference points is not aligned with the second set of reference points.

[0041] Example 35 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: flashing a set of reference points of the controller, the set of reference points being arranged according to a known geometric shape; causing a head-mounted device camera of the head-mounted device to capture an image; identifying a set of objects corresponding to the reference points in the image; capturing rotation measurements using a controller inertial measurement unit of the controller; and associating the set of objects with the set of reference points based on the known geometric shape by: repeatedly selecting a subset of the set of objects, and calculating a posture of the controller by associating the subset with multiple sets of reference points; calculating statistics of the associated subsets based on the compatibility of the postures of the multiple sets of reference points and based on the rotation measurements; and finding a correct association between the set of objects and the set of reference points based on the calculated statistics.

[0042] Example 36 is a method according to Example 35, wherein calculating the pose includes: inputting the subset of the set of objects into a perspective two-point algorithm configured to output the pose of the controller.

[0043] Example 37 is a method according to Example 36, wherein the perspective two-point algorithm includes a gravity-based perspective two-point algorithm.

[0044] Example 38 is a method according to Example 35, wherein the set of objects corresponds to the set of reference points and / or one or more light sources projected in the image.

[0045] Example 39 is a method according to Example 35, wherein calculating the statistics of the associated subset includes: determining, for each of the postures, the number of the set of objects aligned with the set of reference points.

[0046] Example 40 is a method according to Example 35, further comprising: identifying a second set of reference points belonging to a second controller in the image; and in response to identifying the second set of reference points, causing the controller to modify the period, frequency or offset of the flashing of the set of reference points so that the set of reference points is not aligned with the second set of reference points.

[0047] Example 41 is a method according to Example 35, wherein the wearable system includes: a head-mounted device, which includes: the head-mounted device camera; and a head-mounted device inertial measurement unit; and a controller, which includes: the set of reference points arranged in the known geometric shape; one or more controller cameras; and the controller inertial measurement unit; wherein the wearable system is configured to determine the position or orientation of the head-mounted device or the controller based on data captured by the head-mounted device camera, the one or more controller cameras, the head-mounted device inertial measurement unit, or the controller inertial measurement unit.

[0048] Example 42 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: flashing a set of reference points of the controller, the set of reference points being arranged in a known geometric shape; causing a head-mounted device camera of the head-mounted device to capture an image; identifying a set of objects corresponding to the reference points in the image; using hand tracking data to identify a position of a hand in the image; and associating the set of objects with the set of reference points by: determining a region of interest in the image based on the position of the hand in the image; excluding a first subset of the set of objects that are outside the region of interest; and associating a second subset of the set of objects that are within the region of interest with the set of reference points, wherein the first subset and the second subset are mutually exclusive.

[0049] Example 43 is a method according to Example 42, further comprising: using the hand tracking data to identify the orientation of the hand in the image; and determining the region of interest based on the orientation of the hand in the image.

[0050] Example 44 is a method according to Example 43, wherein determining the region of interest based on the orientation of the hand in the image includes: tilting the region of interest in the direction in which the controller is held according to the orientation.

[0051] Example 45 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: flashing a set of reference points of the controller, the set of reference points being arranged in a known geometric shape; causing a head-mounted device camera of the head-mounted device to capture an image; using hand tracking data to identify a position of a hand in the image; determining a region of interest in the image based on the position of the hand in the image; identifying a set of objects corresponding to the reference points in the region of interest in the image; and associating the set of objects in the region of interest with the set of reference points.

[0052] Example 46 is a method according to Example 45, further comprising: using the hand tracking data to identify the orientation of the hand in the image; and determining the region of interest based on the orientation of the hand in the image.

[0053] Example 47 is a method according to Example 45, wherein determining the region of interest based on the orientation of the hand in the image includes: tilting the region of interest in the direction of holding the controller according to the orientation.

[0054] Example 48 is a method of operating a wearable system having a head-mounted device and a controller, the method comprising: maintaining a calibration profile, the calibration profile modeling a physical relationship between a first head-mounted device camera of the head-mounted device and a second head-mounted device camera of the head-mounted device; flashing a set of reference points of the controller, the set of reference points being arranged in a known geometric shape; causing the first head-mounted device camera to capture a first image and causing the second head-mounted device camera to capture a second image; identifying the set of reference points in the first image and the second image; and performing one or both of the following: detecting a calibration level of the calibration profile based on the set of reference points identified in the first image and the second image and based on the known geometric shape; or modifying the calibration profile based on the set of reference points identified in the first image and the second image and based on the known geometric shape.

[0055] Example 49 is a method according to Example 48, wherein the calibration profile includes a translation parameter corresponding to a relative distance between the first head-mounted device camera and the second head-mounted device camera.

[0056] Example 50 is a method according to Example 49, wherein the calibration profile also includes a rotation parameter corresponding to the relative angular orientation between the first head-mounted device camera and the second head-mounted device camera.

[0057] Example 51 is a method according to Example 50, wherein each of the translation parameters and the rotation parameters comprises: a single quantity, a one-dimensional matrix, a multi-dimensional matrix, an array, or a vector.

[0058] Example 52 is a method according to Example 49, further comprising: determining a center point between the first head-mounted device camera and the second head-mounted device camera, wherein a first distance between the first head-mounted device camera and the center point and a second distance between the second head-mounted device camera and the center point are equal to the translation parameter.

[0059] Example 53 is a method according to Example 48, wherein determining the calibration level includes: generating an epipolar line based on the first image; projecting the epipolar line onto the second image using the calibration profile; and determining the calibration level based on a deviation between the set of reference points in the second image and the epipolar line.

[0060] Example 54 is a method according to Example 53, wherein the deviation corresponds to a calibration error between the first head-mounted device camera and the second head-mounted device camera, and wherein the method further includes: adjusting the first head-mounted device camera and / or the second head-mounted device camera based on the deviation.

[0061] Example 55 is a wearable system configured to perform a method according to any one of Examples 1-54.

[0062] Example 56 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any of the methods of Examples 1-54. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 An augmented reality scene observed through a wearable augmented reality device is shown;

[0064] Figure 2A-2B An augmented reality device with a fixed focal plane is shown;

[0065] Figure 3 A schematic diagram of an example wearable system is shown;

[0066] Figure 4 An example showing how a visual tracking system can be incorporated into an augmented reality system with wearable and handheld devices;

[0067] Figure 5 A diagram showing a localization task performed by an augmented reality system;

[0068] Figure 6A perspective view of a controller of an enhancement system is shown;

[0069] Figure 7A-7B Examples of camera exposure and intervals of reference flashes under different lighting conditions are shown;

[0070] Figure 8 An example of the interval of camera exposure and reference flash for head mounted device tracking and constellation tracking is shown;

[0071] Fig. 9 shows an example synchronization of camera exposure and reference flash interval under bright lighting conditions;

[0072] Fig.10 An example of synchronizing a reference flash with a low exposure interval is shown;

[0073] Fig.11 An example method of headset camera exposure and reference flicker synchronization according to some embodiments of the present invention is shown;

[0074] Figures 12A-12C The use of an independently controlled benchmark group for disambiguation is shown;

[0075] Figures 13A-13D The use of misaligned flashes of fiducials for disambiguation is shown;

[0076] Figures 14A-14C The use of flashing coding of controller reference points for disambiguation is shown;

[0077] Figures 15A-15C The use of independently controlled fiducial point groups and flicker coding for disambiguation is shown;

[0078] Fig.16 An example method of reference flicker disambiguation for a controller according to some embodiments of the present invention is shown;

[0079] Fig.17 An example of a mobile device that can be used as a handheld device for a wearable system is shown;

[0080] Fig.18 An example method of using a device as a controller and displaying a reference point on a display of the device according to some embodiments of the present invention is shown;

[0081] Figures 19A-19B shows an exemplary internal perspective view of a controller of a wearable system;

[0082] Figures 20A-20B shows a perspective view of an exemplary module for maximum reuse;

[0083] Figures 21A-21B shows a perspective view of an exemplary mixing module;

[0084] Figures 22A-22B shows a perspective view of another exemplary module;

[0085] Figures 23A-23B A modular controller applied to a drone is shown;

[0086] Fig.24 shows a constellation of passive or active fiducial points included in a controller of a head mounted device sensing wearable system;

[0087] Fig.25 Table showing the iterations required to test all perspective 3-point combinations;

[0088] Fig.26A An example of a set of reference points arranged in a plurality of symmetry groups for reference point association is shown;

[0089] Fig.26B A wearable system including a head mounted device having an imaging device that captures an image of a controller is shown;

[0090] Fig.26C shows the general association problem using the perspective 3-point algorithm;

[0091] Fig.27A shows a failure to associate detections with fiducials to determine the pose of the controller;

[0092] Fig.27B shows the success of associating detections with reference points to determine the pose of the controller;

[0093] Fig.28 Another example of associating a reference point of a controller with an object in an image is shown;

[0094] Fig.29 An example method of using rotationally symmetric fiducial association according to some embodiments of the present invention is shown;

[0095] Fig. 30A Another head mounted device is shown that senses a constellation of passive or active fiducials included in a controller of a wearable system;

[0096] Fig. 30B Table showing the iterations required to test all perspective 2-point combinations;

[0097] Fig.31A An example of a projected reference point for a controller is shown;

[0098] Fig.31BA wearable system is shown, including a head mounted device having an inertial measurement unit and an imaging device that captures an image of a controller;

[0099] Fig.31C shows the general association problem using the perspective 2-point algorithm;

[0100] Fig.32A shows a failure to associate detections with fiducials to determine the pose of the controller;

[0101] Fig.32B shows the success of associating detections with reference points to determine the pose of the controller;

[0102] Fig.33 An example method of associating reference points using inertial measurement unit measurements according to some embodiments of the present invention is shown;

[0103] Fig.34 An example of a hand tracking region of interest is shown;

[0104] Figures 35A-35E Exemplary steps for performing hand tracking using the position of the hand to reduce the image search space are shown;

[0105] Figures 36A-36E Exemplary steps for performing hand tracking using the position and orientation of the hand to reduce an image search space are shown;

[0106] Figures 37A-37F Exemplary steps for performing hand tracking to reduce the image search space are shown;

[0107] Fig.38 An example method 3800 for narrowing a constellation search area using hand tracking according to some embodiments of the present invention is shown;

[0108] Fig.39 A diagram illustrating an example of a calibration profile for a wearable device;

[0109] Fig.40 Various examples of calibration deformation amounts are shown;

[0110] Fig.41 Calibrating a head mounted device or other wearable device using an image containing fiducials positioned on a controller is shown;

[0111] Fig.42 An example method of head mounted device camera calibration using reference points according to some embodiments of the present invention is shown; and

[0112] Fig.43 A simplified computer system is shown. DETAILED DESCRIPTION

[0113] Some aspects of the present disclosure relate to positioning (e.g., location, orientation, and / or distance) of a handheld device (e.g., a controller) relative to a wearable device (e.g., an augmented reality (AR), virtual reality (VR), or mixed reality (MR) headset). In some cases, six degrees of freedom (6DOF) pose tracking of the headset can be performed using one or more headset sensors (e.g., one or more headset inertial measurement units (IMUs)) and one or more headset cameras, a technique referred to as "headset inside-out tracking." For each image captured by the headset camera, features can be identified in the image, and the pixel locations of the identified features can be compared to the pixel locations of the same features in other images, allowing the 6DOF pose of the headset to be calculated for each image.

[0114] At the same time, the 6DOF posture tracking of the controller can be performed using a combination of head-mounted device sensors and controller sensors or components, and using one or both of two independent technologies. The first technology is called "controller inside-out tracking", that is, the controller is tracked for 6DOF posture based on the image of the real-world environment captured by the camera on the controller. For each captured image, features can be identified, and the pixel position of the identified features can be compared with the pixel position of the same features in other images, allowing the 6DOF posture of the controller to be calculated for each image. The second technology is called "constellation tracking", that is, based on the image of the reference point (e.g., light-emitting diode (LED)) fixed on the controller captured by the camera on the head-mounted device, the controller is tracked for 6DOF posture. The reference point can be programmed to flash (i.e., emit light) when the head-mounted device camera is exposed, so that each captured image contains flashing reference points, which can then be identified by the imaging processing routine. The pixel position of the identified reference point in the image can be determined, and the identified reference point can be associated with the known geometry of the reference, so that the 6DOF posture of the controller can be determined.

[0115] During operation of the wearable system, one or both of these controller pose tracking techniques may be used. For example, if controller inside-out tracking is not available (e.g., the image of the environment does not include a sufficient number of features due to low light conditions), the wearable system may rely on constellation tracking. Conversely, if constellation tracking is not available (e.g., the controller fiducials are not within the field of view of the head mounted device camera), the wearable system may rely on controller inside-out tracking. Furthermore, if both tracking techniques are available, the tracking data generated by the two techniques may be fused together.

[0116] In traditional VR or AR systems, 6DOF posture tracking of peripheral devices is achieved by combining a series of electromagnetic sensors and transmitters that are strategically placed on the user's AR headset, remote device, and / or other auxiliary devices (e.g., totems, tactile devices, gaming devices, etc.). Typically, an electromagnetic tracking system includes at least one electromagnetic field transmitter and at least one electromagnetic field sensor. Since the emitted electromagnetic field has a known distribution, the detected field can be analyzed to determine the position and / or orientation of the peripheral device. Although such a system provides a simple solution to the positioning problem, additional solutions are needed to provide higher precision positioning. Embodiments of the present disclosure may replace or supplement electromagnetic tracking systems.

[0117] When used in an AR system, the 6DOF pose tracking information of the handheld device can facilitate the operation of the AR system. For example, the AR system can generate virtual content that represents or interacts with a controller, and the virtual content makes the user feel comfortable. For example, during a game played by multiple users using a virtual ball and a virtual bat, the AR system can generate virtual content for the virtual bat that is accurately positioned and oriented with the controller of the user holding the bat.

[0118] Figure 1 An AR scene 100 as viewed through a wearable AR device according to some embodiments of the present disclosure is shown. In the AR scene 100, a user of the AR technology sees a real-world park-like environment 106 having various real-world objects 130, such as people, trees, buildings, and a real-world concrete platform 120 in the background. In addition to these items, the user of the AR technology also feels that they "see" various virtual objects 102, such as a robot statue 102-2 standing on the real-world concrete platform 120, and a cartoon avatar character 102-1 flying by, which appears to be a personification of a bumblebee, even though these elements (character 102-1 and statue 102-2) do not exist in the real world. Because human visual perception and nervous systems are extremely complex, it is extremely challenging to develop a VR or AR technology that can comfortably, naturally, and richly present virtual image elements in virtual or real-world image elements.

[0119] Figure 2AAn AR device 200A with a single fixed focal plane is shown according to some embodiments of the present disclosure. During operation, the projector 214 of the AR device 200A can project virtual image light 222 (i.e., light associated with virtual content) onto the eyepiece 202-1, which can cause the light field (i.e., the angular representation of the virtual content) to be projected onto the user's retina in a manner that causes the user to perceive the corresponding virtual content to be located at a certain location within the user's environment. For example, the virtual image light 222 coupled out of the eyepiece 202-1 can cause the user to perceive the character 102-1 to be located at a first virtual depth plane 210-1, while the statue 102-2 is located at a second virtual depth plane 210-2. The user perceives the virtual content and the world light 232 corresponding to one or more world objects 230 (e.g., the platform 120). In some embodiments, the AR device 200A includes a first lens assembly 205-1 located on the user side of the eyepiece 202-1 (the side of the eyepiece 202-1 closest to the user's eyes) and a second lens assembly 205-2 located on the world side of the eyepiece 202-1. Each of the lens assemblies 205-1, 205-2 can be configured to impart optical power to light passing therethrough.

[0120] Figure 2B An AR device 200B with two fixed focal planes is shown in accordance with some embodiments of the present disclosure. During operation, the projector 214 can project virtual image light 222 onto the first eyepiece 202-1 and the second eyepiece 202-2, which can cause the light field to be projected onto the user's retina in a manner that causes the user to perceive the corresponding virtual content to be located at a certain location within the user's environment. For example, the virtual image light 222 coupled out of the first eyepiece 202-1 can cause the user to perceive the character 102-1 to be located at the first virtual depth plane 210-1, while the virtual image light 222 coupled out of the second eyepiece 202-2 can cause the user to perceive the statue 102-2 to be located at the second virtual depth plane 210-2.

[0121] Figure 3 A schematic diagram of an example wearable system 300 according to some embodiments of the present disclosure is shown. The wearable system 300 may include a wearable device 301 and at least one remote device 303 remote from the wearable device 301 (e.g., separate hardware but communicatively coupled). When the wearable device 301 is worn by a user (typically as a head-mounted device), the remote device 303 may be held by the user (e.g., as a handheld controller) or mounted in various configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user (e.g., in a backpack configuration, in a belt-coupled configuration, etc.).

[0122] The wearable device 301 may include a left eyepiece 302A and a left lens assembly 305A configured side by side, and the left eyepiece 302A and the left lens assembly 305A constitute a left optical stack. The left lens assembly 305A may include an accommodating lens on the user side of the left optical stack and a compensating lens on the world side of the left optical stack. Similarly, the wearable device 301 may include a right eyepiece 302B and a right lens assembly 305B configured side by side, and the right eyepiece 302B and the right lens assembly 305B constitute a right optical stack. The right lens assembly 305B may include an accommodating lens on the user side of the right optical stack and a compensating lens on the world side of the right optical stack.

[0123] In some embodiments, the wearable device 301 includes one or more sensors, including but not limited to: a left-front facing world camera 306A directly attached to or near the left eyepiece 302A, a right-front facing world camera 306B directly attached to or near the right eyepiece 302B, a left-facing world camera 306C directly attached to or near the left eyepiece 302A, a right-facing world camera 306D directly attached to or near the right eyepiece 302B, a left eye tracking camera 326A pointed at the left eye, a right eye tracking camera 326B pointed at the right eye, and a depth sensor 328 attached between the eyepieces 302. The wearable device 301 may include one or more image projection devices, such as a left projector 314A optically linked to the left eyepiece 302A and a right projector 314B optically linked to the right eyepiece.

[0124] The wearable system 300 may include a processing module 350 for collecting, processing, and / or controlling data within the system. Components of the processing module 350 may be distributed between the wearable device 301 and the remote device 303. For example, the processing module 350 may include a local processing module 352 on the wearable portion of the wearable system 300 and a remote processing module 356 that is physically separated from and communicatively connected to the local processing module 352. Each of the local processing module 352 and the remote processing module 356 may include one or more processing units (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.) and one or more storage devices, such as non-volatile memory (e.g., flash memory).

[0125] The processing module 350 may collect data captured by various sensors of the wearable system 300, such as the camera 306, the eye tracking camera 326, the depth sensor 328, the remote sensor 330, the ambient light sensor, the microphone, the inertial measurement unit (IMU), the accelerometer, the compass, the global navigation satellite system (GNSS) unit, the radio device, and / or the gyroscope. For example, the processing module 350 may receive the image 320 from the camera 306. Specifically, the processing module 350 may receive the left front image 320A from the left front facing world camera 306A, the right front image 320B from the right front facing world camera 306B, the left side image 320C from the left side facing camera 306C, and the right side image 320D from the right side facing world camera 306D. In some embodiments, the image 320 may include a single image, a pair of images, a video including an image stream, a video including a paired image stream, etc. Image 320 may be generated and sent to processing module 350 periodically when wearable system 300 is powered on, or may be generated in response to instructions sent by processing module 350 to one or more cameras.

[0126] Camera 306 can be configured in various positions and orientations on the outer surface of wearable device 301 to capture images around the user. In some cases, cameras 306A, 306B can be positioned to capture images that substantially overlap the FOV of the user's left eye and right eye, respectively. Therefore, the position of camera 306 can be close to the user's eyes, but not too close to block the user's FOV. Alternatively or in addition, cameras 306A, 306B can be positioned to align with the incoupling positions of virtual image light 322A, 322B, respectively. Cameras 306C, 306D can be positioned to capture images of the side of the user, for example, in the user's peripheral vision or outside the user's peripheral vision. Images 320C, 320D captured using cameras 306C, 306D do not necessarily need to overlap with images 320A, 320B captured using cameras 306A, 306B.

[0127] In some embodiments, the processing module 350 may receive ambient light information from an ambient light sensor. The ambient light information may indicate a range of brightness values ​​or spatially resolved brightness values. The depth sensor 328 may capture a depth image 332 of the wearable device 301 facing forward. Each value of the depth image 332 may correspond to the distance between the depth sensor 328 and the nearest detected object in a particular direction. As another example, the processing module 350 may receive eye tracking data 334 from the eye tracking camera 326, which may include images of the left eye and the right eye. As another example, the processing module 350 may receive projected image brightness values ​​from one or both of the projectors 314. The remote sensor 330 located within the remote device 303 may include any of the above-mentioned sensors having similar functionality.

[0128] The virtual content is delivered to the user of the wearable system 300 using the projector 314 and the eyepiece 302 and other components in the optical stack. For example, the eyepieces 302A, 302B may include transparent or semi-transparent waveguides configured to guide and outcouple light generated by the projectors 314A, 314B, respectively. Specifically, the processing module 350 may cause the left projector 314A to output the left virtual image light 322A to the left eyepiece 302A, and may cause the right projector 314B to output the right virtual image light 322B to the right eyepiece 302B. In some embodiments, the projector 314 may include a micro-electromechanical system (MEMS) spatial light modulator (SLM) scanning device. In some embodiments, each of the eyepieces 302A, 302B may include a plurality of waveguides corresponding to different colors. In some embodiments, the lens assemblies 305A, 305B may be coupled and / or integrated with the eyepieces 302A, 302B. For example, the lens assemblies 305A, 305B may be incorporated into a multi-layer eyepiece and may form one or more layers that make up one of the eyepieces 302A, 302B.

[0129] Figure 4 An example of how a visual tracking system can be incorporated into an AR system having a wearable device 401 (e.g., a head-mounted device) and a handheld device 404 (e.g., a controller) is shown. In some embodiments, the handheld device 404 is a handheld controller that allows a user to provide input to the AR system. For example, the handheld device 404 can be a totem for a game scene. The handheld device 404 can be a tactile device and can include one or more tactile surfaces utilizing various sensor types. During operation of the AR system, the user can hold the handheld device 404 in the left or right hand by actively grasping the handheld device 404 and / or by securing an attachment mechanism (e.g., a wraparound band) to the user's hand.

[0130] The handheld device 404 may include one or more fiducials 422 positioned along one or more exterior surfaces of the handheld device 404 such that the fiducials may be within the field of view of an imaging device external to the handheld device 404. The fiducials 422 may have a known geometric relationship relative to one another such that an imaging device may determine its position and / or orientation relative to the handheld device 404 by capturing an image of one or more fiducials 422. The fiducials 422 may be dynamic, static, powered, unpowered, and in some embodiments may be distinguishable from one another. For example, a first fiducial may be an LED having a first wavelength, while a second fiducial may be an LED having a second wavelength. Alternatively or additionally, different fiducials may have different brightness and / or may pulse at different frequencies (e.g., a first fiducial may pulse at 100 Hz, while a second fiducial may pulse at 150 Hz).

[0131] The handheld device 404 may include one or more imaging devices (referred to herein as controller cameras 426) positioned in such a manner that when the user holds the handheld device 404, some features of the wearable device 401 and / or the surrounding of the handheld device 404 are within the field of view of the imaging device. For example, the front controller camera 426A may be positioned so that its field of view is away from the user toward one or more features around the handheld device 404, and the rear controller camera 426B may be positioned so that its field of view is toward the wearable device 401. The controller cameras 426 may include one or more forward-facing imaging devices and / or one or more rear-facing imaging devices to create the desired cumulative field of view. In some embodiments, the controller cameras 426 may capture still or moving images.

[0132] The handheld device 404 may include an IMU (referred to herein as a controller IMU 424) that is rigidly fixed within the handheld device 404 so that rotational and linear motion of the handheld device 404 are also experienced by the controller IMU 424. In some cases, the controller IMU 424 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, the controller IMU 424 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, the controller IMU 424 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, the controller IMU 424 may output raw data from which any of the above-described forms of processed data may be calculated.

[0133] The handheld device 404 may include a rechargeable and / or replaceable battery 428 or other power source that powers the fiducials 422, the controller camera 426, the controller IMU 424, and any other components of the handheld device 404. Figure 4 Although not shown, the handheld device 404 may include circuitry for enabling wireless communication with the wearable device 401 and / or the remote device 440. For example, when the controller camera 426 and the controller IMU 424 are used to detect or capture data, the handheld device 404 may transmit raw data or processed data to the wearable device 401 and / or the remote device 440.

[0134] The wearable device 401 may include one or more imaging devices (referred to herein as head mounted device cameras 410) positioned such that when the user holds the hand-held device 404, the hand-held device 404 including the reference point 422 is within the field of view of the imaging device. For example, one or more head mounted device cameras 410 may be located on the front of the wearable device 401, above, below, and / or to the side of the optical see-through component of the wearable device 401. In one embodiment, two head mounted device cameras 410 may be positioned on opposite sides of the optical see-through component of the wearable device 401. In some embodiments, the head mounted device cameras 410 may capture still or moving images.

[0135] The wearable device 401 may include a head mounted device IMU 408 that is rigidly fixed within the wearable device 401 such that rotational and linear motion of the wearable device 401 are also experienced by the head mounted device IMU 408. In some cases, the head mounted device IMU 408 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, the head mounted device IMU 408 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, the head mounted device IMU 408 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, the head mounted device IMU 408 may output raw data from which any form of processed data described above may be calculated.

[0136] In some embodiments, the AR system may include a remote device 440, which may include a computing device (e.g., one or more processors and associated memory) for performing positioning of the handheld device 404 relative to the wearable device 401. Alternatively or additionally, the computing device may reside in the wearable device 401 itself, or even in the handheld device 404. The computing device may receive raw data or processed data from each of the head-mounted device IMU 408, the head-mounted device camera 410, the controller IMU 424, and the controller camera 426 (via wired and / or wireless connections), and may calculate the geospatial position of the handheld device 404 (relative to the geospatial position of the wearable device 401) and the orientation of the handheld device 404 (relative to the orientation of the wearable device 401). In one or more embodiments, the computing device may also include a mapping database 442 (e.g., a traversable world model, a coordinate space, etc.) to detect gestures, determine coordinates of real objects and virtual objects, and may even connect to cloud resources and a traversable world model. In some embodiments, images captured using the head mounted device camera 410 and / or the controller camera 426 may be used to construct a traversable world model. For example, features may be detected in the captured images, and the collected data (e.g., sparse points) may be used to construct a traversable world model or environment map.

[0137] Figure 5 FIG. 1 shows a diagram of a positioning task performed by an AR system, wherein the position and orientation of a handheld device 504 relative to a wearable device 501 is determined. In the illustration, the wearable device 501 has a position relative to a world reference defined as (X WP , Y WP , Z WP )’s geospatial location (“wearable location”) and relative to a world reference defined as (X WO , Y WO , Z WO ) ("wearable orientation"). In some cases, the geospatial location of wearable device 501 is expressed as longitude, latitude, and altitude values, and the orientation of wearable device 501 is expressed as pitch, yaw, and roll angle values.

[0138] As shown, the handheld device 504 has a geospatial location (X) relative to the wearable device 501. WP , Y WP , Z WP ) is defined as (X' HP , Y' HP , Z' HP )'s geospatial location ("handheld location") and orientation (X) relative to the wearable device 501 WO , Y WO , ZWO ) is defined as (X' HO , Y' HO , Z' HO ) ("hand-held orientation"). In some cases, the geospatial position of the handheld device 504 is expressed in X, Y, and Z Cartesian values, and the orientation of the handheld device 504 is expressed in pitch, yaw, and roll values. As a specific example, when the handheld device 504 is held by the user, the geospatial position of the handheld device 504 may be equal to (0.7m, -0.5m, 0.1m), and the orientation of the handheld device 504 may be equal to (10.2°, -46.2°, 15.2°).

[0139] Figure 6 A perspective view of a controller 604 of an AR system is shown. In some embodiments, the controller 604 includes reference points 622 positioned along one or more outer surfaces of the controller 604 so that the reference points can be located within the field of view of an imaging device outside the controller 604. For example, the controller 604 may include nine reference points positioned on the top surface 606 of the controller 604 and two reference points positioned on opposite sides of the controller 604. The two reference points on both sides of the controller 604 can be positioned near the controller camera 626. In other examples, the controller 604 may include more or fewer reference points 622. The reference points 622 can have a known relationship relative to each other so that the imaging device can determine its position and / or orientation relative to the controller 604 by capturing images of one or more reference points 622. As shown, the reference points 622 on the top surface 606 are arranged in a circle, but other configurations are also possible.

[0140] In some cases, 6DOF pose tracking of a head mounted device (e.g., wearable device 401, 501, 601) can be performed using images captured by a camera of the head mounted device in combination with a head mounted device IMU. This technique is referred to as head mounted device inside-out tracking. For each captured image, features can be identified, and the pixel positions of the identified features can be compared to the pixel positions of the same features in other images, so that the 6DOF pose of the head mounted device can be calculated for each image.

[0141] In some cases, 6DOF pose tracking of a controller 604, which can be an example of a handheld device 404 and 504, can be performed using one or both of two separate techniques. The first technique is (1) controller inside-out tracking, which is 6DOF pose tracking of the controller 604 based on images of the real-world environment captured by a controller camera 626 on the controller 604. Similar to head-mounted device inside-out tracking, for each captured image, features can be identified and the pixel locations of the identified features can be compared to the pixel locations of the same features in other images, allowing the 6DOF pose of the controller 604 to be calculated for each image. The second technique is (2) constellation tracking, which is 6DOF pose tracking of the controller 604 based on images of fiducials 622 (e.g., LEDs) fixed to the controller 604 captured by a camera on the head-mounted device. The 6DOF pose of the controller 604 can be calculated based on the images of the controller 604 captured by the head-mounted device camera. The image can be any single frame in which at least three fiducials 622 are visible. One or both of these techniques can be used during operation of the wearable system. For example, if inside-out tracking of the controller is not available (e.g., due to low light conditions, the image of the environment does not include a sufficient number of features), the wearable system can rely on constellation tracking. Conversely, if constellation tracking is not available (e.g., the controller fiducials are not within the field of view of the head mounted device camera), the wearable system can rely on inside-out tracking of the controller. Furthermore, if both tracking technologies are available, the tracking data generated by the two technologies can be fused together.

[0142] Since images captured by the headset are used for both inside-out tracking of the headset and constellation tracking, a problem may arise in that fiducial 622 is visible in the images used for headset tracking because the blinking fiducial may appear as a recognizable feature during inside-out tracking of the headset. To address this issue, different images may be used for inside-out tracking of the headset (or simply "headset tracking") and constellation tracking, and fiducial 622 may be controlled to blink when capturing images for constellation tracking. In addition, the blinking "on" time may be shortened during constellation tracking to prevent image blur, reduce power consumption, and allow fiducial 622 to be easily identified. During constellation tracking, the exposure interval of the camera may also be shortened to reduce power consumption. To ensure that the image for constellation tracking includes the blinking fiducial, fiducial 622 may be controlled to blink multiple times around the exposure interval of the camera, for example, the fiducial interval may be calculated to be centered around the exposure interval of the camera.

[0143] The reference points 622 can be dynamic, static, powered, unpowered, and in some embodiments can be distinguished from each other. For example, a first reference point can be an LED having a first wavelength, and a second reference point can be an LED having a second wavelength. Alternatively or in addition, different reference points can have different brightness and / or can pulse at different frequencies (for example, a first reference point can pulse at 100 Hz, and a second reference point can pulse at 150 Hz). Reference points 622 can normally flash at a first frequency, but can flash at a second frequency when inside-out tracking is lost. For example, reference points 622 can normally flash at 2 Hz, but can flash at 30 Hz when inside-out tracking is lost.

[0144] The controller 604 includes a controller camera 626 positioned so that when the user holds the controller 604, certain features of the head mounted device and / or the surrounding of the controller 604 are within the field of view of the controller camera 626. For example, the controller 604 may include a front controller camera positioned so that its field of view is oriented away from the user toward one or more features around the controller 604, and a rear controller camera positioned so that its field of view is oriented toward the head mounted device. The controller camera 626 may include one or more forward facing imaging devices and / or one or more rear facing imaging devices to create a desired cumulative field of view. In some embodiments, the controller camera 626 may capture still or moving images.

[0145] Fig. 7A Examples of intervals of camera exposure and reference flicker in nominal mode are shown. Nominal mode can be used for lighting conditions where the camera exposure of a world camera (e.g., one or more cameras of a head mounted device) remains relatively large (e.g., about 1 millisecond or greater). The flicker of the reference point can be synchronized with the world camera exposure so that the flicker occurs during the world camera exposure to ensure that the flicker of the reference point is captured by the world camera. As shown, the world camera exposure may be greater than 1 millisecond and the reference flicker may be 0.01 milliseconds, but other times are possible. The reference flicker can occur at a known offset from the world camera exposure. For example, the reference flicker can occur at the beginning of the world camera exposure, centered at the world camera exposure, or at the end of the world camera exposure (e.g., as shown in FIG. 1 ). Fig. 7A ). Thus, when the world camera exposure is turned on at a predetermined interval, the fiducials may be flashed at the same predetermined interval so that the flashing always occurs during the world camera exposure, thereby allowing the controller to be continuously tracked.

[0146] Figure 7BAn example of the interval of camera exposure and reference flicker in high ambient light mode is shown. High ambient light mode can be used for lighting conditions where the camera exposure time of the world camera is short (e.g., less than 1 millisecond) due to bright ambient light. It may be difficult to synchronize the reference pulse with the short camera exposure, so the reference can be flickered at a rate with a period equal to the world camera exposure time, rather than synchronizing the flicker and the world camera exposure. Therefore, the world camera is guaranteed to capture the flicker of the reference point. For example, if the world camera exposure is 0.5 milliseconds, the reference point can be flickered by 0.01 milliseconds every 0.5 milliseconds, thereby ensuring that the flicker occurs during the world camera exposure to ensure that the flicker of the reference point is captured by the world camera. The reference flicker is shown as occurring in the middle of the world camera exposure, but the reference flicker can also occur at the beginning of the world camera exposure or at the end of the world camera exposure.

[0147] Figure 8 Examples of intervals of camera exposure and reference blinking for head mounted device tracking and constellation tracking are shown. To track the head mounted device and the controller, the wearable system can alternate between head mounted device tracking frames and constellation tracking frames. Each frame can be ~16ms (16.6ms at 60Hz), so the length of a pair of head mounted device tracking frames and constellation tracking frames can be 33ms. The process can start with a head mounted device tracking frame, where the head mounted device captures an image of the world to determine the pose of the head mounted device. The world camera exposure of the head mounted device camera can be at least 1ms. As shown, the world camera exposure for head mounted device tracking is shown as lasting 16ms duration of the head mounted device tracking frame. Since the head mounted device camera does not position the controller relative to the head mounted device during head mounted device tracking, the controller's reference point does not blink during the head mounted device tracking frame.

[0148] After the head mounted device tracking frame, a constellation tracking frame may occur in which the head mounted device captures images of the fiducials of the controller to determine the pose of the controller relative to the head mounted device. The world camera exposure of the head mounted device camera may be less than 1 millisecond. As shown, the world camera exposure for head mounted device tracking is shown as lasting only a portion of the 16 millisecond duration of the constellation tracking frame. Since the world camera exposure is less than 1 millisecond, the fiducials may be flashed with a period equal to the world camera exposure duration to ensure that the head mounted device camera captures an image of the flashing fiducials during the constellation tracking frame. After the constellation tracking frame, the wearable system may return to head mounted device tracking using another head mounted device tracking frame. Alternating between head mounted device tracking frames and constellation tracking frames may continue until the wearable system is powered off.

[0149] like Figure 8As shown, for each constellation tracking frame, the wearable system can determine a reference interval 820 defined by a reference start time 822 and a reference end time 824, during which the controller's set of reference points will flash a plurality of times at a reference frequency (with an associated reference period 826). In some examples, the duration of each flash (referred to as a flash pulse width) can be adjusted by the wearable system. The wearable system can determine the reference interval such that the world camera exposure interval during each constellation tracking frame at least partially overlaps with the reference interval.

[0150] Fig. 9 An example synchronization of camera exposure and reference blinking interval in bright lighting conditions is shown. The reference points of the head mounted device camera and the controller can be synchronized via Bluetooth so that the reference blinking occurs during the camera exposure. Thus, the head mounted device can send a signal to the controller indicating the time or interval that the world camera exposure will be on so that the controller can cause the reference point to blink at the same time or during the interval. However, Bluetooth communication may increase the uncertainty of time synchronization along with other factors such as internal clock drift in the head mounted device and the controller. In some examples, the uncertainty may exceed 200μs.

[0151] Under normal and low light conditions (e.g., indoors), the world camera exposure is significantly larger than the synchronization uncertainty and the reference pulse width, as shown by camera exposure 912A and flash 914A. Therefore, the synchronization algorithm can use Bluetooth to synchronize camera exposure 912A and flash 914A by placing flash 914A at the center of camera exposure 912A. However, in bright light environments (e.g., sunny outdoor environments), the values ​​of the camera exposure and synchronization error intervals become close and synchronization fails. The reference pulse can be increased to equal the uncertainty interval to fix the failure, but at the expense of the reference appearing brighter on the image because the total integration time of the reference pulse is increased. Conversely, synchronization can be fixed by generating a reference flash sequence with the same pulse width and a period equal to the camera exposure, while maintaining equivalent reference brightness in the camera image. Therefore, the flash sequence width may be equal to the uncertainty window.

[0152] Camera exposures 912B-912D and corresponding flickers 914B-914D show reference flickers with a reference period (i.e., the period between consecutive reference flickers) that is equal to the duration of the world camera exposure (i.e., the "exposure duration"), where the exposure duration of camera exposure 912C is shorter than the exposure duration of camera exposure 912B, and the exposure duration of camera exposure 912D is shorter than the exposure duration of camera exposure 912C. Flicker 914B occurs at the beginning of camera exposure 912B, flicker 914C occurs at the end of camera exposure 912C, and flicker 914D is centered around camera exposure 912D. In any case, the total integrated reference pulse may be substantially the same as in a normal brightness environment.

[0153] Fig.10 An example of synchronizing a reference flash with a low exposure interval 1030 is shown. The low exposure interval 1030 is a time interval during which the world camera exposure of the head mounted device camera is turned on for constellation tracking under bright lighting conditions. To synchronize the reference flash with the low exposure interval 1030, the head mounted device determines a low exposure offset 1032. The low exposure offset 1032 can be a time between a reference time and the beginning, end, or middle of the low exposure interval 1030. Fig.10 As shown, a first exemplary low exposure offset 1032 is the time between the start of world camera exposure during a head mounted device tracking frame and the middle of a low exposure interval 1030 during a constellation tracking frame that occurs immediately after the head mounted device tracking frame. A second exemplary low exposure offset 1032 is the time between the start of a constellation tracking frame and the middle of a low exposure interval 1030 during a constellation tracking frame.

[0154] The head mounted device may transmit an indication of the low exposure offset 1032 and the exposure duration of the low exposure interval 1030 to the controller so that the controller may determine a reference interval 1020 in which multiple reference flashes will occur. The reference interval 1020 may be centered about the low exposure interval 1030 to increase the likelihood that at least one reference flash overlaps with the low exposure interval 1030. Fig.10 As shown, if five reference flashes occur during a constellation tracking frame, the reference interval 1020 can align the middle of the third reference flash with the middle of the low exposure interval 1030 based on the low exposure offset 1032.

[0155] Fig.11 An example method 1100 for head mounted device camera exposure and reference flicker synchronization according to some embodiments of the present invention is shown. One or more steps of the method 1100 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 1100 may be omitted during the execution of the method 1100. In addition, two or more steps of the method 1100 may be performed simultaneously or concurrently.

[0156] At step 1102, a head mounted device camera is caused to capture a first exposure image having an exposure above a threshold during a head mounted device tracking frame. The first exposure image is captured during a head mounted device tracking frame. The wearable system may alternate between performing head mounted device tracking and controller tracking. The head mounted device camera of the head mounted device repeatedly captures images during the head mounted device tracking frame and the controller tracking frame. The threshold may be 1 millisecond, so the first exposure image may be associated with an exposure greater than 1 millisecond. The first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration.

[0157] At step 1104, the head mounted device camera is caused to capture a second exposure image having an exposure below a threshold during the controller tracking frame. The second exposure image is captured during the controller tracking frame. Since the threshold may be 1 ms, the second exposure image may be associated with an exposure of less than 1 ms. The second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration. The second exposure duration is less than the first exposure duration.

[0158] At step 1106, a reference interval is determined during the controller tracking frame. The reference interval is an interval during which the reference point set will flash a plurality of times. The reference interval is defined by a reference start time and a reference end time, during which the reference point set will flash at a reference frequency and a reference period. The reference interval is determined such that the second exposure interval at least partially overlaps with the reference interval.

[0159] At step 1108, the reference point set is flashed a plurality of times within a reference interval during the controller tracking frame. The reference point set is flashed according to a reference frequency and a reference period. Thus, the reference point set may be captured in a second exposure image during the controller tracking frame. Furthermore, the posture of the controller may be determined based on the reference point set captured in the second exposure image.

[0160] During constellation tracking, issues may arise if flashing fiducials of multiple controllers are visible in the same headset image. For example, it may be difficult for the wearable system to determine which fiducials belong to which controller, which may render constellation tracking ineffective if not resolved. Both controllers may be held by the user of the wearable system, or by two different users of two different wearable systems.

[0161] In some cases, the wearable system may perform several steps to perform one or more of a plurality of multi-controller disambiguation techniques. Figures 12-15 illustrate exemplary techniques for multi-controller disambiguation. Optionally, before performing any of these techniques, a set of preliminary steps may be performed. For example, the wearable system may capture an image using a head mounted device camera, identify a set of reference points in the image, and determine that at least one of the identified set of reference points belongs to a first controller and at least one of the identified set of reference points belongs to a second controller. The wearable system may determine that the set of reference points belongs to the first controller and the second controller based on a determined configuration of reference points in the image, a number of reference points in the image, or other techniques.

[0162] Figures 12A-12C The use of independently controlled sets of fiducials for disambiguation is shown. Fig. 12A , the controller 1204 is shown as having 11 reference points 1222, 9 of which are located on the top surface of the controller 1204 and 2 are located on opposite sides of the controller 1204. The reference points 1222 can be LEDs arranged in multiple groups, which can be controlled independently. These groups can be asymmetric so that one group cannot look the same as another group for any translation, rotation, or inversion of the pattern. These groups play an important role in disambiguating multiple devices that can be seen by a single wearable device.

[0163] Fig. 12B 1222I. Thus, the first group is asymmetric to the second group. In some examples, the two groups of reference points may include at least one common reference point, while in other examples, the two groups of reference points may be mutually exclusive.

[0164] To disambiguate between the first controller 1204A and the second controller 1204B, the wearable system may flash a first set of reference points 1222 at the first controller 1204A and a second set of reference points 1222 at the second controller 1204B, such as Fig. 12C The head mounted device can capture the first and second sets of flashing images and detect which set of fiducials is depicted in the images. Fig. 12C) and captured in an image, the wearable system can also distinguish between the first controller 1204A and the second controller 1204B based on the pattern of the group of reference points. The wearable system can store or access a mapping between the controllers 1204A-1204B and the groups of reference points to determine which controller corresponds to a particular group of reference points detected in the image. Fig. 12C Each instance of the reference point 1222 of the first controller 1204A and the second controller 1204B in the flash corresponds to one image capture. That is, the world camera exposure 912 of the head mounted device camera can occur during each flash.

[0165] Figures 13A-13D The disambiguation of flicker using reference point misalignment is shown. Fig.13A , first controller 1304A and second controller 1304B are shown as each having a reference point 1322 located on a top surface. Rather than separating reference points 1322 into unique groups for each of controllers 1304A-1304B, controllers 1304A-1304B may be made to flash at different periods, frequencies, or offsets relative to one another. For example, Fig. 13B As shown, the flashing of the reference point 1322A of the first controller 1304A can be caused to occur in an offset manner relative to the flashing of the reference point 1322B of the second controller 1304B, so that the reference point pairs from different controllers do not flash at the same time. In one example, the offset can be 2 milliseconds, and the wearable system can flash the reference point 1322B of the second controller 1304B at 0 milliseconds, 4 milliseconds, and 8 milliseconds, and the wearable system can flash the reference point 1322A of the first controller 1304A at 2 milliseconds, 6 milliseconds, and 10 milliseconds. Fig. 13B Each instance of the reference points 1322A-1322B of the first controller 1304A and the second controller 1304B in the flash corresponds to one image capture. That is, the world camera exposure 912 of the head mounted device camera can occur during each flash.

[0166] Fig. 13C The modification of the period or frequency of flashing of the reference point 1322A of the first controller 1304A relative to the flashing of the reference point 1322B of the second controller 1304B is shown. As an example, the wearable system may flash the reference point 1322 of the second controller 1304B once every 4 milliseconds (e.g., at 0 milliseconds, 4 milliseconds, and 8 milliseconds), but the wearable system may flash the reference point 1322 of the first controller 1304A once every 6 milliseconds (e.g., at 0 milliseconds, 6 milliseconds, and 12 milliseconds). In addition, Fig.13DFIG. 1 shows the modification of the offset and period of the flashing of the reference point 1322A of the first controller 1304A relative to the flashing of the reference point 1322B of the second controller 1304B, which is actually Figures 13B-13C In some cases, the period, frequency, or offset of the second controller 1304B may be modified in addition to or in lieu of modification of the first controller 1304A. Regardless, by knowing the pattern in which the reference points of the controllers flash and the time at which the images of the controllers were captured, the wearable system can distinguish between multiple controllers that may be depicted in the image. Figures 13B-13D Each instance of the reference points 1322A-1322B of the first controller 1304A and the second controller 1304B in the flash corresponds to one image capture. That is, the world camera exposure 912 of the head mounted device camera can occur during each flash.

[0167] Figures 14A-14C The use of flashing codes for controller reference points for disambiguation is shown. Fig.14A , the first controller 1404A and the second controller 1404B are shown as each having a reference point 1422 located on the top surface. The wearable system can introduce a random code into the flashes so that the reference point 1422 of each controller 1404A-1404B can be uniquely identified. Each controller can have a unique encoding. That is, the flashes of the first controller 1404A can be encoded with a first encoding, while the flashes of the second controller 1404B can be encoded with a second encoding. In some examples, the encoding can indicate which flashes are omitted during the flash sequence, resulting in the flashes no longer being periodic. The encoding can be associated with a binary pattern (e.g., 110110110).

[0168] The encoding may be applied to only one of the controllers 1404A, such as Fig. 14B As shown in the first controller 1404A, the encoding has a binary pattern of 110110110. Therefore, while the flashing of the reference point 1422B of the second controller 1404B is periodic, the flashing of the reference point 1422A of the first controller 1404A is not periodic. As shown, the first controller 1404A and the second controller 1404B flash simultaneously for two time periods, and then the second controller 1404B flashes in a third time period when the first controller 1404A does not flash.

[0169] Fig. 14CThe introduction of encoding to each controller 1404A-1404B is shown. If each blinking cycle corresponds to 1 millisecond, the first encoding of the first controller 1404A (having the binary pattern 110110110) may cause the reference point 1422A to blink at 0 milliseconds, 1 millisecond, 3 milliseconds, 4 milliseconds, 6 milliseconds, and 7 milliseconds. The second encoding of the second controller 1404B (having the binary pattern 101101011) may cause the reference point 1422B to blink at 0 milliseconds, 2 milliseconds, 3 milliseconds, 5 milliseconds, 7 milliseconds, and 8 milliseconds. Regardless of whether encoding is introduced to one or both of the controllers 1404A-1404B, by knowing the pattern of the controller's reference point blinking and the time when the image of the controller was captured, the wearable system can distinguish between multiple controllers that may be depicted in the image. In some examples, one or both of the first encoding or the second encoding may be a random code. Figures 14B-14C Each instance of the reference points 1422A-1422B of the first controller 1404A and the second controller 1404B in a flash corresponds to one image capture. That is, the world camera exposure 912 of the head mounted device camera can occur during each flash.

[0170] Figures 15A-15C The use of independently controlled fiducial point groups and flicker coding for disambiguation is shown. Fig.15A , the first controller 1504A and the second controller 1504B are shown as each having a reference point 1522 located on the top surface. The reference points 1522 can be arranged in multiple groups that can be independently controlled, such as Fig. 12B These groups can be asymmetric so that one group cannot look the same as another for any translation, rotation, or inversion of the pattern. These groups play an important role in disambiguating multiple devices that can be seen by a single wearable device.

[0171] Fig. 15B 15. The reference point groups of two controllers are shown. For example, the first controller 1504A can be associated with a first group of reference points 1522, while the second controller 1504B can be associated with a second group of reference points 1522. The first group includes reference points 1522A, 1522D, 1522F, and 1522H, while the second group includes reference points 1522B, 1522C, 1522E, 1522G, and 1522I. Thus, the first group is asymmetric to the second group. In addition to these two groups, the reference points 1522 of the controllers 1504A-1504B can also be encoded with different flash memory encodings, such as Figures 14B-14C and Fig. 15C Thus, the combination of groups and different blink codes can enable the wearable system to distinguish between controllers 1504A-1504B. Fig. 15CEach instance of the reference point 1522 of the first controller 1504A and the second controller 1504B in the flash corresponds to one image capture. That is, the world camera exposure 912 of the head mounted device camera can occur during each flash.

[0172] Fig.16 An example method 1600 of reference point flicker disambiguation for a controller (e.g., controller 604) according to some embodiments of the present invention is shown. One or more steps of the method 1600 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 1600 may be omitted during the execution of the method 1600. In addition, two or more steps of the method 1600 may be performed simultaneously or concurrently.

[0173] A set of images is captured using a head mounted device camera of a head mounted device at step 1602. The images may show one or more controllers within a field of view of the head mounted device camera.

[0174] In step 1604, a reference point is identified in the set of repeatedly flashed images. The reference point may flash at a reference frequency and a reference period based on the exposure of the head mounted device camera during a reference interval. The identified reference point may belong to one or more controllers.

[0175] In step 1606, it is determined that at least some of the reference points include a first set of reference points belonging to the first controller and a second set of reference points belonging to the second controller. Based on the number or location of the reference points, the wearable system can determine that some of the reference points belong to the first controller and some of the reference points belong to the second controller.

[0176] In step 1608, it is determined that the flashing of the first set of reference points is at least partially aligned in time with the flashing of the second set of reference points. The wearable system can determine that the reference interval of the first set of reference points is the same as the reference interval of the second set of reference points, so they are aligned in time. Alternatively, the wearable system can determine that the first set of reference points and the second set of reference points both appear in each image in the image set and are therefore aligned in time. If the reference points flash with the same frequency and period, the wearable system may not be able to accurately track the posture of the first controller and the second controller.

[0177] At step 1610, a period, frequency, or offset associated with at least one of the first set of reference points or the second set of reference points is modified such that blinking of the first set of reference points is not aligned with blinking of the second set of reference points. The wearable system may blink a first subset of the first set of reference points at the first controller and a second subset of the second set of reference points at the second controller. The first subset may be asymmetric with the second subset. Additionally or alternatively, blinking of the first set of reference points and / or the second set of reference points may be encoded with an encoding.

[0178] Fig.17 An example of a mobile device that can be used as a handheld device for a wearable system is shown. The mobile device can be a cell phone, tablet, or other device with a display. Alternatively, in some examples, a sticker pattern placed on a surface of known concavity may be appropriate. Figure 4 Similar to the handheld device 404 in , the mobile device 1704 includes a camera and an IMU. Mobile phones already have cameras and IMUs. In some examples, software such as ARCore and ARKit can use the camera and IMU of the mobile device 1704 for SLAM tracking. The mobile device 1704 also includes a user interface 1706, which can display reference points 1722 based on a set of pixel positions of the display of the mobile device 1704 for 6DOF pose tracking of the mobile device 1704. The user interface 1706 can also display buttons 1724 based on other pixel positions that can receive user input, which together with the camera, IMU, and reference points 1722 provide all the functions of the controller.

[0179] Displaying reference points 1722 on user interface 1706 may allow each mobile device to have a unique reference point configuration. That is, reference points 1722 are not limited to circles, and thus user interface 1706 may display reference points 1722 in a unique pattern that may be used to distinguish mobile device 1704 from other controllers. For example, reference points 1722 may be displayed on user interface 1706 as squares, while another mobile device may display reference points in a star shape.

[0180] To use the mobile device 1704 as a controller, the fiducials 1722 can be displayed as bright spots on a dark digital screen (e.g., user interface 1706) or sticker background. The head-mounted device of the wearable system can capture an image including the mobile device 1704 and process the image using an algorithm that detects the pattern of fiducials 1722 to identify the mobile device 1704 as a controller. The fiducials 1722 can be an always-on display to limit the dynamics to slow motion in the FOV. The wearable system can cause the fiducials 1722 to flash according to a period and frequency. In some embodiments, the always-on display may be suitable for tasks such as writing on a virtual whiteboard.

[0181] In some embodiments, the wearable system may cause the mobile device 1704 to modify the period, frequency, or set of pixel locations at which the fiducials 1722 are displayed. For example, the fiducials 1722 may be displayed for shorter known time periods that are coordinated with the head mounted device camera, which allows tracking at faster dynamics because motion blur may be limited. Alternatively, if the wearable system determines that the image of the mobile device 1704 also depicts another controller, the wearable system may cause the mobile device 1704 to modify the period, frequency, or set of pixel locations to distinguish the fiducials 1722 of the mobile device 1704 from the fiducials of the other controller. The wearable system may synchronize the display of the fiducials 1722 with the exposure interval of the head mounted device camera so that the fiducials 1722 are visible in the image captured by the head mounted device camera. The exposure interval may be based on Figure 8-11 The IMU of the mobile device 1704 can be further used to refine the determined position and orientation of the mobile device 1704, and the tracking time can be slightly extended when all the reference points 1722 are visible in the camera FOV. The camera of the mobile device 1704 can be used to perform visual inertial odometry (VIO) to provide tracking outside the FOV of the head mounted device camera and increase the dynamic range of the tracking.

[0182] Fig.18 An example method 1800 is shown according to some embodiments of the present invention, which uses a device as a controller and displays a reference point on a display of the device. One or more steps of the method 1800 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 1800 may be omitted during the execution of the method 1800. In addition, two or more steps of the method 1800 may be performed simultaneously or concurrently.

[0183] In step 1802, a controller is caused to display a reference point set on a display. The reference point set is displayed according to a pixel position set. The controller may also display one or more buttons on the display according to another pixel position set, the buttons being configured to receive user input.

[0184] A set of images is captured using a head mounted device camera of the head mounted device at step 1804. If the controller is within the field of view of the head mounted device camera, the image may show the controller.

[0185] In step 1806, a set of fiducial points is identified in the set of images. Fiducial points of another controller depicted in the set of images may also be identified.

[0186] In step 1808, the position and / or orientation of the controller relative to the head mounted device is determined based on the identified set of reference points. Knowing the position and orientation of the head mounted device, the wearable system can use the identified reference points to determine the pose of the controller. In addition, the wearable system can modify the period, frequency, or pixel position set to disambiguate between the reference point set and the reference points of another controller.

[0187] Figures 19A-19B An exemplary internal perspective view of a controller for a wearable system is shown. The optical 6DOF platform of the controller 1904 may include a VIO sensor module 1912, a constellation module 1914, a main printed circuit board (PCB) 1916, a wireless communication engine 1918, a battery 1920, and a haptic engine 1922. Other components such as triggers, bumpers, touch pads, and input buttons may also be included. The VIO sensor module 1912 may include an IMU and one or more cameras on a rigid base. The constellation module 1914 may include a reference array (e.g., LED). The main PCB 1916 includes a through-hole pad (VIP) and LED drive electronics for the controller 1904. The wireless communication engine 1918 includes components for communicating with a head-mounted device or other devices of the wearable system. The battery 1920 provides power to the controller 1904, and the haptic engine 1922 provides vibration and other sensory outputs in response to input received by the controller 1904.

[0188] In some cases, only the constellation module 1914, the main PCB 1916, and the input buttons are needed to provide 6DOF tracking. Other components can be selected depending on the application. Thus, the controller 1904 can be modular. Modules can be used to control and / or track external devices - one example of which is a drone. Another example (e.g., for pure tracking) could be to attach one of these modules (or a complete controller) to a gun for military or law enforcement training, so the direction the rifle is pointing can be tracked and / or "showed" to the user via a head mounted device. Such an embodiment could be useful in combat or training.

[0189] Figures 20A-20B A perspective view of an exemplary module for maximum reuse is shown. Module 2000 includes a VIO sensor module 1912, a constellation module 1914, a main PCB 1916, a wireless communication engine 1918, and a battery 1920. Module 2000 may also include input buttons and LED user indicators. However, module 2000 may lack other user inputs, such as a touch pad and a trigger. In addition, module 2000 may lack a haptic engine. In one example, the dimensions of module 2000 are approximately 84 mm in length, 64 mm in width, and 18 mm in thickness, and may be powered and communicated via a universal serial bus (USB).

[0190] Figures 21A-21B A perspective view of an exemplary hybrid module is shown. The hybrid module 2100 includes a VIO sensor module 1912, a constellation module 1914, a main PCB 1916, and a wireless communication engine 1918. The hybrid module 2100 may also include input buttons and LED user indications. However, the hybrid module 2100 may lack other user inputs, such as a touch pad and a trigger. In addition, the hybrid module 2100 may lack a battery and a haptic engine. Therefore, the hybrid module 2100 can be powered by a USB connection. In one example, the approximate dimensions of the hybrid module 2100 may be 64 mm in diameter and 18 mm in thickness.

[0191] Figures 22A-22B A perspective view of another exemplary module is shown. Module 2200 includes constellation module 1914, main PCB 1916, wireless communication engine 1918, and battery 1920. Module 2200 may also include input buttons and LED user indications. However, module 2200 may lack other user inputs, such as a touch pad and trigger. In addition, module 2200 may lack a VIO sensor module and a haptic engine. Therefore, module 2200 can only be used for constellation tracking when it is in the FOV of the head-mounted device camera, and may be limited to tracking at 30 frames per second, which can extend the battery life of module 2200. In one example, the approximate dimensions of module 2200 may be 64 mm in diameter and 18 mm in thickness.

[0192] Other modules may also be used. For example, the smallest possible module may include a VIO sensor module, a constellation module, a main PCB, input buttons, and user indications. Thus, the smallest module may lack a wireless communication engine, a battery, other user inputs, and a haptic engine. Thus, the module may be powered and communicate via USB. In one example, the approximate dimensions of a module may be 50 mm in diameter and 15 mm thick.

[0193] The modular nature of the controller allows the controller to be used in drone and unmanned aerial vehicle (UAV) applications such as Figures 23A-23B Modules (e.g. Figures 21A-21B The hybrid module 2100 in FIG. 2100 may be connected to the drone 2330. Alternatively, the hardware of the hybrid module 2100 may be integrated into the drone 2330 to allow for a larger baseline between the constellation reference points 2322. For example, Fig. 23B As shown, if the reference points 2322 are incorporated into the drone 2330, the distance between them may be farther than if they were part of the hybrid module 2100 connected to the drone 2330, as shown. Fig.23A shown.

[0194] Drone 2330 can be controlled by an application on wearable system 2300. Thus, when the user's device (e.g., head mounted device 2302) sees reference point 2322 on drone 2330, drone 2330 is precisely located. Drone 2330 can then fly out of the user's line of sight and rely on VIO for navigation and communication of position and altitude to the user. When drone 2330 returns to line of sight, it is precisely located again and any errors in the path can be improved. Drones with controller modules can be used for package and item delivery applications, reconnaissance and mapping applications, and remote inspections of construction sites or other dangerous areas. In some cases, drones can be attached with other sensors, including GPS, radar, etc.

[0195] Fig.24 A head mounted device 2401 of a wearable system is shown, which includes an image sensor 2426 that can sense a constellation of passive or active fiducials included in a controller 2404 of the wearable system. The constellation can be fiducials 2422. The head mounted device 2401 can also include an algorithm to detect fiducials. However, the algorithm can be affected by a high percentage of outlier detections. For example, the algorithm can detect other lights near the controller 2404 and erroneously associate those lights with the controller 2404 (e.g., Fig.24 A good correlation is when the algorithm detects only the reference point 2422 of the controller 2404 for tracking.

[0196] To associate the controller's fiducials, the headset 2401 can aim to find an association between an array of fiducials and their corresponding detections, where the number of detections is typically greater than the number of fiducials. This association problem can be solved in a brute force fashion by employing a voting matrix and a minimal pose estimator called perspective 3-point (P3P), iteratively voting after trying correspondences to obtain the correct re-projection. The problem scales with the factorial of the number of fiducials and detections:

[0197]

[0198] Among them, n D is the number of detections caused by the projected light source, n L is the number of active reference points of the controller.

[0199] Fig.25 A table showing the iterations required to test all P3P combinations is shown. The columns represent the number of detections and the rows represent the number of fiducials. The expansion of this problem has a large impact on the bandwidth of algorithms that rely on this sensing modality.

[0200] Fig.26AAn example set of reference points arranged in multiple symmetry groups for reference point association is shown. Since the constellation is rotationally symmetric at the gap, enough constellation detections can be found to calculate the median distance between the reference points. Due to the rotational symmetry, Equation 1 can be simplified to:

[0201]

[0202] Group 2634 may include three fiducials that can be matched to any three detections. The wearable system may compute the pose using a P3P algorithm. The remaining fiducials may then be projected using the pose, and inliers may be computed by comparing them to a fixed re-projection tolerance. The group with the largest number of inliers may be selected so that the association is correct up to the axis of rotational symmetry (e.g., gap). The wearable system may project the pattern onto a plane and rotate the pattern around the axis of symmetry to find the best match. In other words, gaps in the constellation may be found. The process may work by making the rotational symmetry imperfect at the gap, i.e., the gap is less than or greater than the median adjacent marker distance.

[0203] Fig.26B A wearable system is shown, including a head mounted device 2601 having an imaging device 2626 that captures an image of a controller 2604 having fiducials 2622, many of which are activated (i.e., illuminated). The image can be analyzed to identify multiple detections that may correspond to groups of pixels having a particular range of values ​​that are believed to correspond to activated fiducials. In some cases, one or more detections may be caused by an interfering light source 2610 that projects light onto the imaging device 2626. Such a light source 2610 may be from an internal or external lighting source and may be direct light or reflected light (e.g., light that reflects from one or more surfaces before reaching the imaging device 2626). Thus, the image may include detections corresponding to the fiducials 2622 of the controller 2604 as well as detections caused by other light sources, and the wearable system may need to determine which detections are associated with the fiducials 2622 of the controller 2604 so that the controller 2604 can be accurately tracked.

[0204] Fig.26C The general association problem using the P3P algorithm is shown. The algorithm can receive an image 2620 captured by an imaging device 2626 of a head mounted device 2601 and a model of a constellation of controllers 2604 being tracked. The output of the algorithm can be a 6DOF pose of the controller 2604.

[0205] Fig.27AA failure to associate detections with reference points to determine a posture of a controller 2704 is shown. The wearable system receives an image 2720A of a field of view of a head mounted device camera and identifies a set of objects 2724A-2724C in the image 2720A. The set of objects 2724A-2724C may include detections of projections of reference points and / or projections of other light sources. The wearable system selects a subset of the set of objects in the image 2720A. The subset may include a predetermined number (e.g., at least three) of objects. The wearable system may also select a subset of the set of objects from a model 2742A of the controller 2704 that includes the reference points. Thus, the selected objects 2724A-2724C in the image 2720A may be two-dimensional points, while the selected objects in the model 2742A may be three-dimensional points. Fig.27A , the predetermined number of subsets is three.

[0206] A subset of objects may be input into a P3P algorithm, which outputs a 6DOF pose of the controller 2704. The pose of the controller 2704 is calculated by associating the subset with a set of reference points that are rotationally symmetric with respect to each other and each include a predetermined number of reference points. The wearable system may calculate statistics for the associated subset based on the pose compatibility of the reference point sets. The statistics may include, for example, errors associated with fitting the reference point set to the subset of objects. In some examples, the wearable system may project the pose onto the image 2720A and verify the alignment of the object subset 2724A-2724C with the remaining points in the image 2720A. The wearable system may determine the number of sets of objects 2724A-2724C that are aligned with the reference point set of the pose. Fig.27A This may be a failure case because only three objects were determined to match the pose calculated by the algorithm. Objects 2724A-2724C match the pose because they were the objects originally selected in image 2720A. However, only object 2724C corresponds to reference point 2722 of controller 2704, so the six reference points of the projected pose do not match. Therefore, the wearable system can select a different subset of the set of objects and calculate a new pose to improve the association of the pose with the reference points.

[0207] Fig.27B The successful association of detections with fiducials to determine the posture of controller 2704 is shown. The wearable system receives image 2720B of the field of view of the head mounted device camera and identifies a set of objects 2724D-2724F in image 2720B. The set of objects 2724D-2724F may include detections of projections of fiducials and / or projections of other light sources. Fig.27B, objects 2724D-2724F each correspond to a reference point of controller 2704. The wearable system selects a subset of the set of objects 2724D-2724F in image 2720B. The subset may include a predetermined number (e.g., at least three) of objects. The wearable system may also select a subset of the set of objects from model 2742B of controller 2704 that includes the reference points. Thus, the selected objects 2724D-2724F in image 2720B may be two-dimensional points, while the selected objects in model 2742B may be three-dimensional points. Fig.27B , the predetermined number of subsets is three.

[0208] The subset of objects may be input into a P3P algorithm, which outputs a 6DOF pose of the controller 2704. The pose of the controller 2704 is calculated by associating the subset with a set of reference points that are rotationally symmetric with respect to each other and each include a predetermined number of reference points. The wearable system may calculate statistics for the associated subset based on pose compatibility of the set of reference points. To do this, the wearable system may project the pose into the image 2720B and verify alignment of the subset of objects with the remaining points in the image 2720B. The wearable system may determine the number of sets of objects 2724D-2724FC that are aligned with the set of reference points for the pose. Fig.27B This may be a success because all nine reference points of the controller 2704 are determined to match the pose calculated by the algorithm. Therefore, the wearable system can determine based on the statistics that the association between the set of objects and the set of reference points is correct, thereby determining that the pose calculated for the controller 2704 is accurate.

[0209] Fig.28 Another example of associating the controller's fiducials with objects in an image is shown. The wearable system can create a circular fiducial pattern so that the distances between triplets of fiducials 2822 are the same. This means that the triangles constructed from each triplet are the same when rotating around the center of the circle. The gap between two adjacent fiducials is larger than the gaps between other adjacent pairs of fiducials. L / 3A triangle can be constructed from the triplets of reference point 2822.

[0210] The wearable system attempts to associate objects detected in the image with the controller's fiducials 2822. To do this, the wearable system selects a subset of objects depicted in the image that form triplets. The subset of objects is input into the P3P algorithm, which computes the pose in the following way:

[0211]

[0212] The pose can be rotated about the center of the triplet to determine how all possible other triangles fit into the set of objects. This process can be repeated for multiple subsets of objects and multiple poses. The pose that matches the most fiducials can be determined as the pose of the controller. Fig.28 , it can be determined that the first pose 2844A matches all but one of the reference points, the second pose 2844B does not match multiple reference points, and the third pose 2844C matches all of the reference points. Boxes 2846A-2846B illustrate the misalignment of the reference points in the first pose 2844A and the second pose 2844B, respectively. The wearable system can determine that the third pose 2844C is the most accurate pose for the controller because it matches the most reference points.

[0213] Fig.29 An example method 2900 using rotationally symmetric fiducial association according to some embodiments of the present invention is shown. One or more steps of the method 2900 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 2900 may be omitted during the execution of the method 2900. In addition, two or more steps of the method 2900 may be performed simultaneously or concurrently.

[0214] At step 2902, a controller's set of reference points is flashed. The reference points are arranged in a known geometric shape (e.g., a circle) that includes multiple groups of reference points that are rotationally symmetric relative to each other. The number of each group in the multiple groups of reference points can be equal to a predetermined number (e.g., at least three).

[0215] At step 2904, a head mounted device camera of the head mounted device is caused to capture an image. If the controller is within the field of view of the head mounted device camera when the image is captured, the image may show the controller.

[0216] In step 2906, a set of objects in the image corresponding to the reference points is identified. The set of objects may correspond to a set of reference points of the controller and / or one or more light sources projected in the image.

[0217] At step 2908, the set of objects is associated with the set of reference points based on the known geometry. Subsets of the set of objects may be repeatedly selected, and a pose of the controller may be calculated by associating the subsets with multiple sets of reference points. Each subset may have a number equal to a predetermined number. Statistics of the associated subsets may be calculated based on pose compatibility of the multiple sets of reference points, and a correct association between the set of objects and the set of reference points may be found based on the statistics. The subset of the set of objects may be input into a P3P algorithm configured to output a pose of the controller.

[0218] Fig. 30AA head mounted device 3001 of a wearable system is shown, which includes an image sensor 3026 that can sense a constellation of passive or active fiducials contained in a controller 3004 of the wearable system. The constellation may be fiducials 3022. The head mounted device 3001 may execute an algorithm to detect fiducials. However, the algorithm may be affected by a high percentage of outlier detections. For example, the algorithm may detect other lights near the controller 3004 and erroneously associate those lights with the controller 3004 (e.g., Fig. 30A A good correlation is when the algorithm detects only the reference point 3022 of the controller 3004 for tracking.

[0219] To associate the controller's fiducials, the headset 3001 can aim to find an association between an array of fiducials and their corresponding detections, where the number of detections is typically greater than the number of fiducials. This association problem can be solved in a brute force fashion by employing a voting matrix and a minimum pose estimator called P3P, iterating the voting after trying correspondences to obtain the correct re-projection. As shown in Equation 1, this problem scales with the factorial of the number of fiducials and detections.

[0220] Fig. 30B A table showing the number of iterations required to test all P2P combinations is shown. If the controller 3004 includes an IMU capable of determining rotational measurements of the controller 3004 relative to the head mounted device 3001, a P2P algorithm may be used instead of a P3P algorithm. The P2P algorithm may be a standard P2P algorithm or a gravity based P2P algorithm. The columns represent the number of detections and the rows represent the number of reference points. By Fig. 30B The table with Fig.25 The number of iterations required to test all combinations can be greatly reduced by comparing the IMU measurements made by the controller 3004.

[0221] Fig.31A An example of a projected reference point of controller 3104 is shown. Since controller 3104 includes IMU 3112B, reference point 3122 is projected at a known orientation. Therefore, Equation 1 can be simplified to:

[0222]

[0223] Fig.31BA wearable system is shown, including a head mounted device 3102 having an IMU 3112A and an imaging device 3126 that captures an image of a controller 3104 having fiducials 3122, many of which are activated (i.e., emitting light). The image can be analyzed to identify multiple detections that may correspond to groups of pixels having a particular range of values ​​that are believed to correspond to activated fiducials. In some cases, one or more detections may be caused by an interfering light source 3110 that projects light onto the imaging device 3126. Such a light source 3110 may be from an internal or external lighting source and may be direct light or reflected light (e.g., light that reflects from one or more surfaces before reaching the imaging device 3126). Thus, the image may include detections corresponding to the fiducials 3122 of the controller 3104 as well as detections caused by other light sources, and the wearable system may need to determine which detections are associated with the fiducials 3122 of the controller 3104 so that the controller 3104 can be accurately tracked. The head mounted device 3101 may include an IMU 3112A, and the controller 3104 may include an IMU 3112B so that the rotation of the controller 3104 relative to the head mounted device 3101 may be determined.

[0224] Fig.31C The general association problem using a P2P algorithm is shown. The algorithm can receive an image 3120 captured by an imaging device 3126 of a head mounted device 3102 and a model of a constellation of controllers 3104 being tracked. The output of the algorithm can be a 6DOF pose of the controller 3104.

[0225] Fig.32A A failure to associate detections with reference points to determine a posture of a controller 3204 is shown. The wearable system receives an image 3220A of a field of view of a head mounted device camera and identifies a set of objects 3224A-3224B in the image 3220A. The set of objects 3224A-3224B may include detections of reference projections and / or other light source projections. Object 3224A is a reference point for the controller 3204, while object 3224B is a detection of another light source projection. The wearable system selects a subset of the set of objects in the image 3220A. The subset may include a predetermined number (e.g., at least two) of objects. The wearable system may also select a subset of the set of objects from a model 3242A of the controller 3204 that includes the reference point. Thus, the selected objects in the image 3220A may be two-dimensional points, while the selected objects in the model 3242A may be three-dimensional points. In Fig.32A , the predetermined number of subsets is three.

[0226] A subset of objects and a rotation measurement (e.g., 3DOF orientation) of the controller 3204 determined from the IMU can be input into the P3P algorithm, which outputs a 6DOF pose of the controller 3204. The pose of the controller 3204 is calculated by associating the subset with the reference points based on the known geometry of the reference point arrangement. The wearable system can repeatedly select a subset of the object set and calculate the pose of the controller 3204 by associating the subset with the reference point group. The wearable system calculates statistics of the associated subset based on the pose compatibility of the reference point group and based on the rotation measurement. The statistics may include, for example, errors associated with fitting the reference point group to the subset of objects. In some examples, the wearable system can project the pose into the image 3220A and verify the alignment of the subset of objects 3224A-3224B with the remaining points in the image 3220A. The wearable system can determine the number of object sets that are aligned with the reference set of the pose. Fig.32A This may be a failure situation because only three of the fiducials of controller 3204 were determined to match the pose calculated by the algorithm. Objects 3224A-3224C were determined to match the pose, while the other six objects were determined not to match. Therefore, the wearable system may select a different subset of the set of objects and calculate a new pose to improve the association of the pose with the fiducials.

[0227] Fig.32B Success in associating detections with reference points to determine a posture of controller 3204 is shown. The wearable system receives image 3220B of the field of view of a head mounted device camera and identifies a set of objects 3224D-3224E in image 3220B. The set of objects 3224D-3224E may include detections of reference projections and / or other light source projections. Objects 3224D-3224E each correspond to a reference point of controller 3204. The wearable system selects a subset of the set of objects in image 3220B. The subset may include a predetermined number (e.g., at least two) of objects. The wearable system may also select a subset of the set of objects from a model 3242B of the controller 3204 that includes the reference points. Thus, the selected objects in image 3220B may be two-dimensional points, while the selected objects in model 3242B may be three-dimensional points. Fig.32B , the predetermined number of subsets is three.

[0228] The subset of objects and the rotational measurements (e.g., 3DOF orientation) of the controller 3204 determined from the IMU can be input into the P2P algorithm, which outputs a 6DOF pose of the controller 3204. The pose of the controller 3204 is calculated by associating the subset with the reference points based on the known geometry of the reference point arrangement. The wearable system can repeatedly select subsets of the object set and calculate the pose of the controller 3204 by associating the subset with the reference point group. The wearable system calculates statistics of the associated subsets based on the pose compatibility of the reference point group and based on the rotational measurements. To do this, the wearable system can project the pose into the image 3220A and verify the alignment of the object subset with the remaining points in the image 3220A. The wearable system can determine the number of object sets that are aligned with the reference point set of the pose. Fig.32B This may be a success case because all nine reference points of the controller 3204 are determined to match the pose calculated by the algorithm. Therefore, the wearable system can determine from the statistics that the association between the set of objects and the set of reference points is correct, and therefore the calculated pose is accurate for the controller 3204.

[0229] Fig.33 An example method 3300 for associating reference points using IMU measurements according to some embodiments of the present invention is shown. One or more steps of the method 3300 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 3300 may be omitted during the execution of the method 3300. In addition, two or more steps of the method 3300 may be performed simultaneously or concurrently.

[0230] In step 3302, a reference point set of the controller is flashed. The reference point set is arranged in a known geometric shape. The reference point set can flash at a reference frequency and a reference period.

[0231] At step 3304, a head mounted device camera of the head mounted device is caused to capture an image. If the controller is within the field of view of the head mounted device camera when the image is captured, the image may show the controller.

[0232] At step 3306, a set of objects in the image corresponding to the reference points is identified. The set of objects may correspond to a set of reference points of the controller and / or one or more light sources projected in the image.

[0233] In step 3308, rotation measurements are captured using the controller inertial measurement unit of the controller. The rotation measurements may correspond to the position and orientation of the controller relative to the head mounted device.

[0234] At step 3310, the set of objects is associated with the set of reference points based on the known geometry. Subsets of the set of objects may be repeatedly selected, and the pose of the controller may be calculated by associating the subsets with multiple sets of reference points. Statistics of the associated subsets may be calculated based on pose compatibility of the multiple sets of reference points, and the correct association between the set of objects and the set of reference points may be found based on the statistics. The subset of the set of objects may be input into a P2P algorithm configured to output the pose of the controller.

[0235] Reducing the image search area during constellation tracking has several benefits. First, portions of the image that contain fiducials of other controllers can be avoided, thereby eliminating the need to perform one or more controller disambiguation methods. Second, reducing the image search area can reduce the likelihood of false positives when identifying fiducials (e.g., due to "LED-like" features on LEDs or other objects), which can be more problematic for 6DOF pose tracking algorithms than false negatives. Third, reducing the image search area can reduce search time and allow for faster calculation of the controller's 6DOF pose.

[0236] Fig.34 An example of a hand tracking region of interest is shown. In some embodiments, the hand tracking data is used to reduce the image search area by reducing the size of the image and / or the search area of ​​the image based on the hand gesture. Alternatively or additionally, the fiducials may still be identified throughout the image, and any identified fiducials outside the region of interest may then be eliminated. In one example, a camera 3426 of a head mounted device 3401 may capture an image 3420 containing fiducials 3422 of a controller 3404. The head mounted device 3401 may detect a hand 3406 in the image and generate hand tracking data about the hand 3406. The hand tracking data may be used to identify the position and / or orientation of the user's hand 3406 in the image 3420. A region of interest 3408 in the image 3420 may be determined based on the position and / or orientation of the hand 3406. As an example, the region of interest 3408 may be a predetermined area around the hand 3406 (e.g., a box starting 2 inches above the hand). Next, the region of interest 3408 may be searched to identify one or more fiducials 3422. The identified reference point 3422 can be associated with the reference point 3422 of the controller to calculate the 6DOF pose of the controller 3404.

[0237] In many cases, hand tracking is already performed when the controller 3404 is in the user's hand 3406, so the cost of utilizing this hand tracking data during constellation tracking is low. Alternatively or additionally, some embodiments may perform hand tracking in response to generating a command to shrink the image search area.

[0238] Figures 35A-35EExemplary steps for performing hand tracking using the position of the hand to reduce the image search space are shown. Fig.35A , image 3520 is captured using a head mounted device camera. Image 3520 may depict a constellation of reference points of the user's hand 3506 and controller 3504. Thus, the head mounted device camera may generate hand tracking data that may be used to determine the position and / or orientation of hand 3506 in the image. Fig.35B , the hand tracking data is used to determine the position and / or orientation of the hand 3506. The position can be expressed as the distance of the hand 3506 in each direction from the head mounted device, and the orientation can be expressed as the rotation of the hand 3506 relative to the head mounted device in each direction.

[0239] exist Fig.35C , the head mounted device may determine an area of ​​interest 3508 based on the position of the hand 3506. As an example, a reference point may be determined, and the area of ​​interest 3508 may be a circular area of ​​a certain radius around the reference point. For example, the reference point may be a point where the hand 3506 is determined to intersect with the controller 3504. A predefined radius may be established around the reference point, and the area within the radius may be the area of ​​interest 3508. In an example, the radius may be 8 inches.

[0240] exist Fig.35D In the image, objects in the region of interest 3508 are identified. The objects can be light projections corresponding to the reference points of the controller 3504, or other light sources projected into the image. For example, in the region of interest 3508, nine objects can be identified, each corresponding to a reference point of the controller 3504. Fig.35E In the embodiment of the present invention, the wearable system determines the 6DOF pose of the controller 3504 based on the identified object. The wearable system can use the P3P algorithm to perform the same Fig.29 A process similar to that described in , or using the P2P algorithm and the IMU data of the controller 3504 to perform the same Fig.33 A process similar to the process described in is used to determine the 6DOF pose of controller 3504.

[0241] Figures 36A-36E Exemplary steps for performing hand tracking using the position and orientation of the hand to reduce the image search space are shown. Fig.36A , image 3620 is captured using a head mounted device camera. Image 3620 may depict a constellation of reference points of the user's hand 3606 and controller 3604. Thus, the head mounted device camera may generate hand tracking data that may be used to determine the position and / or orientation of hand 3606 in image 3620. Fig.36B, the hand tracking data is used to determine the position and / or orientation of the hand 3606. The position can be expressed as the distance of the hand 3606 from the head mounted device in each direction, and the orientation can be expressed as the rotation of the hand 3606 relative to the head mounted device in each direction.

[0242] exist Fig.36C , the head mounted device may determine an area of ​​interest 3608 based on the position and orientation of the hand 3606. As an example, a reference point may be determined, and the area of ​​interest 3608 may be an elliptical area of ​​a specific radius around the reference point. For example, the reference point may be a point where the hand 3606 is determined to intersect the controller 3604. An ellipse with a predefined radius may be established around the reference point, and the area within the ellipse may be the area of ​​interest 3608. The orientation of the hand 3606 may be used to tilt the area of ​​interest 3608 to the direction in which the controller 3604 is held.

[0243] exist Fig.36D In the image, objects in the region of interest 3608 are identified. The objects can be light projections corresponding to the reference points of the controller 3604, or other light sources projected into the image. For example, in the region of interest 3608, nine objects can be identified, each corresponding to a reference point of the controller 3604. Fig.36E In the embodiment of the present invention, the wearable system determines the 6DOF pose of the controller 3604 based on the identified object. The wearable system can use the P3P algorithm to perform the same Fig.29 A process similar to that described in , or using the P2P algorithm and the IMU data of the controller 3604 to perform the same Fig.33 A process similar to the process described in is used to determine the 6DOF pose of controller 3604.

[0244] Figures 37A-37F Exemplary steps for performing hand tracking to reduce the image search space are shown. Fig.37A , image 3720 is captured with a head mounted device camera. Image 3720 may depict a user's hand 3706 and at least one constellation of reference points for at least one controller. As shown, image 3720 may depict a first constellation of reference points for controller 3704A and a second constellation of reference points for controller 3704B. The head mounted device may generate hand tracking data that may be used to determine a position and / or orientation of hand 3706 in image 3720. Fig.37B , an object is identified in image 3720. The object may be a light projection corresponding to a reference point of controller 3704A and / or other light sources projected in image 3720 (e.g., a reference point of controller 3704B).

[0245] exist Fig.37CIn , the hand tracking data is used to determine the position and / or orientation of the hand 3706. The position can be expressed as the distance between the hand 3706 and the head mounted device in each direction, and the orientation can be expressed as the rotation of the hand 3706 relative to the head mounted device in each direction. Fig.37D , the head mounted device may determine an area of ​​interest 3708 based on the position of the hand 3706. As an example, a reference point may be determined, and the area of ​​interest 3708 may be a circular area of ​​a certain radius around the reference point. For example, the reference point may be a point where the hand 3706 is determined to intersect with the controller 3704. A predefined radius may be established around the reference point, and the area within the radius may be the area of ​​interest 3708. In an example, the radius may be 8 inches.

[0246] exist Fig.37E , the wearable system excludes identified objects outside of region of interest 3708. Thus, since the reference point of controller 3704A is within region of interest 3708 and the reference point of controller 3704B is outside region of interest 3708, the reference point of controller 3704B may be excluded from constellation gesture tracking. Fig.37F In the embodiment of the present invention, the wearable system determines the 6DOF pose of the controller 3704A based on the identified objects. The wearable system can use the P3P algorithm to perform the same Fig.29 or using the P2P algorithm and the IMU data of the controller 3704A to perform the same Fig.33 A similar process as described in is used to determine the 6DOF pose of controller 3704A.

[0247] Fig.38 An example method 3800 for narrowing a constellation search area using hand tracking according to some embodiments of the present invention is shown. One or more steps of the method 3800 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 3800 may be omitted during the execution of the method 3800. In addition, two or more steps of the method 3800 may be performed simultaneously or concurrently.

[0248] In step 3802, a reference point set of the controller is flashed. The reference point set is arranged in a known geometric shape. The reference point set can flash at a reference frequency and a reference period.

[0249] At step 3804, a head mounted device camera of the head mounted device is caused to capture an image. If the controller is within the field of view of the head mounted device camera when the image is captured, the image may show the controller.

[0250] A set of objects in the image corresponding to the reference points is identified at step 3806. The set of objects may correspond to the controller's set of reference points, other sets of reference points of other controllers in the image, and / or one or more light sources projected in the image.

[0251] At step 3808, the hand tracking data is used to identify the position of the hand in the image. The hand may be detected in the image, and the hand tracking data may identify the position of the hand based on the detection. The hand tracking data may also be used to identify the orientation of the hand in the image.

[0252] At step 3810, the set of objects is associated with the set of fiducial points. In an example, a region of interest may be determined in the image based on the position and / or orientation of a hand in the image. A first subset of the set of objects outside the region of interest may be excluded, and a second subset of the set of objects within the region of interest may be associated with the set of fiducial points. The first subset and the second subset may be mutually exclusive. In another example, a region of interest may be determined based on the position and / or orientation of a hand in the image, a set of objects corresponding to the fiducial points in the region of interest in the image may be identified, and the set of objects in the region of interest may be associated with the set of fiducial points.

[0253] Although a wearable device (e.g., a head-mounted device in any of the preceding figures) may be calibrated at the factory using highly sophisticated instrumentation, during use, due to heat, use, and various forms of wear, the wearable device may deform, causing the factory calibration to become inaccurate. One possible solution is for the user to repeatedly take the wearable device back to the factory for recalibration. To avoid the obvious cost of this solution, some embodiments allow for accurate and robust runtime calibration while the wearable device is in use, eliminating the need for factory recalibration. An embodiment may predict the current calibration level of the wearable device, and may perform calibration based on the predicted calibration level.

[0254] During operation, the wearable device may use one or more parameters from the calibration profile to interpret the front-facing camera (e.g. Figure 3 The wearable device may also use a calibration profile to account for the spacing and orientation differences between the eyepieces (the left-front facing world camera 306A and the right-front facing world camera 306B) when generating the virtual image light, so that the user can comfortably view the virtual image elements in the correct alignment. To achieve this, the wearable device may repeatedly access the calibration profile to ensure that the parameters used reflect the latest and most accurate parameters available. In some cases, the wearable device may retrieve the parameters from the calibration profile immediately after performing the calibration process.

[0255] Fig.39An example diagram of a calibration profile 3900 for a wearable device is shown. In some embodiments, the calibration profile 3900 is maintained by the wearable device to model the physical spatial relationship between the left-facing camera and the right-facing camera 3906. The calibration profile 3900 may include a translation parameter T corresponding to the relative distance between the left-facing camera 3906A and the right-facing camera 3906B, and a rotation parameter R corresponding to the relative angular orientation between the left-facing camera 3906A and the right-facing camera 3906B. Each of the translation parameter T and the rotation parameter R may take a variety of data types. For example, the translation parameter T may be a single quantity (e.g., 0.1 meters), a one-dimensional matrix (e.g., [0.1; 0; 0] meters), a multi-dimensional matrix (e.g., [[0.1; 0; 0][0; 0; 0][0; 0; 0]] meters), an array, a vector, or any other possible representation of a single or multiple quantity. Similarly, the rotation parameter R can be a single quantity (e.g., 0.5 degrees), a one-dimensional matrix (e.g., [0.5; 0; 0] degrees), a multi-dimensional matrix (e.g., [[0.5; 0; 0][0; 0; 0][0; 0; 0]] degrees), an array, a vector, or any other possible representation of a single or multiple quantities.

[0256] The calibration profile 3900 can represent each forward-facing camera 3906 as occupying a single point using a pinhole camera model. The center point 3950 between the left-facing front camera 3906A and the right-facing front camera 3906B can be used to track the position of the wearable device relative to the world origin in the environment, and can also be used as a baseline for translation and rotation adjustments. In some embodiments, the relative distance between the left-facing front camera 3906A and the right-facing front camera 3906B and the center point 3950 can be equal to a translation parameter T, where the translation parameter T represents a 3×1 matrix corresponding to a three-dimensional (3D) vector (e.g., [0.1 0.2 0.1] meters). In some embodiments, the relative angular orientation between the left-facing front camera 3906A and the right-facing front camera 3906B and the center point 3950 can be equal to a rotation parameter R, where the rotation parameter R represents a 3×3 matrix. Thus, the transformation between the camera 3906B facing front right and the center point 3950 can be modeled by the transformation [T|R], while the transformation between the camera 3906A facing front left and the center point 3950 can be modeled by the transformation [T|R]. -1 Modeling.

[0257] The calibration level associated with the calibration profile 3900 can be periodically determined. Based on the calibration level, the wearable device can cause one of several types of calibration to occur. To determine the calibration level associated with the calibration profile 3900, a deformation amount D of the wearable device can be determined, where the deformation amount D is inversely proportional to the calibration level. Fig.40Various examples of deformation amounts are shown. In some cases, the calibration level is determined by identifying the same reference points in two images 3920 captured by the left-front facing camera 3906A and the right-front facing camera 3906B. After determining that both images 3920 include at least two identical reference points, epipolar lines 4052 can be generated based on one image (e.g., the left image 3920A) and can be projected onto the other image (e.g., the right image 3920B). The epipolar lines 4052 can be projected onto the right image 3920B using the latest version of the calibration profile 3900. The positional deviation of the reference points from the epipolar lines 4052 indicates that there is a calibration error between the left-front facing camera 3906A and the right-front facing camera 3906B. Therefore, the comparison between the reference points and the epipolar lines 4052 can be used to recalibrate the cameras.

[0258] As shown, when the detected reference point projected from the left image 3920A to the right image 3920B is substantially centered on the epipolar line 4052A (e.g., the deformation amount D is less than 20), the wearable device can be determined to have low deformation. When the detected reference point projected from the left image 3920A to the right image 3920B is slightly offset from the epipolar line 4052B (e.g., the deformation amount D is between 20 and 80), the wearable device can be determined to have medium deformation. If one of the detected reference points is substantially centered on the epipolar line 4052B, and the other detected reference point is not aligned with the epipolar line 4052B, the detected reference point may be slightly offset. When the detected reference point projected from the left image to the right image has a large displacement from the epipolar line 4052C (e.g., the deformation amount D is greater than 80), the wearable device can be determined to have a large deformation. If both detected reference points are not aligned with the epipolar line 4052C, the detected reference points may be significantly displaced.

[0259] Fig.41 A head mounted device 4102 or other wearable device is shown calibrating using an image 4120 containing reference points located on a controller 4101. The head mounted device 4102 may include stereo cameras 4106, including a camera 4106A facing forward to the left and a camera 4106B facing forward to the right. The head mounted device 4102 may be susceptible to deformation because the cameras 4106 are far apart and point triangulation is sensitive to camera extrinsic calibration. Given the location of features in the environment and their projected two-dimensional locations in the cameras, the extrinsic features of the stereo system can be recovered. The quality of the calibration may depend on camera intrinsic calibration, constellation extrinsic calibration, and constellation detection on the image. The head mounted device 4102 may be easily deformed because the cameras 4106 are far apart and point triangulation is sensitive to camera extrinsic calibration. Figure 39-40 Several image pairs in the method described in 4120 are used to improve the quality of the calibration.

[0260] In some examples, the controller 4101 can be calibrated with a head mounted device including a wearable stereo camera. In order to recover the position of the controller using a constellation of fiducials, it may be important to have a good calibration of the fiducials relative to the controller rig. Given a calibrated head mounted device and the 2D positions of the constellations projected on the head mounted device camera, each fiducial can be triangulated and the relative positions of each other can be recovered. The quality of the calibration may depend on the camera intrinsic calibration, the constellation extrinsic calibration, and the detection of constellations on the image. The calibration can be performed using Figure 39-40 The method described in

[23] was used to improve the quality of the calibration by using several image pairs.

[0261] Fig.42 An example method 4200 for head mounted device camera calibration using reference points according to some embodiments of the present invention is shown. One or more steps of the method 4200 may be performed in a different order than the illustrated embodiment, and one or more steps of the method 4200 may be omitted during the execution of the method 4200. In addition, two or more steps of the method 4200 may be performed simultaneously or concurrently.

[0262] At step 4202, a calibration profile is maintained that models a physical relationship between a first head mounted device camera and a second head mounted device camera. The first head mounted device camera may be a front left facing camera and the second head mounted device camera may be a front right facing camera. The calibration profile may include a translation parameter corresponding to a relative distance between the first head mounted device camera and the second head mounted device camera and a rotation parameter corresponding to a relative angular orientation between the first head mounted device camera and the second head mounted device camera.

[0263] In step 4204, the controller's set of reference points is flashed.

[0264] In step 4206, the first head mounted device camera is caused to capture a first image and the second head mounted device camera is caused to capture a second image.

[0265] At step 4208, a set of reference points are identified in the first image and the second image. After determining that both images include at least two identical reference points, epipolar lines can be generated based on one image (e.g., the first image in the first images) and the epipolar lines can be projected onto the other image (e.g., the second image in the second images).

[0266] At step 4210, a calibration level of the calibration profile is detected based on the set of fiducial points identified in the first image and the second image and the known geometry. The calibration level may be determined based on the deviation of the set of fiducial points from the epipolar lines in the second image. A higher deviation may correspond to a higher amount of deformation and a lower calibration level.

[0267] At step 4212, the calibration profile is modified based on the set of fiducial points identified in the first image and the second image and the known geometry. The calibration profile can be modified so that the set of fiducial points identified in the first image is aligned with the set of fiducial points identified in the second image.

[0268] Fig.43 4300 is shown in accordance with an embodiment described herein. Fig.43 The computer system 4300 shown may be incorporated into the devices described herein. Fig.43 A schematic diagram of an embodiment of a computer system 4300 is provided, which can perform some or all steps of the methods provided in various embodiments. It should be noted that Fig.43 It is intended only to provide a general description of the various components, any or all of which may be used as required. Fig.43 It is broadly described how various system elements may be implemented in a relatively discrete or relatively more integrated manner.

[0269] Computer system 4300 is shown to include hardware elements that may be electrically coupled via bus 4305 or may communicate in other ways (as appropriate). The hardware elements may include one or more processors 4310, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors, such as digital signal processing chips, graphics acceleration processors, etc.; one or more input devices 4315, which may include but are not limited to a mouse, keyboard, camera, etc.; and one or more output devices 4320, which may include but are not limited to a display device, a printer, etc.

[0270] The computer system 4300 may also include and / or communicate with one or more non-transitory storage devices 4325, which may include, but are not limited to, local and / or network accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices (e.g., random access memory ("RAM")), and / or read-only memory ("ROM"), which may be programmable, flash-updatable, and / or the like. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0271] The computer system 4300 may also include a communication subsystem 4319, which may include, but is not limited to, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset (e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication facility, etc.), and / or the like. The communication subsystem 4319 may include one or more input and / or output communication interfaces to allow data to be exchanged with a network (e.g., the network described below, as an example), other computer systems, a television, and / or any other device described herein. Depending on the desired functionality and / or other implementation considerations, a portable electronic device or similar device may transmit images and / or other information via the communication subsystem 4319. In other embodiments, a portable electronic device (e.g., a first electronic device) may be incorporated into the computer system 4300, for example, the electronic device as an input device 4315. In some embodiments, the computer system 4300 will also include a working memory 4335, which may include a RAM or ROM device, as described above.

[0272] The computer system 4300 may also include software elements, as shown, currently located in the working memory 4335, including an operating system 4340, device drivers, executable libraries and / or other code, such as one or more application programs 4345, which may include computer programs provided by various embodiments, and / or may be designed to implement methods, and / or configure systems provided by other embodiments, as described herein. By way of example only, one or more programs described with respect to the above methods may be implemented as codes and / or instructions executable by a computer and / or a processor within a computer; then, in one aspect, such codes and / or instructions may be used to configure and / or adjust a general purpose computer or other device to perform one or more operations according to the methods.

[0273] The set of these instructions and / or codes may be stored on a non-transitory computer-readable storage medium, such as the storage device 4325 described above. In some cases, the storage medium may be incorporated into a computer system, such as the computer system 4300. In other embodiments, the storage medium may be separate from the computer system, such as a removable medium, such as a compact disc, and / or provided in the form of an installation package, such that the storage medium may be used to program, configure, and / or adjust a general-purpose computer having the instructions / code stored therein. The instructions may be in the form of executable code, executable by the computer system 4300, and / or may be in the form of source code and / or installable code, which, when compiled and / or installed on the computer system 4300, such as using various commonly available compilers, installers, compression / decompression utilities, etc., then takes the form of executable code.

[0274] Those skilled in the art will appreciate that substantial changes may be made depending on specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets, etc.), or both. In addition, connections to other computing devices (such as network input / output devices) may be employed.

[0275] As described above, in one aspect, some embodiments may employ a computer system (e.g., computer system 4300) to perform methods according to various embodiments of the present technology. According to one set of embodiments, some or all of the procedures of these methods are performed by computer system 4300 in response to processor 4310 executing one or more sequences of one or more instructions, which may be incorporated into operating system 4340 and / or other code (e.g., application 4345) contained in working memory 4335. These instructions may be read into working memory 4335 from another computer-readable medium (e.g., one or more of storage devices 4325). By way of example only, execution of the sequence of instructions contained in working memory 4335 may cause processor 4310 to perform one or more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be performed by dedicated hardware.

[0276] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a particular manner. In an embodiment implemented using computer system 4300, various computer-readable media may participate in providing instructions / code to processor 4310 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, computer-readable media is a physical and / or tangible storage medium. Such media may take the form of non-volatile media or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 4325. Volatile media include, but are not limited to, dynamic memory, such as working memory 4335.

[0277] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette, or any other medium from which a computer can read instructions and / or code.

[0278] Various forms of computer readable media may be used to transmit one or more sequences of one or more instructions to the processor 4310 for execution. By way of example only, the instructions may initially be transmitted to a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over the transmission medium for receipt and / or execution by the computer system 4300.

[0279] The communication subsystem 4319 and / or its components typically receive the signal, and the bus 4305 may then transfer the signal and / or the data, instructions, etc. carried by the signal to the working memory 4335, from which the processor 4310 retrieves and executes the instructions. The instructions received by the working memory 4335 may optionally be stored on the non-temporary storage device 4325 before or after execution by the processor 4310.

[0280] The methods, systems, and devices discussed above are examples. Various configurations may omit, replace, or add various procedures or components as desired. For example, in alternative configurations, the methods may be performed in an order different from that described, and / or various stages may be added, omitted, and / or combined. Furthermore, features described for certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Furthermore, technology is constantly evolving, and therefore, many of the elements are examples and do not limit the scope of the present disclosure or claims.

[0281] Specific details are given in the specification so that the exemplary configurations, including the implementations, are fully understood. However, the configurations can be implemented without these specific details. For example, in order to avoid confusing the configurations, well-known circuits, processes, algorithms, structures, and techniques have been shown, but no unnecessary details are given. This specification only provides example configurations and does not limit the scope, applicability, or configuration of the claims. On the contrary, the foregoing description of the configurations will provide a favorable description of implementing the technology for those skilled in the art. Various changes may be made to the functions and arrangements of the elements without departing from the spirit or scope of the present disclosure.

[0282] In addition, the configuration can be described as a process depicted in a schematic flow chart or block diagram. Although each configuration can describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process may have additional steps not included in the figure. In addition, the example of the method can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segment performing the necessary tasks can be stored in a non-temporary computer readable medium (e.g., a storage medium). The processor can perform the described tasks.

[0283] Several example configurations have been described, and various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the present disclosure. For example, the above elements may be components of a larger system, in which other rules may take precedence over or otherwise modify the application of the technology. In addition, many steps may be taken before, during, or after considering the above elements. Therefore, the above description does not restrict the scope of the claims.

[0284] As used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a user" includes a plurality of such users and reference to "the processor" includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0285] In addition, when used in this specification and the following claims, the words "comprises", "comprising", "containing", "includes", "including" and "comprising" are intended to specify the presence of stated features, integers, components or steps, but they do not exclude the presence or addition of one or more other features, integers, components, steps, actions or groups.

[0286] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and that those skilled in the art may make various modifications or changes based thereon, and that these modifications or changes should be included within the spirit and scope of the present application and the scope of the appended claims.

Claims

1. A method of operating a wearable system having a head mounted device and a controller, the method comprising: alternately performing head mounted device tracking and performing controller tracking by repeatedly capturing images using a head mounted device camera of the head mounted device during head mounted device tracking frames and controller tracking frames, respectively; During each headset tracking frame: causing the head mounted device camera to capture a first exposure image of the image having an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration; During each of the controller tracking frames: causing the head mounted device camera to capture a second exposure image of the image having an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration; determining a reference interval defined by a reference start time and a reference end time, during which the reference point set of the controller will flash a plurality of times at a reference frequency and a reference period, wherein the reference interval is determined so that the second exposure interval at least partially overlaps with the reference interval; and The reference point set is caused to flash a plurality of times during the reference interval according to the reference frequency and the reference period.

2. The method according to claim 1, wherein: The wearable system comprises: The head mounted device comprises: the head mounted device camera; and A head mounted device inertial measurement unit; and The controller comprises: The set of reference points arranged in a known geometric shape; one or more controller cameras; and Controller Inertial Measurement Unit; Wherein, the wearable system is configured to determine the position or orientation of the head-mounted device or the controller based on data captured by the head-mounted device camera, the one or more controller cameras, the head-mounted device inertial measurement unit, or the controller inertial measurement unit.

3. The method according to claim 2, wherein: Operating the wearable system includes: determining a first pose of the head mounted device relative to a reference frame based on data captured by the head mounted device camera of the head mounted device or the head mounted device inertial measurement unit of the head mounted device; causing the reference point set of the controller to flash; A second posture of the controller relative to the head mounted device is determined by: capturing a head mounted device image using the head mounted device camera; identifying the set of reference points in the head mounted device image; Based on the set of reference points identified in the head mounted device image and the known geometric shape, a second pose of the controller relative to the head mounted device is determined.

4. The method according to claim 1, wherein: The reference interval is determined so that the second exposure interval is centered on the reference interval.

5. The method according to claim 1, wherein: A first time length of a head mounted device tracking frame in the head mounted device tracking frame is equal to a second time length of a controller tracking frame in the controller tracking frame.

6. The method according to claim 1, wherein: The first exposure duration comprises at least one millisecond.

7. A method of operating a wearable system including a head mounted device, the method comprising: capturing a set of images using a head mounted device camera of the head mounted device; identifying a plurality of fiducial points that flicker repeatedly in the image set; determining that at least some of the plurality of reference points include a first set of reference points belonging to a first controller and a second set of reference points belonging to a second controller different from the first controller; determining that glints of the first set of reference points are at least partially temporally aligned with glints of the second set of reference points; A period, frequency, or offset associated with at least one of the first set of reference points or the second set of reference points is modified such that flickering of the first set of reference points is not aligned with flickering of the second set of reference points.

8. The method according to claim 7, wherein: Such that the modification includes: The first controller is caused to modify a first period, a first frequency, or a first offset associated with the first set of reference points so that flickering of the first set of reference points is not aligned with flickering of the second set of reference points.

9. The method according to claim 8, wherein: The modification further includes: The second controller is caused to modify a second period, a second frequency, or a second offset associated with the second set of reference points to misalign the flickering of the first set of reference points with the flickering of the second set of reference points.

10. The method according to claim 7, wherein: Such that the modification includes: The first controller is caused to modify the flashing of the first set of reference points to be encoded using a first encoding.

11. The method according to claim 10, wherein: The modification further includes: The second controller is caused to modify the flashing of the second set of reference points to be encoded using a second encoding.

12. The method according to claim 7, further comprising: Controller tracing is performed on the first controller by: causing the head mounted device camera to capture an image in the set of images having an exposure below a threshold, wherein the image is associated with an exposure interval defined by an exposure start time, an exposure end time, and an exposure duration; determining a reference interval defined by a reference start time and a reference end time, during which the first reference point set of the first controller will flash a plurality of times at a reference frequency and a reference period, wherein the reference interval is determined so that the exposure interval at least partially overlaps with the reference interval; and The first reference point set is caused to flash a plurality of times according to the reference frequency and the reference period during the reference interval.

13. The method according to claim 7, wherein: The first controller includes the first set of reference points arranged in a known geometric shape, and the second controller includes the second set of reference points arranged in the known geometric shape, and wherein the method further comprises: causing a first subset of the first set of reference points to flash at the first controller; and A second subset of the second set of reference points is caused to flash at the second controller, wherein the first subset and the second subset are asymmetric to each other.

14. A method of operating a wearable system, the wearable system comprising a head mounted device and a controller having a display, the method comprising: causing the controller to display a set of reference points on the display according to the set of pixel positions; capturing a set of images using a head mounted device camera of the head mounted device; identifying the set of reference points in the set of images; as well as Based on a set of reference points identified in the set of images, a position and / or orientation of the controller relative to the head mounted device is determined.

15. The method according to claim 14, further comprising: The reference point set is caused to flash according to a period and a frequency.

16. The method according to claim 15, further comprising: The controller is caused to modify the period, the frequency, or the set of pixel positions.

17. The method according to claim 14, further comprising: identifying in the set of images a second set of reference points belonging to a second controller; as well as The controller is caused to modify the period, the frequency, or the set of pixel locations in response to identifying the second set of reference points.

18. The method according to claim 17, further comprising: identifying a first geometric shape of the second set of reference points; as well as The controller is caused to display the set of reference points in a second geometric shape different from the first geometric shape.

19. The method according to claim 14, further comprising: The display of the set of fiducial points is synchronized with at least one exposure interval of the head mounted device camera.

20. The method according to claim 19, wherein: The at least one exposure interval comprises a first exposure interval during a head mounted device tracking frame and a second exposure interval during a controller tracking frame, wherein the head mounted device tracking frame comprises: causing the head mounted device camera to capture a first exposure image in the set of images having an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration; and Wherein, the controller tracking frame includes: causing the head mounted device camera to capture a second exposure image in the set of images having a value below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration; and A reference interval defined by a reference start time and a reference end time is determined, during which the reference point set flashes a plurality of times at a reference frequency and a reference period, wherein the reference interval is determined such that the second exposure interval at least partially overlaps with the reference interval.

21. The method of claim 14, further comprising: The controller is caused to display one or more buttons configured to receive user input on the display according to the second set of pixel positions.

22. The method according to claim 14, wherein: The controller includes a mobile device.

23. A modular controller for a wearable system, the modular controller comprising one or more of a group of components, the group of components comprising: Visual Inertial Odometry (VIO) module; Constellation module; Main printed circuit board (PCB); Battery; Wireless communication engine; A user input includes at least one of: (i) a trigger, (ii) a bumper, or (iii) a touch pad; Haptic Engine; and / or User indicators; One or more of the set of components may be independently removed or added while maintaining at least some functionality of the modular controller.

24. The modular controller of claim 23, wherein: The modular controller is powered and communicates via a Universal Serial Bus (USB) connection.

25. The modular controller of claim 23, wherein: The minimum dimensions of the modular controller are 84 mm in length, 64 mm in width and 18 mm in thickness.

26. The modular controller of claim 23, wherein: The minimum dimensions of the modular controller are 64 mm in diameter and 18 mm in thickness.

27. The modular controller of claim 23, wherein: The minimum dimensions of the modular controller are 50 mm in diameter and 15 mm in thickness.

28. The modular controller of claim 23, wherein: The modular controller is integrated into a drone that is controllable by an application on the wearable system, and wherein the wearable system is configured to identify a set of reference points of the constellation module to locate the drone.

29. A method of operating a wearable system having a head mounted device and a controller, the method comprising: causing a set of reference points of the controller to flash, the set of reference points being arranged in a known geometric shape and comprising a plurality of groups of reference points that are rotationally symmetric relative to each other, wherein the number of each group of the plurality of groups of reference points is equal to a predetermined number, wherein the predetermined number is at least three; causing a head mounted device camera of the head mounted device to capture an image; identifying a set of objects in the image corresponding to a reference point; The set of objects is associated with the set of reference points based on known geometric shapes by: repeatedly selecting subsets of the set of objects, wherein the number of each subset is equal to the predetermined number, and calculating a pose of the controller by associating the subsets with the plurality of sets of reference points; Based on the compatibility of the poses of the plurality of sets of reference points, calculating statistics of associated subsets; and Based on the calculated statistics, a correct association between the set of objects and the set of reference points is found.

30. The method of claim 29, wherein: Calculating the posture includes: The subset of the set of objects is input into a perspective three-point algorithm configured to output the pose of the controller.

31. The method of claim 29, wherein: The known geometric shape includes a first gap between a first pair of adjacent reference points in the set of reference points, the first gap being larger than other gaps between other pairs of adjacent reference points in the set of reference points.

32. The method of claim 29, wherein: The set of objects corresponds to the set of reference points and / or one or more light sources projected in the image.

33. The method of claim 29, wherein: Calculating the statistics of the associated subset comprises: A number of the set of objects aligned with the set of reference points is determined for each of the poses.

34. The method of claim 29, further comprising: identifying a second set of reference points in the image belonging to a second controller; as well as In response to identifying the second set of reference points, the controller is caused to modify a period, frequency, or offset of flashing of the set of reference points to misalign the set of reference points with the second set of reference points.

35. A method of operating a wearable system having a head mounted device and a controller, the method comprising: causing a set of reference points of the controller to flash, the set of reference points being arranged according to a known geometric shape; causing a head mounted device camera of the head mounted device to capture an image; identifying a set of objects corresponding to reference points in the image; capturing rotational measurements using a controller inertial measurement unit of the controller; as well as Associating the set of objects with the set of reference points based on the known geometric shape by: repeatedly selecting a subset of the set of objects and computing a pose of the controller by associating the subset with a plurality of sets of reference points; calculating statistics of associated subsets based on compatibility of the poses of the plurality of sets of reference points and based on the rotation measurements; as well as Based on the calculated statistics, a correct association between the set of objects and the set of reference points is found.

36. The method of claim 35, wherein: Calculating the posture includes: The subset of the set of objects is input into a perspective two-point algorithm configured to output the pose of the controller.

37. The method of claim 36, wherein: The perspective two-point algorithm includes a gravity-based perspective two-point algorithm.

38. The method of claim 35, wherein: The set of objects corresponds to the set of reference points and / or one or more light sources projected in the image.

39. The method of claim 35, wherein: Calculating the statistics of the associated subset includes: A number of the set of objects aligned with the set of reference points is determined for each of the poses.

40. The method of claim 35, further comprising: identifying in the image a second set of reference points belonging to a second controller; as well as In response to identifying the second set of reference points, the controller is caused to modify a period, frequency, or offset of flashing of the set of reference points to misalign the set of reference points with the second set of reference points.

41. The method of claim 35, wherein: The wearable system comprises: A head mounted device comprising: the head mounted device camera; and A head mounted device inertial measurement unit; and A controller comprising: the set of reference points arranged in the known geometric shape; one or more controller cameras; and the controller inertial measurement unit; Wherein, the wearable system is configured to determine the position or orientation of the head-mounted device or the controller based on data captured by the head-mounted device camera, the one or more controller cameras, the head-mounted device inertial measurement unit, or the controller inertial measurement unit.

42. A method of operating a wearable system having a head mounted device and a controller, the method comprising: causing a set of reference points of the controller to flash, the set of reference points being arranged in a known geometric shape; causing a head mounted device camera of the head mounted device to capture an image; identifying a set of objects corresponding to reference points in the image; using the hand tracking data to identify a position of a hand in the image; as well as The object set is associated with the reference point set by: determining a region of interest in the image based on the position of the hand in the image; excluding a first subset of the set of objects outside the region of interest; as well as A second subset of the set of objects within the region of interest is associated with the set of reference points, wherein the first subset and the second subset are mutually exclusive.

43. The method of claim 42, further comprising: using the hand tracking data to identify an orientation of the hand in the image; as well as Based on the orientation of the hand in the image, the region of interest is determined.

44. The method of claim 43, wherein: Determining the region of interest based on the orientation of the hand in the image includes tilting the region of interest in the direction in which the controller is held according to the orientation.

45. A method of operating a wearable system having a head mounted device and a controller, the method comprising: causing a set of reference points of the controller to flash, the set of reference points being arranged in a known geometric shape; causing a head mounted device camera of the head mounted device to capture an image; using the hand tracking data to identify a position of a hand in the image; Determining a region of interest in the image based on a position of the hand in the image; identifying a set of objects corresponding to reference points in the region of interest in the image; The set of objects in the region of interest are associated with the set of reference points.

46. ​​The method of claim 45, further comprising: using the hand tracking data to identify an orientation of the hand in the image; as well as Based on the orientation of the hand in the image, the region of interest is determined.

47. The method of claim 45, wherein: Determining the region of interest based on the orientation of the hand in the image includes tilting the region of interest in the direction in which the controller is held according to the orientation.

48. A method of operating a wearable system having a head mounted device and a controller, the method comprising: maintaining a calibration profile that models a physical relationship between a first head mounted device camera of the head mounted device and a second head mounted device camera of the head mounted device; causing a set of reference points of the controller to flash, the set of reference points being arranged in a known geometric shape; causing the first head mounted device camera to capture a first image and causing the second head mounted device camera to capture a second image; identifying the set of reference points in the first image and the second image; as well as Do one or both of the following: detecting a calibration level of the calibration profile based on a set of fiducial points identified in the first image and the second image and based on the known geometric shape; or The calibration profile is modified based on the set of fiducial points identified in the first image and the second image and based on the known geometry.

49. The method of claim 48, wherein: The calibration profile includes a translation parameter corresponding to a relative distance between the first head mounted device camera and the second head mounted device camera.

50. The method of claim 49, wherein: The calibration profile also includes rotation parameters corresponding to a relative angular orientation between the first head mounted device camera and the second head mounted device camera.

51. The method of claim 50, wherein: Each of the translation parameter and the rotation parameter comprises: a single quantity, a one-dimensional matrix, a multi-dimensional matrix, an array, or a vector.

52. The method of claim 49, further comprising: Determining a center point between the first head mounted device camera and the second head mounted device camera, wherein a first distance between the first head mounted device camera and the center point and a second distance between the second head mounted device camera and the center point are equal to the translation parameter.

53. The method of claim 48, wherein: Determining the calibration level includes: generating epipolar lines based on the first image; projecting the epipolar lines onto the second image using the calibration configuration file; and The calibration level is determined based on a deviation of the set of reference points from the epipolar line in the second image.

54. The method of claim 53, wherein: The deviation corresponds to a calibration error between the first head mounted device camera and the second head mounted device camera, and wherein the method further comprises: Based on the deviation, the first head mounted device camera and / or the second head mounted device camera are adjusted.

Images