Visual tracking of peripheral devices

Abstract

The invention relates to visual tracking of peripheral devices. Techniques for performing positioning of a handheld device relative to a wearable device are disclosed. At least one sensor mounted on the handheld device, such as an inertial measurement unit (IMU), may obtain handheld data indicative of movement of the handheld device relative to the world. An imaging device mounted on a handheld device or wearable device may capture a fiducial image containing several fiducials secured to another device. The number of reference points included in the image is determined. At least one of a position and an orientation of the handheld device relative to the wearable device is updated based on the image and the handheld data according to the first operating state, the second operating state, or the third operating state based on the number of reference points.

Description

[0001] This application is a divisional application of an application filed on March 6, 2019, with application number 202110689396.5 and invention name “Visual tracking of peripheral devices”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 640,009, filed on March 7, 2018, entitled “VISUAL TRACKING OF PERIPHERAL DEVICES,” and U.S. Provisional Patent Application No. 62 / 640,299, filed on March 7, 2018, entitled “VISUAL TRACKING OF PERIPHERAL DEVICES,” the entire disclosures of which are hereby incorporated by reference into this document for all purposes as if fully set forth herein. Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality or "VR" scenarios generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0005] Despite the advances that have been made in these display technologies, there remains a need in the art for improved methods, systems, and apparatus related to augmented reality systems, and particularly display systems. Summary of the Invention

[0006] The present invention generally relates to the positioning (position, orientation, and / or distance) of peripheral devices. More specifically, embodiments of the present invention provide systems, devices, and methods for positioning a handheld device relative to a wearable device. Although some portions of the present invention are described with reference to augmented reality (AR systems), the present disclosure is applicable to a variety of applications.

[0007] According to a first aspect of the present invention, a method for performing positioning of a handheld device relative to a wearable device is provided. The method includes obtaining handheld data indicating the movement of the handheld device relative to the world by at least one sensor mounted on the handheld device. In some embodiments, obtaining the handheld data includes detecting the linear acceleration and rotational speed of the handheld device by an inertial measurement unit (IMU) mounted on the handheld device. In some embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device by a handheld camera mounted on the handheld device. The method may further include obtaining fiducial data indicating the movement of the handheld device relative to the wearable device by a wearable camera mounted on the wearable device. In some embodiments, obtaining the fiducial data includes capturing a fiducial image including a plurality of light emitting diodes (LEDs) by the wearable camera, the plurality of LEDs being affixed to a handheld device having a plurality of LEDs affixed to the handheld device.

[0008] The method may further include determining the number of LEDs included in the reference point image. The method may further include, in response to determining that the number of LEDs is equal to or greater than 3, updating the position and orientation of the handheld device relative to the wearable device based solely on the reference data according to a first operating state. The method may further include, in response to determining that the number of LEDs is equal to 1 or 2, updating the position and orientation of the handheld device relative to the wearable device based on the reference data and the handheld data according to a second operating state. The method may further include, in response to determining that the number of LEDs is equal to 0, updating the position and orientation of the handheld device relative to the wearable device based solely on the handheld data according to a third operating state.

[0009] According to a second aspect of the present invention, a method for performing positioning of a handheld device relative to a wearable device. The method may include obtaining handheld data indicating the movement of the handheld device relative to the world via at least one sensor mounted on the handheld device. The method may also include obtaining reference data indicating the movement of the handheld device relative to the wearable device via an imaging device mounted on a first device. In some embodiments, the first device is the handheld device or the wearable device. In some embodiments, obtaining the reference data includes capturing, via the imaging device, a reference point image comprising a plurality of fiducials fixed to a second device different from the first device. In some embodiments, the second device is the handheld device or the wearable device. The method may further include determining a number of fiducials contained in the reference point image. The method may further include updating the position and orientation of the handheld device relative to the wearable device based on the reference data and the handheld data according to a first operating state or a second operating state based on the number of fiducials contained in the reference point image.

[0010] In some embodiments, obtaining the handheld data includes detecting rotational movement of the handheld device via an IMU mounted on the handheld device. In some embodiments, the imaging device is mounted on the handheld device, and a plurality of fiducials including the plurality of fiducials are affixed to the wearable device. In some embodiments, the imaging device is mounted on the wearable device, and a plurality of fiducials including the plurality of fiducials are affixed to the handheld device. In some embodiments, the imaging device is mounted on the handheld device, and a plurality of fiducials including the plurality of fiducials are affixed to the wearable device. In some embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device via a second handheld imaging device mounted on the handheld device. In some embodiments, the imaging device is mounted on the wearable device, and a single fiducial including the plurality of fiducials is affixed to the handheld device. In such embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device via a second handheld imaging device mounted on the handheld device.

[0011] In some embodiments, the imaging device is mounted on the wearable device, and a plurality of fiducials including the plurality of fiducials are affixed to the handheld device. In such embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device using a second handheld imaging device mounted on the handheld device. The method may further include, in response to determining that the number of fiducials is equal to or greater than three, updating the position and orientation of the handheld device relative to the wearable device based on the fiducial data according to a first operating state. The method may further include, in response to determining that the number of fiducials is equal to one or two, updating the position and orientation of the handheld device relative to the wearable device based on the fiducial data and the handheld data according to a second operating state. The method may further include, in response to determining that the number of fiducials is equal to zero, updating the position and orientation of the handheld device relative to the wearable device based on the handheld data according to a third operating state. In some embodiments, the position and orientation of the handheld device relative to the wearable device is updated based solely on the fiducial data according to the first operating state. In some embodiments, the position and orientation of the handheld device relative to the wearable device is updated based solely on the handheld data according to the third operating state.

[0012] According to a third aspect of the present invention, a system for performing positioning of a handheld device relative to a wearable device is provided. The system may include the wearable device. The system may also include the handheld device. The system may further include one or more processors communicatively coupled to the wearable device and the handheld device. In some embodiments, the one or more processors are configured to perform operations including: obtaining, via at least one sensor mounted on the handheld device, handheld data indicating movement of the handheld device relative to the world. The operations may also include obtaining, via an imaging device mounted on a first device, reference data indicating movement of the handheld device relative to the wearable device. In some embodiments, the first device is the handheld device or the wearable device. In some embodiments, obtaining the reference data includes capturing, via the imaging device, a reference point image including a plurality of reference points fixed to a second device different from the first device. In some embodiments, the second device is the handheld device or the wearable device. The operations may further include determining a number of reference points included in the reference point image. The operations may further include updating, based on the number of reference points included in the reference point image, the position and orientation of the handheld device relative to the wearable device based on the reference data and the handheld data, in accordance with a first operating state or a second operating state.

[0013] In some embodiments, obtaining the handheld data includes detecting rotational movement of the handheld device via an IMU mounted on the handheld device. In some embodiments, the imaging device is mounted on the handheld device, and a plurality of fiducials, including the plurality of fiducials, are affixed to the wearable device. In some embodiments, the imaging device is mounted on the wearable device, and a plurality of fiducials, including the plurality of fiducials, are affixed to the handheld device. In some embodiments, the imaging device is mounted on the handheld device, and a plurality of fiducials, including the plurality of fiducials, are affixed to the wearable device. In such embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device via a second handheld imaging device mounted on the handheld device. In some embodiments, the imaging device is mounted on the wearable device, and a single fiducial, including the plurality of fiducials, is affixed to the handheld device. In such embodiments, obtaining the handheld data includes capturing an image of the world including one or more features located around the handheld device via a second handheld imaging device mounted on the handheld device. In some embodiments, the imaging device is mounted on the wearable device, and a plurality of fiducials, including the plurality of fiducials, are affixed to the handheld device. In such embodiments, obtaining the handheld data includes capturing, via a second handheld imaging device mounted on the handheld device, an image of the world including one or more features located around the handheld device.

[0014] In some embodiments, the operations further include, in response to determining that the number of reference points is equal to or greater than 3, updating the position and orientation of the handheld device relative to the wearable device based on the reference data according to a first operating state. In some embodiments, the operations further include, in response to determining that the number of reference points is equal to 1 or 2, updating the position and orientation of the handheld device relative to the wearable device based on the reference data and the handheld data according to a second operating state. In some embodiments, the operations further include, in response to determining that the number of reference points is equal to 0, updating the position and orientation of the handheld device relative to the wearable device based on the handheld data according to a third operating state.

[0015] The present invention provides a variety of benefits over conventional techniques. For example, embodiments of the present invention provide more accurate positioning of a handheld device than conventional techniques such as electromagnetic tracking systems employing a series of magnetic coils. Embodiments of the present invention can also utilize hardware already employed in AR systems, such as forward-facing or side-facing world cameras on a headset. Embodiments can extend beyond AR systems and into any application where the positioning of one device relative to another is important. Other benefits of the present invention will be apparent to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An augmented reality (AR) scene is shown viewed through a wearable AR device according to embodiments described herein.

[0017] Figure 2 Various possible components of an AR system are shown.

[0018] Figure 3 An example showing how a visual tracking system can be incorporated into an AR system with wearable and handheld devices is shown.

[0019] Figure 4 A diagram showing the localization task.

[0020] Figure 5 An example configuration of an AR system is shown.

[0021] Figure 6 Shows the use Figure 5 The example configuration shown performs the positioning method.

[0022] Figure 7 An example configuration of an AR system is shown.

[0023] Figure 8 Shows the use Figure 7 The example configuration shown performs the positioning method.

[0024] Figure 9 An example configuration of an AR system is shown.

[0025] Figure 10 Shows the use Figure 9 The example configuration shown performs the positioning method.

[0026] Figure 11A An example configuration of an AR system is shown.

[0027] Figure 11B An example configuration of an AR system is shown.

[0028] Figure 12 Shows the use Figure 11A and11B The example configuration shown performs the positioning method.

[0029] Figure 13 A method of performing positioning using any of the previously shown example configurations is shown.

[0030] Figure 14 A simplified computer system is shown according to some embodiments described herein. DETAILED DESCRIPTION

[0031] In traditional virtual reality (VR) or augmented reality (AR) systems, six degrees of freedom tracking of peripheral devices is achieved by including a series of electromagnetic sensors and transmitters strategically placed on the user's AR headset, waist pack and / or other auxiliary devices (e.g., totems, haptic devices, gaming tools, etc.). Typically, an electromagnetic tracking system includes at least one electromagnetic field transmitter and at least one electromagnetic field sensor. Since the emitted electromagnetic field has a known distribution, the detected field can be analyzed to determine the position and / or orientation of the peripheral device. Although such a system provides a simple solution to the positioning problem, there is a need for additional solutions that provide higher precision positioning. Embodiments of the present invention can replace or supplement electromagnetic tracking systems.

[0032] Embodiments of the present invention provide a visual tracking system for performing high-precision positioning of a handheld device (e.g., a totem) relative to a wearable device (e.g., a head-mounted device). An imaging device is mounted on one of the devices and can capture images of one or more reference points fixed to the other device. Additional imaging devices can be mounted on the handheld device to capture various environmental markers. Based on the number of reference points in the captured image, different data processing schemes can be implemented that utilize reference data (i.e., data with a local reference based on the reference point image) and handheld data (data with a world reference collected from sensors mounted on the handheld device) differently. Each data processing scheme, referred to herein as an operating state, enables accurate estimation of the position and / or orientation of the handheld device relative to the wearable device. The tracking system can inform the AR system of the estimated positioning, and the AR system can use the positioning information to generate virtual content that makes the user feel comfortable.

[0033] Figure 11 shows an AR scene viewed through a wearable AR device according to an embodiment described herein. The AR scene 100 is depicted in which a user of the AR technology sees a real-world park-like setting 106 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that they "see" a robotic statue 110 standing on the real-world platform 120, and a cartoon-like avatar character 102 flying by that appears to be an avatar of a bumblebee, even though these elements (character 102 and statue 110) do not exist in the real world. Because human visual perception and nervous systems are extremely complex, it is challenging to create VR or AR technology that promotes comfortable, natural-feeling, rich presentation of virtual image elements in addition to other virtual or real-world image elements.

[0034] Figure 2 Various possible components of an AR system are shown. In the illustrated embodiment, an AR system user 260 is shown wearing a head-mounted assembly 258, which features a frame 264 structure coupled to a display system 262 located in front of the user's eyes. A speaker 266 is coupled to the frame 264 in the illustrated configuration and positioned near the user's ear canal (in one embodiment, another speaker, not shown, is positioned near the user's other ear canal to provide stereo / shapeable sound control). The display 262 is operably coupled (as shown at 268) to a local processing and data module 270, such as by a wired lead or a wireless connection. The local processing and data module 270 can be mounted in various configurations, such as fixedly attached to the frame 264, fixedly attached to a helmet or hat, removably attached to the torso of the user 260 in a backpack-type configuration, or removably attached to the hip of the user 260 in a belt-coupled configuration.

[0035] The local processing and data module 270 may include a power-efficient processor or controller, and digital memory (such as flash memory), both of which may be used to assist in processing, caching, and storing the following data: a) data captured from sensors operably coupled to the frame 264, such as, for example, image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes; and / or b) data acquired and / or processed using the remote processing module 272 and / or remote data repository 274, which data may be transmitted to the display 262 after such processing or retrieval.

[0036] The local processing and data module 270 is operatively coupled (as shown at 276, 278) to a remote processing module 272 and a remote data repository 274, for example, via wired or wireless communication links, such that these remote modules 272, 274 are operatively coupled to each other and available as resources to the local processing and data module 270. In one embodiment, the remote processing module 272 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 274 may include a relatively large-scale digital data storage facility accessible via the Internet or other network configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all computations are performed in the local processing and data module, allowing for full autonomous use from any remote module.

[0037] Figure 3 An example of how a visual tracking system can be incorporated into an AR system having a wearable device 302 (e.g., a head-mounted device) and a handheld device 304 (e.g., a controller) is shown. In some embodiments, the handheld device 304 can be a handheld controller that allows a user to provide input to the AR system. For example, the handheld device 304 can be a totem used in a gaming scene. The handheld device 304 can be a tactile device and can include one or more tactile surfaces that utilize various sensor types. During operation of the AR system, the user can hold the handheld device 304 in his / her left or right hand by actively grasping the handheld device 304 and / or by securing it to the user's hand with an attachment device (e.g., a wrap).

[0038] The handheld device 304 may include one or more fiducials (referred to herein as handheld fiducials 322) positioned along one or more exterior surfaces of the handheld device 304 such that the fiducials may be located within the field of view of an imaging device on the exterior of the handheld device. The handheld fiducials 322 may have a known relationship relative to one another such that an imaging device may determine its position and / or orientation relative to the handheld device 304 by capturing an image of one or more handheld fiducials 322. The handheld fiducials 322 may be dynamic, static, powered, unpowered, and in some embodiments, may be distinguishable from one another. For example, a first fiducial may be a light emitting diode (LED) having a first wavelength and a second fiducial may be an LED having a second wavelength. Alternatively or additionally, different fiducials may have different brightness and / or may pulse at different frequencies (e.g., a first fiducial may pulse at 100 Hz and a second fiducial may pulse at 150 Hz).

[0039] The handheld device 304 may include one or more imaging devices (referred to herein as handheld imaging devices 326) positioned so that when the handheld device 304 is held by the user, a feature surrounding the handheld device 304 and / or the wearable device 302 is within the field of view of the imaging device. For example, the front handheld imaging device 326A may be positioned so that its field of view is oriented away from the user toward one or more features in the surrounding environment of the handheld device 304, and the rear handheld imaging device 326B may be positioned so that its field of view is oriented toward the wearable device 302. The handheld imaging devices 326 may include one or more forward-facing imaging devices and / or one or more rear-facing imaging devices to produce the desired cumulative field of view. In some embodiments, the handheld imaging device 326 may be an optical device such as a camera and may capture still or moving images.

[0040] The handheld device 304 may include an inertial measurement unit (IMU) (referred to herein as a handheld IMU 324) that is rigidly fixed within the handheld device 304 such that rotational and linear motion of the handheld device 304 is similarly experienced by the handheld IMU 324. In some cases, the handheld IMU 324 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, the handheld IMU 324 may include an accelerometer, a gyroscope, and a magnetometer for each of three axes. For each axis, the handheld IMU 324 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, the handheld IMU 324 may output raw data from which any of the above-described forms of processed data may be calculated.

[0041] The handheld device 304 may include a rechargeable and / or replaceable battery 328 or other power source to power the handheld fiducial 322, the handheld imaging device 326, the handheld IMU 324, and any other components of the handheld device 304. Figure 3 Although not shown, the handheld device 304 may include circuitry for enabling wireless communication with the wearable device 302 and / or the belt pack 340. For example, when detecting or capturing data using the handheld imaging device 326 and the handheld IMU 324, the handheld device 304 may transmit raw or processed data to the wearable device 302 and / or the belt pack 340.

[0042] The wearable device 302 may include one or more fiducials (referred to herein as wearable fiducials 306) positioned along one or more exterior surfaces of the wearable device 302 such that the fiducials may be within the field of view of the rear handheld imaging device 326B. The wearable fiducials 306 may have a known relationship relative to one another such that the imaging device may determine its position and / or orientation relative to the wearable device 302 by capturing an image of one or more wearable fiducials 306. In some embodiments, the wearable fiducials 306 may be dynamic, static, powered, unpowered, and distinguishable from one another. For example, a first fiducial may be an LED having a first wavelength and a second fiducial may be an LED having a second wavelength. Alternatively or additionally, different fiducials may have different brightness and / or may pulse at different frequencies.

[0043] Wearable device 302 may include one or more imaging devices (referred to herein as wearable imaging devices 310) positioned such that when a user holds handheld device 304, handheld device 304 (particularly handheld fiducials 322) is within the imaging device's field of view. For example, one or more wearable imaging devices 310 may be positioned on wearable device 302 facing forward, above, below, and / or to the side of an optically see-through component of wearable device 302. In one embodiment, two wearable imaging devices 310 may be positioned on opposite sides of an optically see-through component of wearable device 302. In some embodiments, wearable imaging device 310 may be an optical device such as a camera and may capture still or moving images.

[0044] The wearable device 302 may include an IMU (referred to herein as the wearable IMU 308) that is rigidly fixed within the wearable device 302 so that rotational and linear motion of the wearable device 302 is similarly experienced by the wearable IMU 308. In some cases, the wearable IMU 308 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software to convert raw measurements into processed data. For example, the wearable IMU 308 may include an accelerometer, a gyroscope, and a magnetometer for each of three axes. For each axis, the wearable IMU 308 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively or additionally, the wearable IMU 308 may output raw data from which any of the above-described forms of processed data may be calculated.

[0045] In some embodiments, the AR system may include a waist pack 340, which may include computing means (e.g., one or more processors and associated memory) for performing positioning of the handheld device 304 relative to the wearable device 302. Alternatively or additionally, the computing means may reside in the wearable device 302 itself, or even in the handheld device 304. The computing means may receive raw or processed data (via wired and / or wireless connections) from each of the wearable IMU 308, the wearable imaging device 310, the handheld IMU 324, and the handheld imaging device 326, and may calculate the geospatial position of the handheld device 304 (relative to the geospatial position of the wearable device 302) and the orientation of the handheld device 304 (relative to the orientation of the wearable device 302). In one or more embodiments, the computing means may also include a mapping database 342 (e.g., a transferable world model, a coordinate space, etc.) to detect gestures, determine coordinates of real and virtual objects, and may even connect to cloud resources and a transferable world model. In some embodiments, images captured using the wearable imaging device 310 and / or the handheld imaging device 326 may be used to construct a transferable world model. For example, features may be detected in the captured images, and a transferable world model or map of the environment may be constructed using the collected data (e.g., sparse points).

[0046] Figure 4 A diagram illustrating a positioning task performed by an AR system is shown, wherein the position and orientation of a handheld device 304 relative to a wearable device 302 is determined. In the diagram shown, the wearable device 302 has a geospatial position ("wearable position") and an orientation ("wearable orientation"), the geospatial position being defined relative to a world reference as (X WP , Y WP , Z WP ), the orientation is defined relative to the world reference as (X WO , Y WO , Z WO In some cases, the geospatial location of wearable device 302 is expressed in longitude, latitude, and elevation values, and the orientation of wearable device 302 is expressed in pitch, yaw, and roll values.

[0047] As shown, the handheld device 304 has a geospatial location (X) relative to the wearable device 302. WP , Y WP , Z WP ) is defined as (X' HP , Y′ HP , Z′ HP ) and the orientation (X WO , Y WO, Z WO ) is defined as (X' HO , Y′ HO , Z′ HO ) ("handheld orientation"). In some cases, the geospatial position of handheld device 304 is expressed in X, Y, and Z Cartesian values, and the orientation of handheld device 304 is expressed in pitch, yaw, and roll values. As a specific example, when the user is holding handheld device 304, the geospatial position of handheld device 304 may be equal to (0.7m, -0.5m, 0.1m), and the orientation of handheld device 304 may be equal to (10.2°, -46.2°, 15.2°).

[0048] Figure 5 An example configuration of an AR system 500 is shown in which a wearable device 302 includes one or more wearable fiducials 306, and a handheld device 304 includes one or more rear-facing handheld imaging devices 326, the field of view of which at least partially and at least temporarily includes the wearable fiducials 306 when the user holds the handheld device 304 in normal operation. The AR system 500 can include additional sensors mounted on the handheld device 304, such as a handheld IMU 324. One advantage of this configuration can be that the handheld device 304 has all the data required to perform its own positioning relative to the wearable device 302, thereby reducing the processing load on the wearable device 302. The AR system 500 can include additional sensors mounted on the wearable device 302, such as a wearable IMU 308.

[0049] Figure 6 A method 600 is shown for performing positioning of the handheld device 304 relative to the wearable device 302 using the AR system 500. One or more steps of the method 600 may be omitted or performed in a different order than the illustrated embodiment, and one or more steps of the method 600 may be performed at one or more processing devices located within the wearable device 302, the handheld device 304, and / or the waist pack 340.

[0050] At step 602, an image ("fiducial image") is captured by handheld imaging device 326. The fiducial image may contain several of the wearable fiducials 306. For example, if there are three wearable fiducials 306, the fiducial image may be analyzed to determine whether it contains zero, one, two, or three fiducials.

[0051] At step 604, the position and / or orientation of handheld device 304 relative to wearable device 302 is calculated, for example, based on the fiducial image. For example, the fiducial image can be analyzed to determine the location of any fiducials in wearable fiducials 306, and the position and / or orientation can be determined based on the positions of the fiducials within the fiducial image and the known physical relationships between the wearable fiducials 306. The position and / or orientation of handheld device 304 can be used to determine a posture of handheld device 304 relative to wearable device 302. The output of step 604 is referred to as fiducial data 630.

[0052] At step 610, data indicating at least rotational motion of the handheld device 304 relative to the world (and / or relative to the wearable device 302) ("IMU data") is detected by the handheld IMU 324. The IMU data may include rotational velocity or raw data from which rotational velocity is calculated. In some embodiments, the IMU data also indicates linear motion of the handheld device 304 and may include linear acceleration or raw data from which linear acceleration is calculated.

[0053] At step 612, the position and / or orientation of the handheld device 304 is calculated based on the IMU data. In some embodiments, the position and / or orientation of the handheld device 304 is calculated relative to the world (using a previously known and / or estimated orientation relative to the world), and / or in some other embodiments, the position and / or orientation of the handheld device 304 is calculated relative to the wearable device 302 (using a previously known and / or estimated orientation relative to the wearable device 302).

[0054] At step 614, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated based on the fiducial data 630 and / or the handheld data 632. The fiducial data 630 may include the fiducial image and / or the position and / or orientation calculations based on the fiducial image performed in step 604. The handheld data 632 may include IMU data and / or the position and / or orientation calculations based on the IMU data performed in step 612. The position and orientation calculations at step 614 may be performed according to one of various operating states based on the number of fiducials found in the fiducial image. Each operating state may process the fiducial data 630 and the handheld data 632 differently and may emphasize one type of data more than the other type of data. Reference will be made to Figure 13 These operating states are described in more detail.

[0055] At step 616, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is output, for example, to an external device and / or a process for operating the AR system 500. For example, the position and / or orientation may be output to the AR system 500 to generate and display virtual content.

[0056] Figure 7 An example configuration of an AR system 700 is shown, in which a wearable device 302 includes one or more wearable imaging devices 310, and when a user holds a handheld device 304 in normal operation, the field of view of the wearable imaging device 310 at least partially and at least temporarily includes a handheld fiducial 322, and the handheld device 304 includes one or more handheld fiducials 322. The AR system 700 can include additional sensors mounted on the handheld device 304, such as a handheld IMU 324. One advantage of this configuration can be the simplicity and low power consumption of the handheld device 304. The AR system 700 can include additional sensors mounted on the wearable device 302, such as a wearable IMU 308.

[0057] Figure 8 A method 800 is shown for performing positioning of the handheld device 304 relative to the wearable device 302 using the AR system 700. One or more steps of the method 800 may be omitted or performed in a different order than the illustrated embodiment, and one or more steps of the method 800 may be performed at one or more processing devices located within the wearable device 302, the handheld device 304, and / or the waist pack 340.

[0058] At step 802, an image ("fiducial image") is captured by wearable imaging device 310. The fiducial image may contain several of handheld fiducials 322. For example, if there are three handheld fiducials 322, the fiducial image may be analyzed to determine whether it contains zero, one, two, or three fiducials.

[0059] At step 804, the position and / or orientation of handheld device 304 relative to wearable device 302 is calculated, for example, based on the fiducial image. For example, the fiducial image can be analyzed to determine the position of any fiducials in handheld fiducials 322, and the position and / or orientation can be determined based on the positions of the fiducials within the fiducial image and the known physical relationship between handheld fiducials 322. The position and / or orientation of handheld device 304 can be used to determine a posture of handheld device 304 relative to wearable device 302. The output of step 804 is referred to as fiducial data 830.

[0060] At step 810, data indicating at least rotational motion of the handheld device 304 relative to the world ("IMU data") is detected by the handheld IMU 324. The IMU data may include rotational velocity or raw data from which rotational velocity is calculated. In some embodiments, the IMU data also indicates linear motion of the handheld device 304 and may include linear acceleration or raw data from which linear acceleration is calculated.

[0061] At step 812, the position and / or orientation of the handheld device 304 is calculated based on the IMU data. In some embodiments, the position and / or orientation of the handheld device 304 relative to the world is calculated (using a previously known and / or estimated orientation relative to the world), and / or in some embodiments, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated (using a previously known and / or estimated orientation relative to the wearable device 302). The output of step 812 can be referred to as handheld data 832.

[0062] At step 814, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated based on the fiducial data 830 and / or the handheld data 832. The fiducial data 830 may include the fiducial image and / or the position and / or orientation calculation based on the fiducial image performed in step 804. The handheld data 832 may include IMU data and / or the position and / or orientation calculation based on the IMU data performed in step 812. The position and / or orientation calculation at step 814 may be performed according to one of various operating states based on the number of fiducials found in the fiducial image. Each operating state may process the fiducial data 830 and the handheld data 832 differently and may emphasize one type of data more than the other type of data. Reference will be made to Figure 13 These operating states are described in more detail.

[0063] At step 816, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is output, for example, to an external device and / or a process for operating the AR system 700. For example, the position and / or orientation may be output to the AR system 700 to generate and display virtual content.

[0064] Figure 9 An example configuration of an AR system 900 is shown, in which a handheld device 326 includes a front handheld imaging device 326A and a rear handheld imaging device 326B, wherein the field of view of the front handheld imaging device 326A at least partially and at least temporarily includes one or more surrounding features 344 when the user is holding the handheld device 304, and the field of view of the rear handheld imaging device 326B at least partially and at least temporarily includes one or more wearable fiducials 306 when the user is holding the handheld device 304 in normal operation. In the example configuration, multiple wearable fiducials 322 are fixed to the wearable device 302. The AR system 900 can include additional sensors mounted on the handheld device 304, such as a handheld IMU 324. One advantage of this configuration can be the increased accuracy provided by the multiple imaging devices. The AR system 900 can include additional sensors mounted on the wearable device 302, such as a wearable IMU 308.

[0065] Figure 10A method 1000 is shown for performing positioning of a handheld device 304 relative to a wearable device 302 using the AR system 900. One or more steps of the method 1000 may be omitted or performed in a different order than the illustrated embodiment, and one or more steps of the method 1000 may be performed at one or more processing devices located within the wearable device 302, the handheld device 304, and / or the waist pack 340.

[0066] At step 1002, an image ("fiducial image") is captured by rear handheld imaging device 326B. The fiducial image may contain several of the wearable fiducials 306. For example, if there are three wearable fiducials 306, the fiducial image may be analyzed to determine if it contains zero, one, two, or three fiducials.

[0067] At step 1004, the position and / or orientation of handheld device 304 relative to wearable device 302 is calculated, for example, based on the fiducial image. For example, the fiducial image can be analyzed to determine the position of any fiducials in wearable fiducials 306, and the position and / or orientation can be determined based on the positions of the fiducials within the fiducial image and the known physical relationships between wearable fiducials 306. The position and / or orientation of handheld device 304 can be used to determine a posture of handheld device 304 relative to wearable device 302. The output of step 1004 is referred to as fiducial data 1030.

[0068] At step 1006 , an image (“world image”) is captured by the front handheld imaging device 326A. The world image may include surrounding features 344 .

[0069] At step 1008, the position and / or orientation of handheld device 304 relative to the world is calculated based on the world image. In some cases, the world image is compared to a previous world image to estimate the motion of handheld device 304 using visual odometry techniques, which may include performing feature detection in each world image to establish correspondence between the world images. A motion vector of handheld device 304 that best corresponds to the motion of the features detected in the world image can then be calculated. The output of step 1008 is referred to as handheld data 1032.

[0070] At step 1010, data indicating at least rotational motion of the handheld device 304 relative to the world ("IMU data") is detected by the handheld IMU 324. The IMU data may include rotational velocity or raw data from which rotational velocity can be calculated. In some embodiments, the IMU data also indicates linear motion of the handheld device 304 and may include linear acceleration or raw data from which linear acceleration can be calculated.

[0071] At step 1012, the position and / or orientation of the handheld device 304 is calculated based on the IMU data. In some embodiments, the position and / or orientation of the handheld device 304 relative to the world is calculated (using a previously known and / or estimated orientation relative to the world), and / or in some embodiments, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated (using a previously known and / or estimated orientation relative to the wearable device 302). The output of step 1012 is referred to as handheld data 1032.

[0072] At step 1014, a position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated based on the fiducial data 1030 and / or the handheld data 1032. The fiducial data 1030 may include a fiducial image and / or a position and / or orientation calculation based on the fiducial image performed in step 1004. The handheld data 1032 may include a world image, a position and / or orientation calculation based on the world image performed in step 1008, IMU data, and / or a position and / or orientation calculation based on the IMU data performed in step 1012. The position and orientation calculation at step 1014 may be performed according to one of various operating states based on the number of fiducials found in the fiducial image. Each operating state may process the fiducial data 1030 and the handheld data 1032 differently and may emphasize one type of data more than the other type of data. Reference will be made to Figure 13 These operating states are described in more detail.

[0073] At step 1016, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is output, for example, to an external device and / or a process for operating the AR system 900. For example, the position and / or orientation may be output to the AR system 900 to generate and display virtual content.

[0074] Figure 11A An example configuration of an AR system 1100A is shown, wherein a wearable device 302 includes one or more wearable imaging devices 310, a field of view of the wearable imaging device 310 at least partially and at least temporarily including a handheld fiducial 322 when a user holds the handheld device 304 in normal operation, and wherein the handheld device 304 includes one or more handheld imaging devices 326, a field of view of the handheld imaging device 326 at least partially and at least temporarily including one or more surrounding features 344 when a user holds the handheld device 304 in normal operation. Figure 11A In the example configuration shown, a single handheld fiducial 322 is affixed to the handheld device 304. The AR system 1100 may include additional sensors mounted on the handheld device 304, such as a handheld IMU 324. Figure 11AAdvantages of the illustrated configuration include the increased accuracy provided by multiple imaging devices, and the computational efficiency of calculating position and orientation while being constrained by a single fiducial location. The AR system 1100A may include additional sensors mounted on the wearable device 302 , such as a wearable IMU 308 .

[0075] Figure 11B An example configuration of an AR system 1100B is shown, wherein a wearable device 302 includes one or more wearable imaging devices 310, a field of view of the wearable imaging device 310 at least partially and at least temporarily including a handheld fiducial 322 when a user holds a handheld device 304 in normal operation, and wherein the handheld device 304 includes one or more handheld imaging devices 326, a field of view of the handheld imaging device 326 at least partially and at least temporarily including one or more surrounding features 344 when a user holds the handheld device 304 in normal operation. Figure 11B In the example configuration shown, multiple handheld fiducials 322 are fixed to the handheld device 304. The AR system 1100B can include additional sensors mounted on the handheld device 304, such as a handheld IMU 324. Advantages of such a configuration include the increased accuracy provided by multiple imaging devices and the increased robustness provided by combining fiducial-based tracking with visual odometry techniques. The AR system 1100B can include additional sensors mounted on the wearable device 302, such as a wearable IMU.

[0076] Figure 12 Shows the use Figure 11A AR System 1100A or Figure 11B The AR system 1100B performs a method 1200 for positioning the handheld device 304 relative to the wearable device 302. One or more steps of the method 1200 may be omitted or performed in a different order than the illustrated embodiment, and one or more steps of the method 1200 may be performed at one or more processing devices located within the wearable device 302, the handheld device 304, and / or the waist pack 340.

[0077] At step 1202, an image ("fiducial image") is captured by the wearable imaging device 310. The fiducial image may include several handheld fiducials 322. For example, for Figure 11A If there is a handheld fiducial 322, the fiducial image can be analyzed to determine whether it contains zero or one fiducial. Figure 11B If there are three handheld fiducials 322, the fiducial image may be analyzed to determine if it contains zero, one, two, or three fiducials.

[0078] At step 1204, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated, for example, based on the reference point image. Figure 11A , the fiducial image can be analyzed to determine the location of the fiducials, and position and / or orientation constraints can be determined based on the locations of the fiducials within the fiducial image. Figure 11B The fiducial image may be analyzed to determine the location of any fiducials, and the position and / or orientation may be determined based on the location of the fiducials within the fiducial image and the known physical relationship between the wearable fiducials 306 .

[0079] At step 1206 , an image (“world image”) is captured by handheld imaging device 326 . The world image may include surrounding features 344 .

[0080] At step 1208, the position and / or orientation of handheld device 304 relative to the world is calculated based on the world image. In some cases, the world image is compared to a previous world image to estimate the motion of handheld device 304 using visual odometry techniques, which may include performing feature detection in each world image to establish correspondence between the world images. A motion vector of handheld device 304 that best corresponds to the motion of the features detected in the world image can then be calculated. The output of step 1208 is referred to as handheld data 1232.

[0081] At step 1210, data indicating at least rotational motion of the handheld device 304 relative to the world ("IMU data") is detected by the handheld IMU 324. The IMU data may include rotational velocity or raw data from which rotational velocity may be calculated. In some embodiments, the IMU data also indicates linear motion of the handheld device 304 and may include linear acceleration or raw data from which linear acceleration may be calculated.

[0082] At step 1212, the position and / or orientation of the handheld device 304 is calculated based on the IMU data. In some embodiments, the position and / or orientation of the handheld device 304 relative to the world is calculated (using a previously known and / or estimated orientation relative to the world), and / or in some embodiments, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated (using a previously known and / or estimated orientation relative to the wearable device 302). The output of step 1212 is referred to as handheld data 1232.

[0083] At step 1214, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is calculated based on the reference data 1230 and / or the handheld data 1232. For example, for Figure 11A, the fiducial data 1230 may include fiducial point images and / or constraints based on the position and / or orientation calculations performed in step 1204. For example, for Figure 11B , fiducial data 1230 may include a fiducial image and / or position and / or orientation calculations based on the fiducial image performed in step 1204. Handheld data 1232 may include a world image, a position and / or orientation calculation based on the world image performed in step 1208, IMU data, and / or position and / or orientation calculations based on the IMU data performed in step 1212. The position and orientation calculations at step 1214 may be performed according to one of various operating states based on the number of fiducials found in the fiducial image. Each operating state may process fiducial data 1230 and handheld data 1232 differently and may emphasize one type of data over the other. Reference will be made to Figure 13 These operating states are described in more detail.

[0084] At step 1216, the position and / or orientation of the handheld device 304 relative to the wearable device 302 is output, for example, to an external device and / or a process for operating the AR system 1100. For example, the position and / or orientation may be output to the AR system 1100 to generate and display virtual content.

[0085] Figure 13 A method 1300 is shown for performing positioning of a handheld device 304 relative to a wearable device 302 using any of the AR systems 500, 700, 900, 1100, or any combination thereof. One or more steps of the method 1300 may be omitted or performed in a different order than the illustrated embodiment, and one or more steps of the method 1300 may be performed at one or more processing devices located within the wearable device 302, the handheld device 304, and / or the waist pack 340.

[0086] At step 1302, data indicating the motion of handheld device 304 relative to wearable device 302 ("fiducial data") is obtained using an imaging device. Executing step 1302 may include executing one or both of steps 1304 and 1306. At step 1304, an image containing several wearable fiducials 306 ("fiducial image") is captured by rear handheld imaging device 326B. At step 1306, an image containing several handheld fiducials 322 ("fiducial image") is captured by wearable imaging device 310.

[0087] At step 1308 , data indicating at least rotational motion of the handheld device 304 relative to the world (“handheld data”) is detected. Executing step 1308 may include executing one or both of steps 1310 , 1312 .

[0088] At step 1310, an image ("world image") containing surrounding features 344 is captured by the front handheld imaging device 326A. At step 1312, data indicating at least rotational motion of the handheld device 304 relative to the world ("IMU data") is detected by the handheld IMU 324. The IMU data may include rotational velocity or raw data from which rotational velocity can be calculated. In some embodiments, the IMU data also indicates linear motion of the handheld device 304 and may include linear acceleration or raw data from which linear acceleration can be calculated.

[0089] At step 1314, the number of fiducials contained in the fiducial image and the locations (eg, pixel locations) of the observed fiducials are determined.

[0090] At step 1316, the position and / or orientation of handheld device 304 relative to wearable device 302 is calculated / estimated / updated according to one of three operating states. The operating state is selected based on the number of fiducials observed in the fiducial image. In the illustrated embodiment, a first operating state ("State 1") is selected when three or more fiducials are observed in the fiducial image, a second operating state ("State 2") is selected when one or two fiducials are observed in the fiducial image, and a third operating state ("State 3") is selected when zero fiducials are observed in the fiducial image. Switching between states can occur each time a new fiducial image is captured or at predetermined time intervals. For example, step 1316 can be performed at each camera frame based on one or both of the fiducial data (e.g., fiducial image) and the handheld data (e.g., world image and IMU orientation). Step 1316 can further incorporate previous position and / or orientation calculations to improve estimation accuracy.

[0091] According to a first operating state ("State 1"), position and / or orientation can be calculated with high accuracy (full six degrees of freedom), for example, based solely on fiducial data. When four or more fiducials are observed, position can be fully solved. When exactly three fiducials are observed, there are two possible position solutions, one of which can be discarded based on additional processing and / or comparison with previously calculated positions. In some embodiments, handheld data can be used to supplement and improve the accuracy of the calculations. In some embodiments, an extended Kalman filter can be employed to improve accuracy based on previous position and / or orientation calculations.

[0092] According to the second operating state ("State 2"), for example, both the reference data and the handheld data can be used to calculate position and / or orientation. When two reference points are observed, the reference data enables the calculation of a constrained position and / or orientation, and the handheld data can be used to complete the calculation within the constraints imposed by the reference data. In some embodiments, an extended Kalman filter can be employed to improve accuracy based on previous position and / or orientation calculations. Calculations performed in the second operating state may generally be less accurate than those performed in the first operating state.

[0093] In a third operating state ("State 3"), position and orientation may be calculated based solely on handheld data (i.e., dead reckoning), for example. In some embodiments, an extended Kalman filter may be employed to improve accuracy based on previous position and / or orientation calculations. Calculations performed in the third operating state may generally be less accurate than those performed in the first or second operating states.

[0094] At step 1318, IMU bias correction is performed to increase the accuracy of the IMU data provided as input at step 1316. Since IMU data may drift over time, periodic updates can recalibrate the IMU data. In some embodiments, bias updates are only provided when the first operating state is selected, and can provide high-accuracy bias updates. In some embodiments, bias updates are provided when either the first operating state or the second operating state is selected, as both states utilize reference data in their calculations. Bias updates can be provided at each camera frame or at predetermined time intervals.

[0095] Figure 14 A simplified computer system 1400 is shown according to some embodiments described herein. Figure 14 A schematic diagram of an example of a computer system 1400 that can perform some or all of the steps of the methods provided by various embodiments is provided. Figure 14 It is intended only to provide a general description of the various components, any or all of which may be used as appropriate. Figure 14 It is broadly illustrated how the various system elements may be implemented in a relatively separate or relatively more integrated manner.

[0096] Computer system 1400 is shown as including hardware elements that can be electrically coupled via bus 1405 or can communicate in other suitable ways. These hardware elements may include: one or more processors 1410, which may include but are not limited to one or more general-purpose processors and / or one or more special-purpose processors, such as digital signal processing chips, graphics acceleration processors, and / or the like; one or more input devices 1415, which may include but are not limited to a mouse, keyboard, camera, and / or the like; one or more output devices 1420, which may include but are not limited to a display device, a printer, and / or the like.

[0097] The computer system 1400 may further include and / or communicate with one or more non-transitory storage devices 1425, which may include, but are not limited to, local and / or network accessible memory, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices such as programmable, flash-updatable random access memory ("RAM") and / or read-only memory ("ROM"), and / or the like. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0098] The computer system 1400 may also include a communication subsystem 1419, which may include but is not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device and / or a chipset, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication facilities, etc., and / or the like. The communication subsystem 1419 may include one or more input and / or output communication interfaces to allow data to be exchanged with a network such as the network described by way of example below, other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other implementation issues, a portable electronic device or similar device may transmit images and / or other information via the communication subsystem 1419. In other embodiments, a portable electronic device such as the first electronic device may be incorporated into the computer system 1400, for example as an electronic device of the input device 1415. In some embodiments, the computer system 1400 also includes a working memory 1435, which may include the RAM or ROM devices described above.

[0099] The computer system 1400 may also include software elements, which are shown as currently located within a working memory 1435, including an operating system 1440, device drivers, executable libraries, and / or other code, such as one or more application programs 1445. As described herein, these application programs 1445 may include computer programs provided by various embodiments and / or may be designed to implement methods and / or configure systems provided by other embodiments. By way of example only, one or more processes described with reference to the above methods may be implemented as code and / or instructions executable by a computer and / or a processor within a computer; then, in one aspect, such code and / or instructions may be used to configure and / or adjust a general-purpose computer or other device to perform one or more operations according to the described methods.

[0100] A set of these instructions and / or codes may be stored on a non-transitory computer-readable storage medium, such as the storage device 1425 described above. In some cases, the storage medium may be incorporated into a computer system, such as the computer system 1400. In other embodiments, the storage medium may be separate from the computer system, such as a removable medium, such as an optical disc, and / or provided in an installation package so that the storage medium can be used to program, configure, and / or adjust a general-purpose computer having the instructions / code stored thereon. These instructions may be in the form of executable code that can be executed by the computer system 1400, and / or may be in the form of source code and / or installable code that, when compiled and / or installed on the computer system 1400 using, for example, various commonly available compilers, installers, compression / decompression utilities, etc., then takes the form of executable code.

[0101] It will be apparent to those skilled in the art that substantial variations can be made depending on specific requirements. For example, custom hardware can be used, and / or specific elements can be implemented in hardware and / or software, including portable software (e.g., applets, etc.). In addition, connections to other computing devices, such as network input / output devices, can be employed.

[0102] As described above, in one aspect, some embodiments may employ a computer system such as computer system 1400 to perform methods according to various embodiments of the present technology. According to one set of embodiments, some or all of the processes of these methods are performed by computer system 1400 in response to processor 1410 executing one or more sequences of one or more instructions, which may be incorporated into operating system 1440 and / or other code such as application programs 1445 contained in working memory 1435. Such instructions may be read into working memory 1435 from another computer-readable medium (e.g., one or more storage devices 1425). By way of example only, execution of the sequences of instructions contained in working memory 1435 may cause processor 1410 to perform one or more processes of the methods described herein. Additionally or alternatively, portions of the methods described herein may be performed by dedicated hardware.

[0103] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a specific manner. In embodiments implemented using computer system 1400, various computer-readable media may be involved in providing instructions / code to processor 1410 for execution, and / or may be used to store and / or carry such instructions / code. In many embodiments, computer-readable media is a physical and / or tangible storage medium. Such media may take the form of non-volatile media or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 1425. Volatile media include, but are not limited to, dynamic memory, such as working memory 1435.

[0104] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette, or any other medium from which a computer can read instructions and / or code.

[0105] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor 1410 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium for receipt and / or execution by computer system 1400.

[0106] The communication subsystem 1419 and / or its components will typically receive the signal, and the bus 1405 may then transfer the signal and / or the data, instructions, etc. carried by the signal to the working memory 1435, from which the processor 1410 retrieves and executes the instructions. The instructions received by the working memory 1435 may optionally be stored on the non-transitory storage device 1425 either before or after execution by the processor 1410.

[0107] The methods, systems, and devices discussed above are examples. Various configurations may omit, replace, or add various processes or components as appropriate. For example, in alternative configurations, the methods may be performed in an order different from that described, and / or various stages may be added, omitted, and / or combined. Moreover, features described for a particular configuration may be combined in various other configurations. Different aspects and elements of the configurations may be combined in similar manners. Furthermore, technology is constantly evolving, and therefore many of the elements are examples and do not limit the scope of this disclosure or the claims.

[0108] Specific details are provided in the specification to provide a thorough understanding of the exemplary configurations including the embodiments. However, the configurations can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid confusing the configurations. This description provides only example configurations and does not limit the scope, applicability, or configurations of the claims. On the contrary, the foregoing description of the configurations will provide those skilled in the art with enabling instructions for implementing the described technologies. Various changes may be made to the functions and arrangements of the elements without departing from the spirit or scope of this disclosure.

[0109] In addition, a configuration may be described as a process, which can be depicted as a schematic flow chart or block diagram. Although each process may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the accompanying drawings. In addition, examples of methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. A processor may perform the described tasks.

[0110] Several example configurations have been described, and various modifications, alternative configurations, and equivalents may be used without departing from the spirit of the present disclosure. For example, the above elements may be components of a larger system, in which other rules may take precedence over or otherwise modify the application of the present technology. Furthermore, many steps may be performed before, during, or after considering the above elements. Therefore, the above description does not limit the scope of the claims.

[0111] As used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a user" includes a plurality of such users and reference to "the processor" includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0112] Furthermore, when used in this specification and the following claims, the words "comprises," "includes," "contains," "includes," "has," "covers," and "with" are intended to specify the presence of stated features, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, integers, components, steps, actions, or groups thereof.

[0113] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes based on these examples and embodiments will be suggested to those skilled in the art, and these modifications or changes will be included in the spirit and scope of this application and the scope of the appended claims.

Claims

1. A method for performing positioning of a handheld device relative to a wearable device, the method comprising: obtaining handheld data indicating movement of the handheld device via a sensor mounted on the handheld device; obtaining, by an imaging device mounted on the wearable device, reference data indicating a movement of the handheld device, wherein obtaining the reference data comprises capturing, by the imaging device, a reference point image including a plurality of reference points fixed to the handheld device; determining the number of reference points contained in the reference point image; and The position and orientation of the handheld device are updated using the fiducial data and the handheld data and based on the number of fiducials contained in the fiducial image.

2. The method according to claim 2, wherein: The sensor includes an inertial measurement unit (IMU).

3. The method according to claim 2, wherein: Updating the position and orientation of the handheld device is based on the first operating state or the second operating state.

4. The method according to claim 4, further comprising: in response to determining that the number of reference points is equal to or greater than three, updating at least one of a position and an orientation of the handheld device based on the reference data according to the first operating state; as well as In response to determining that the number of reference points is equal to one or two, at least one of the position and orientation of the handheld device is updated based on the reference data and the handheld data according to the second operating state.

5. The method according to claim 5, further comprising: In response to determining that the number of reference points is equal to zero, at least one of the position and orientation of the handheld device is updated based on the handheld data according to a third operating state.

6. The method according to claim 6, wherein: updating at least one of a position and an orientation of the handheld device based solely on the reference data according to the first operating state; as well as At least one of the position and orientation of the handheld device is updated based solely on the handheld data according to the third operating state.

7. The method according to claim 2, wherein: The plurality of reference points includes a plurality of light emitting diodes (LEDs).

8. The method according to claim 2, wherein: The sensor includes a second imaging device; and Obtaining the handheld data includes capturing, via the second imaging device, an image of the world including one or more features surrounding the handheld device.

9. A method for performing positioning of a handheld device relative to a wearable device, the method comprising: obtaining handheld data from a sensor mounted on the handheld device; obtaining fiducial data from an imaging device mounted on the wearable device or the handheld device, wherein the fiducial data includes a fiducial point image, the fiducial point image including fiducial point images of a plurality of fiducial points fixed to the wearable device or the handheld device; determining the number of reference points contained in the reference point image; determining whether the number of fiducial points included in the fiducial point image is equal to one of a first set of values or one of a second set of values different from the first set of values; in response to determining that the number of fiducials included in the fiducial image is equal to one of the first set of values, updating the position and orientation of the handheld device based on the fiducial data or based on the fiducial data and the handheld data according to a first operational state; as well as In response to determining that the number of reference points included in the reference point image is equal to one of the second set of values, updating the position and orientation of the handheld device based on the reference data and the handheld data or based on the handheld data according to a second operating state different from the first operating state.

10. The method according to claim 10, wherein: The first set of values includes integers equal to or greater than 3.

11. The method according to claim 11, wherein: The position and orientation are updated based solely on the reference data.

12. The method according to claim 10, wherein: The second set of values consists of integers equal to 1 or 2.

13. The method according to claim 10, wherein: The second set of values is equal to 0, and the method further includes: in response to determining that the number of reference points included in the reference point image is equal to 0, updating the position and orientation of the handheld device based only on the handheld data.

14. The method according to claim 10, wherein: The plurality of reference points include a plurality of light emitting diodes (LEDs).

15. The method according to claim 10, wherein The imaging device is mounted on the handheld device, and a plurality of reference points including the several reference points are fixed to the wearable device.

16. The method according to claim 16, wherein: Obtaining the handheld data includes capturing an image of the world including one or more features surrounding the handheld device via a second handheld imaging device mounted on the handheld device.

17. The method according to claim 10, wherein The imaging device is mounted on the wearable device, and a plurality of reference points including the several reference points are fixed to the handheld device.

18. The method according to claim 18, wherein The sensor includes an inertial measurement unit (IMU).

19. The method of claim 18, wherein obtaining the handheld data comprises capturing an image of the world including one or more features surrounding the handheld device via a second handheld imaging device mounted on the handheld device.

20. The method according to claim 10, wherein The sensor includes an inertial measurement unit (IMU).

Images