Illuminating User Input Devices

Abstract

To provide a favorable light-emitting user input device.SOLUTION: A light-emitting user input device can include a touch sensitive portion configured to accept user input (e.g., from a user's thumb) and a light emitting portion configured to output a light pattern. The light pattern can be used to assist the user in interacting with the user input device. Examples include the steps of: emulating a multi-degree-of-freedom controller; indicating scrolling or swiping actions; indicating presence of objects nearby the device; indicating receipt of notifications; assisting pairing the user input device with another device; or assisting calibrating the user input device. The light-emitting user input device can be used to provide user input to a wearable device, such as, e.g., a head mounted display device.SELECTED DRAWING: Figure 17B

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 490,863, filed April 27, 2017, and entitled "LIGHT-EMITTING USER INPUT DEVICE," the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to light-emitting user input devices associated with imaging and visualization systems. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input. Augmented reality or “AR” scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or “MR” relates to the merging of real and virtual worlds to create new environments in which physical and virtual objects coexist and interact in real time. Consequently, the human visual perception system is highly complex, making it challenging to produce VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] Examples of light-emitting user input devices are disclosed. The user input device embodiments can be used to provide input to an AR, VR, or MR device. The light-emitting user input device can also provide visual information of events or objects associated with the AR / VR / MR device to a user or people within the user's environment.

[0005] The light-emitting user input device can include a touch-sensitive portion configured to receive user input (e.g., from a user's thumb) and a light-emitting portion configured to output a light pattern. The light pattern can be used to assist a user in interacting with the user input device. Examples include emulating a multi-degree-of-freedom controller, indicating a scrolling or swiping action, indicating the presence of an object in the vicinity of the device, indicating the receipt of a notification, assisting in the calibration of the user input device, or assisting in the pairing of the user input device with another device. The light-emitting user input device can be used to provide user input to, for example, a head-mounted display system such as a mixed reality display device. The present invention provides, for example, the following. (Item 1) 1. A system comprising: a light emitting assembly of a user input device, the light emitting assembly configured to illuminate a plurality of light patterns to provide information of objects in an environment; a hardware processor communicatively coupled to the light emitting assembly, Identifying objects in a user's environment; accessing context information associated with the object; determining a characteristic of a light pattern illuminated by the light emitting assembly based at least in part on the contextual information; and instructing the light emitting assembly to illuminate according to the light pattern; a hardware processor programmed to perform the A system comprising: (Item 2) Item 10. The system of item 1, wherein the object includes at least one of a physical object or a virtual object. (Item 3) Item 3. The system of item 2, wherein the status of the virtual object includes at least one of a user's current interaction with the virtual object, whether the virtual object has received new information, whether the virtual object is idle, or whether the virtual object is in an error state. (Item 4) Item 10. The system of item 1, wherein the characteristics of the light pattern include at least one of brightness, position, shape, size, or color. (Item 5) Item 10. The system of item 1, wherein the contextual information associated with the object includes at least one of the location of the object, the urgency of the object, the type of the object, the nature of the object, the amount of information associated with the object, or the user's preferences. (Item 6) Item 10. The system of claim 1, further comprising a wearable display device, wherein the object is invisible from the user's view or external through the wearable display device, and wherein the hardware processor is programmed to determine at least one of the size, shape, or color of the light pattern and provide cues to the user for locating the object. (Item 7) Item 10. The system of item 1, wherein the object is a component of a wearable system for presenting virtual content to a user, and the light pattern indicates a status of the component of the wearable system. (Item 8) 8. The system of claim 7, wherein the component includes at least one of the user input device, a wearable display device, or a battery pack. (Item 9) Item 10. The system of item 1, wherein the status includes at least one of a battery status, a power charging status, a wireless pairing status between the wearable display device and the user input device, a status of a calibration process of the user input device, or a status of the wearable display device. (Item 10) Item 10. The system of item 1, wherein the light pattern encodes an alert or information associated with the object. (Item 11) The status is: a user's current interaction with said object; whether the object has received new information; whether the object is in an idle state, or Whether the object is in an error state Item 1, the system including at least one of: (Item 12) Item 10. The system of item 1, wherein the characteristics of the light pattern are configurable by a user via an application programming interface. (Item 13) 1. A light emitting user input device, comprising: a touch component configured to receive user input; a light emitting assembly configured to output a plurality of light patterns, the light emitting assembly at least partially surrounding the touch sensitive component; a hardware processor communicatively coupled to the touch component and the light emitting assembly, Identifying user interface actions supported by the touch component based on the context information; determining a first light pattern associated with the user interface action; generating and transmitting instructions to the light emitting assembly to display a halo having the first light pattern; Receiving user input on the touch component; updating the halo with a second light pattern to reflect the user input; and a hardware processor programmed to perform the 1. A light emitting user input device comprising: (Item 14) The context information is the user's environment; the type of input supported by the light-emitting user input device; information associated with an object with which the handheld user input device is configured to interact; or A characteristic of a wearable device associated with the handheld user input device. Item 14. The luminous user input device of item 13, comprising at least one of: (Item 15) Item 14. The light-emitting user input device of item 13, wherein the light-emitting assembly comprises a light guide and a ring of LEDs. (Item 16) Item 14. The light-emitting user input device of item 13, wherein the light-emitting assembly is configured to receive a plurality of user inputs, and the hardware processor is further programmed to display a halo based, at least in part, on the plurality of user inputs supported by activating the light guide. (Item 17) Item 14. The luminous user input device of item 13, wherein the user input includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 18) Item 14. The illuminated user input device of item 13, wherein the illuminated user input device comprises at least one of a totem, a smartwatch, or a smartphone. (Item 19) Item 14. The illuminating user input device of item 13, wherein the first light pattern provides a cue to a user that the user interface action is supported by the illuminating user input device. (Item 20) The hardware processor further comprises: determining whether the received user input is inappropriate based on the context information; It is programmed to Item 14. The luminous user input device of item 13, wherein the second light pattern, in response to determining that the received user input is inappropriate, provides a visual alert that the received user input is inappropriate. (Item 21) Item 14. The luminous user input device of item 13, wherein at least a portion of the halo appears brighter or larger in the second light pattern compared to the first light pattern. (Item 22) Item 14. The luminous user input device of item 13, wherein the characteristics of the plurality of light patterns are configurable by a user via an application programming interface. (Item 23) 23. The luminous user input device of claim 22, wherein the characteristics include at least one of arc location or movement pattern, color, brightness, shape, or size. (Item 24) Item 14. The luminous user input device of item 13, wherein the second light pattern, in response to the user input, indicates at least one of battery status, power charging status, wireless pairing status between the handheld user input device and another computing device, or whether the handheld user input device is idle. (Item 25) 1. A method comprising: Under the control of a hardware processor, identifying, based on the context information, a type of user input supported by a light emitting user input device, the light emitting user input device comprising a first element for illuminating a plurality of light patterns and a second element for receiving the user input; determining a first light pattern associated with the supported user input type; generating and transmitting to the first element instructions to illuminate a halo having the first light pattern; determining a second light pattern based on user input on the light emitting user input device; updating the halo to the second light pattern in response to the user input; A method comprising: (Item 26) Item 26. The method of item 25, wherein the first element is further configured to receive another user input, and the type of user input for determining the first light pattern is associated with other user inputs supported by the first element. (Item 27) The context information is the user's environment; the type of input supported by the light-emitting user input device; information associated with an object with which the handheld user input device is configured to interact; or A characteristic of a wearable device associated with the handheld user input device. 26. The method according to item 25, comprising at least one of the following: (Item 28) 26. The method of claim 25, wherein the type of user input includes at least one of a swipe, a tap, a press, or a touch input. (Item 29) Item 26. The method of item 25, wherein the first light pattern provides a cue to a user that the type of user input is supported by the light-emitting user input device. (Item 30) determining whether user input received by the light-emitting user input device is inappropriate based on the context information; further comprising 26. The method of claim 25, wherein the second light pattern, in response to determining that the user input is incorrect, provides a visual alert that the user input is incorrect. (Item 31) Item 26. The method of item 25, wherein at least a portion of the halo appears brighter or larger in the second light pattern compared to the first light pattern. (Item 32) 1. A light emitting user input device, comprising: a plurality of interactable areas configured to receive user input, at least one interactable area of ​​the plurality of interactable areas comprising a portion of a light emitting assembly of the light emitting user input device; A hardware processor, detecting a user actuation of the light emitting user input device; determining an interactable area among the plurality of interactable areas corresponding to a user actuation; translating the user action into a user input for performing a user interface action based at least on the type of action and the interaction area associated with the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input; a hardware processor programmed to perform the 1. A light emitting user input device comprising: (Item 33) Item 33. The light-emitting user input device of item 32, wherein the light-emitting assembly further comprises a touch surface, the light-emitting assembly comprises an optical waveguide, and the plurality of interactable areas comprises a first interactable area associated with the optical waveguide and a second interactable area associated with the touch surface. (Item 34) Item 34. The light-emitting user input device of item 33, wherein in response to detecting that the user has activated the first interactive area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a first light pattern associated with the user input in the first interactive area, and in response to detecting that the user has activated the second interactive area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input in the second interactive area. (Item 35) Item 33. The illuminating user input device of item 32, wherein the light pattern is determined at least in part based on contextual information associated with the illuminating user input device. (Item 36) Item 33. The luminous user input device of item 32, wherein the user actuation includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 37) Item 33. The luminous user input device of item 32, wherein the light pattern comprises an arc-shaped region having a color, arc length, or visual effect. (Item 38) Item 33. The light-emitting user input device of item 32, wherein the hardware processor is further programmed to communicate with a wearable device and cause the wearable device to perform a user interface operation based on the user input. (Item 39) 1. A method comprising: Under the control of a hardware processor, Detecting a user actuation of an illuminated user input device, the illuminated user input device comprising a plurality of interactable areas; determining an interactable area among the plurality of interactable areas corresponding to a user actuation; translating the user action into a user input for performing a user interface action based at least on the type of action and the interaction area associated with the action; causing a light emitting assembly of a light emitting user input device to illuminate a light pattern in response to the user input; A method comprising: (Item 40) 40. The method of claim 39, wherein the plurality of interactable areas comprises a first interactable area supporting a first type of user input and a second interactable area supporting a second type of user input. (Item 41) Item 41. The method of item 40, wherein in response to detecting that the user has activated the first interactive area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a first light pattern associated with the user input in the first interactive area, and in response to detecting that the user has activated the second interactive area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input in the second interactive area. (Item 42) Item 41. The method of item 40, wherein at least one of the first type of user input or the second type of user input depends on contextual information associated with the user's interaction with the wearable system, the contextual information including at least one of the type of application the user is interacting with, available user inputs supported by the plurality of interactable areas, or the user's virtual environment. (Item 43) Item 40. The method of item 39, wherein the light-emitting user input device comprises a touch surface divided into one or more interactable areas. (Item 44) 40. The method of claim 39, wherein the light pattern is determined at least in part based on contextual information associated with the light emitting user input device. (Item 45) Item 40. The method of item 39, wherein the plurality of interactable areas comprises a light-emitting assembly of the light-emitting user input device. (Item 46) 40. The method of claim 39, wherein the user actuation includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 47) Item 40. The method of item 39, wherein the light pattern comprises an arc-shaped region having a color, arc length, or visual effect. (Item 48) 1. A system for calibrating a light emitting user input device, the system comprising: an outward-facing imaging system configured to image the environment; a light emitting user input device configured to illuminate a light pattern; a hardware processor in communication with the outward-facing imaging system and the light-emitting user input device, accessing first movement data obtained by a sensor of the light emitting user input device associated with movement of the light emitting user input device from a first pose to a second pose; determining a first image of the light emitting user input device, the first image including a light pattern corresponding to the first orientation of the light emitting user input device; determining a second image of the light-emitting user input device acquired by the outward-facing imaging system, the second image including a light pattern corresponding to the second orientation of the light-emitting user input device; and analyzing the second image and calculating second movement data associated with movement of the light emitting user input device; Detecting a difference between the first movement data and the second movement data; calibrating a sensor of the light emitting user input device in response to determining that the difference reaches a threshold condition; and a hardware processor programmed to perform the A system comprising: (Item 49) Item 49. The system of item 48, wherein moving the light-emitting user input device from the first pose to the second pose includes a change in at least one of a position or an orientation of the light-emitting user input device. (Item 50) Item 49. The system of item 48, wherein the first pose or the second pose corresponds to a base position of the light-emitting user input device. (Item 51) Item 49. The system of item 48, wherein the light pattern is associated with a halo illuminated by a plurality of light emitting diodes surrounding the touchable portion of the light emitting user input device. (Item 52) Item 49. The system of item 48, wherein, to calculate the second movement data, the hardware processor is programmed to calculate a change in the shape of a halo in the first image and the second image. (Item 53) Item 49. The system of item 48, wherein the light-emitting user input device is a totem for interaction with an augmented reality device. (Item 54) Item 54. The system of item 53, wherein the sensor is part of an inertial measurement unit (IMU) of the totem, and the sensor is calibrated by adjusting at least one of the responsiveness of the totem to user movement or a mapping between the user movement and the IMU measurements. (Item 55) Item 54. The system of item 53, wherein the totem is further configured to provide a visual indication to a user to move the light-emitting user input device from the first position to the second position. (Item 56) Item 54. The system of item 53, wherein the totem has three degrees of freedom. (Item 57) Item 49. The system of item 48, wherein the hardware processor is programmed to apply a computer vision algorithm to analyze the first image and the second image and identify light patterns within the first image and the second image. (Item 58) Item 49. The system of item 48, wherein the light-emitting user input device is capable of illuminating another light pattern in response to determining that calibration of the light-emitting user input device is complete. (Item 59) Item 49. The system of item 48, wherein the hardware processor is programmed to determine a type of calibration, and a light pattern illuminated by the light-emitting user input device corresponds to the type of calibration. (Item 60) 1. A method of calibrating a light emitting user input device, the method comprising: Under the control of a hardware processor, receiving movement data of an illuminating user input device in a certain pose, the movement data being obtained by a sensor of the illuminating user input device; receiving an image of the light emitting user input device in the pose; analyzing the image to identify a shape of a light pattern illuminated by the light emitting user input device; calculating at least one of a first position or a first orientation of the light emitting user input device at the pose based on the movement data; calculating at least one of a second position or a second orientation of the light emitting user input device in the pose based on a shape of the light pattern; determining a difference between the first position and the second position or the first orientation and the second orientation; calibrating a sensor of the light emitting user input device in response to determining that the difference reaches a threshold condition; and A method comprising: (Item 61) Item 61. The method according to item 60, wherein the first position, the first orientation, the second position, or the second orientation is calculated with reference to a base pose. (Item 62) Item 61. The method of item 60, wherein the shape of the light pattern in the image is oval, while the shape of the light pattern at the base pose is circular. (Item 63) Item 61. The method of item 60, wherein at least one of the position or movement pattern of the light pattern corresponds to the type of sensor being calibrated. (Item 64) Item 61. The method of item 60, further comprising providing at least one of visual, auditory, or tactile feedback in response to determining that the calibration was successful. (Item 65) Item 61. The method of item 60, wherein calibrating the sensor includes adjusting at least one of a responsiveness of the sensor to user movement or a mapping between the user movement and measurements of the sensor. (Item 66) 1. A system for calibrating a light emitting user input device, the system comprising: an outward-facing imaging system configured to image the environment; a hardware processor in communication with the outward-facing imaging system, receiving an image of the environment; analyzing the image to identify a light emitting user input device; determining a first orientation of the light emitting user input device and a first appearance of an illuminated halo light pattern; identifying a second appearance of the illuminated halo light pattern based at least in part on analysis of the image; and determining a first change to an attitude of the light emitting user input device; receiving movement data of the light-emitting user input device measured by the light-emitting user input device; calculating a second change to the attitude of the light emitting user input device based at least in part on analysis of the image; calculating a difference between the first change and the second change and determining whether the difference reaches a threshold; calibrating a sensor of the light emitting user input device in response to determining that the difference exceeds the threshold condition; and a hardware processor programmed to perform the A system comprising: (Item 67) Item 67. The system of item 66, wherein the pose of the light-emitting user input device includes a position and orientation of the light-emitting user input device. (Item 68) Item 67. The system of item 66, wherein the light-emitting user input device includes at least one of a totem, a smartwatch, or a smartphone, and the sensor is an IMU. (Item 69) Item 67. The system of item 66, wherein in response to determining that the difference does not exceed the threshold, the hardware processor is programmed to provide an indication that the sensor is calibrated. (Item 70) 70. The system of claim 69, wherein the indication comprises visual, auditory, or tactile feedback on the light-emitting user input device. (Item 71) Item 67. The system of item 66, wherein the first position is a base position of the light-emitting user input device. (Item 72) Item 67. The system of item 66, wherein the light-emitting user input device is further configured to illuminate a series of light patterns to guide a user of the light-emitting user input device to position the user input device in a position that includes the first position and the second position. (Item 73) Item 67. The system of item 66, wherein the hardware processor is programmed to determine a deformation of the shape of the light pattern in the second appearance relative to the shape of the light pattern in the first appearance to calculate a second change to the attitude of the light-emitting user input device based at least in part on analysis of the image. (Item 74) Item 67. The system of item 66, wherein the light-emitting user input device is part of a wearable system further comprising a wearable display for presenting virtual content within an augmented reality, virtual reality, or mixed reality environment. (Item 75) Item 67. The system of item 66, wherein to calibrate the sensor, the hardware processor is programmed to adjust at least one of the responsiveness of the sensor to user movement or the mapping between the user movement and the sensor measurement. (Item 76) 1. A method for calibrating a light emitting user input device, the method comprising: under the control of a light emitting user input device comprising a light emitting assembly and a hardware processor; determining a first pose of the light emitting user input device; causing the light emitting assembly to illuminate a first light pattern to guide a user to move the light emitting user input device to a second position; in response to determining that the light emitting user input device has been moved to the second position; obtaining pose data of said light emitting user input device; calibrating the light emitting user input device based at least in part on the pose data; and providing an indication to a user that the calibration process has been completed; and A method comprising: (Item 77) Item 77. The method of item 76, wherein determining the first pose is based, at least in part, on data obtained from an inertial measurement unit (IMU) of the light emitting user input device. (Item 78) 77. The method of claim 76, wherein the indication includes at least one of an audible, visual, or tactile indication. (Item 79) Item 77. The method of item 76, wherein the indication includes a second light pattern. (Item 80) 77. The method of claim 76, wherein the pose data includes at least one of position or orientation data of the light-emitting user input device. (Item 81) 77. The method of claim 76, wherein the pose data for the light emitting user input device includes data obtained when the light emitting user input device is in a plurality of poses. (Item 82) Item 77. The method of item 76, further comprising causing the light emitting assembly to illuminate a third light pattern indicating the start of a calibration process. (Item 83) 1. A system for calibrating a light emitting user input device, the system comprising: one or more sensors configured to obtain movement data of the light-emitting user input device; a light emitting assembly of the light emitting user input device configured to output a plurality of light patterns; A hardware processor, providing a first indication to a user to position the light emitting user input device in a certain position; obtaining movement data of the light emitting user input device to the pose; calibrating the light emitting user input device based at least in part on the movement data; and providing an indication to a user that the calibration process has been completed; and a hardware processor programmed to perform the A system comprising: (Item 84) Item 84. The system of item 83, wherein the first indication includes a light pattern, the location, movement, or combination of which provides a user with guidance for moving the light-emitting user input device to the certain position. (Item 85) Item 84. The system of item 83, wherein the light-emitting user input device is part of a wearable system for interacting with an augmented or mixed reality environment, and the first indication includes a virtual image provided by a head-mounted display of the wearable system showing the pose of the totem to be positioned by the user. (Item 86) The hardware processor further comprises: in response to determining that the light emitting user input device has been calibrated, providing a second indication that the light emitting user input device has been calibrated; or In response to determining that the light emitting user input device is not calibrated, continuing to calibrate the light emitting user input device. Item 84. The system of item 83, programmed to: (Item 87) Item 84. The system of item 83, wherein the hardware processor is further programmed to detect a start condition for initiating a calibration process for the light-emitting user input device. (Item 88) Item 84. The system of item 83, wherein to calibrate the light-emitting user input device, the hardware processor is programmed to access an image including the light-emitting user input device in the pose, analyze the movement data and the image, and identify a difference in at least one of a position or orientation of the light-emitting user input device calculated from the movement data and determined from the image. (Item 89) Item 84. The system of item 83, wherein the light-emitting user input device comprises a totem with a touch surface, and the light-emitting assembly is positioned adjacent to the touch surface. (Item 90) 1. A wearable device for pairing a light emitting user input device, the wearable device comprising: an outward-facing imaging system configured to image the environment; a hardware processor in communication with the outward-facing imaging system, receiving an image acquired by the outward-facing imaging system, the image including a light pattern illuminated by a light-emitting user input device; identifying a light-emitting user input device to be paired with the wearable device; analyzing the image to identify the light pattern encoding information associated with the pairing of the light-emitting user input device and a wearable device; extracting information encoded in the light pattern to pair the light-emitting user input device with the wearable device; pairing the light-emitting user input device with a wearable device based at least in part on the extracted information; and a hardware processor programmed to perform the A wearable device comprising: (Item 91) The hardware processor further comprises: determining whether pairing between the light-emitting user input device and the wearable device is successful; and instructing the light emitting user input device to illuminate another light pattern indicating successful pairing in response to determining that the pairing was successful; Item 91. The wearable device of item 90, programmed to: (Item 92) Item 91. The wearable device of item 90, wherein the hardware processor is further programmed to communicate with the light-emitting user input device via a wireless connection in response to determining that the pairing is successful. (Item 93) Item 93. The wearable device of item 92, wherein the wireless connection includes a Bluetooth® connection. (Item 94) Item 91. The wearable device of item 90, wherein the information encoded by the light pattern includes device information for the light-emitting user input device. (Item 95) Item 91. The wearable device of item 90, wherein the light pattern encodes the information in binary form. (Item 96) Item 91. The wearable device of item 90, wherein the light pattern includes one or more colors that encode information associated with the pairing. (Item 97) Item 91. The wearable device of item 90, wherein a portion of the light pattern includes light in the non-visible portion of the electromagnetic spectrum. (Item 98) Item 90. The wearable system of item 90, wherein the wearable device comprises a head-mounted display for presenting virtual content within a mixed reality environment, and the outward-facing imaging system comprises a camera mounted on the head-mounted display. (Item 99) 1. A method for pairing light emitting user input devices, the method comprising: Under the control of a hardware processor, Initiating a pairing process between the light emitting user input device and the electronic device; accessing an image acquired by a camera, the image including a light pattern illuminated by the light emitting user input device; identifying the light emitting user input device to be paired with the electronic device; analyzing the image to identify the light pattern encoding information associated with the pairing of the light emitting user input device and the electronic device; extracting information encoded in the light pattern to pair the light-emitting user input device with the electronic device; pairing the light-emitting user input device with the electronic device based at least in part on the extracted information; and A method comprising: (Item 100) Item 100. The method of item 99, wherein the electronic device includes a component of a wearable system for presenting virtual content in a mixed reality environment. (Item 101) Item 101. The method of item 100, wherein the electronic device includes another user input device or a head-mounted display. (Item 102) 99. The method of claim 99, further comprising establishing a wireless connection between the light-emitting user input device and the electronic device in response to determining that the pairing process was successful. (Item 103) Item 99. The method of item 99, wherein the information encoded by the light pattern includes device information for the light-emitting user input device, the device information including at least one of a device identifier, identification information about the light-emitting user input device, or a key for pairing the light-emitting user input device with the electronic device. (Item 104) Item 104. The method of item 103, wherein the light pattern encodes the information in binary form. (Item 105) Item 100. The method of item 99, wherein the light pattern includes one or more colors that encode information about the pairing process. (Item 106) 100. The method of claim 99, wherein the portion of the light pattern includes light in the non-visible portion of the electromagnetic spectrum. (Item 107) 1. A system for pairing light emitting user input devices, the system comprising: a plurality of light emitting diodes (LEDs) configured to output a light pattern; A hardware processor, Initiating a pairing process with an electronic device; causing one or more LEDs of the plurality of LEDs to illuminate a first light pattern that encodes information regarding the pairing process; responsive to determining that the electronic device has been successfully paired, illuminating a second light pattern indicating that the pairing was successful; a hardware processor programmed to perform the A system comprising: (Item 108) Item 108. The system of item 107, wherein the first light pattern encodes at least one of device information associated with the user input device or a trigger message that causes another computing device to initiate a pairing process. (Item 109) The hardware processor includes: receiving a response from the electronic device during the pairing process; causing a second one or more LEDs of the plurality of LEDs to illuminate a third light pattern encoding a reply message in response to the response; Item 108. The system of item 107, programmed to: (Item 110) Item 108. The system of item 107, wherein the illumination of one or more of the plurality of LEDs encodes information about the pairing process in binary form. (Item 111) Item 108. The system of item 107, wherein the color associated with the illumination of the one or more LEDs encodes information about the pairing process. (Item 112) Item 108. The system of item 107, wherein a portion of the first light pattern or the second light pattern includes light in a non-visible portion of the electromagnetic spectrum. (Item 113) 1. A method comprising: Under control of a hardware processor of a first electronic device comprising a light emitting assembly for illuminating a plurality of light patterns; initiating communication between the first electronic device and a second electronic device; causing the first electronic device to illuminate a light pattern encoding a message for the communication; receiving a response from the second electronic device; providing an indication to a user of the first electronic device based at least in part on a response from the second electronic device; and A method comprising: (Item 114) Item 114. The method of item 113, wherein the communication includes a pairing process between the first electronic device and the second electronic device. (Item 115) Item 115. The method of item 114, wherein the first light pattern encodes at least one of device information associated with the user input device or a trigger message that causes another computing device to initiate a pairing process. (Item 116) Item 115. The method of item 114, wherein the indication includes a second light pattern illuminated by the first electronic device in response to determining that the first electronic device and the second electronic device have been successfully paired, indicating that pairing between the first electronic device and the second electronic device has been completed. (Item 117) Item 114. The method of item 113, wherein the indication includes a third light pattern illuminated by the first electronic device that encodes a reply message in reply to the response. (Item 118) Item 114. The method of item 113, wherein the first light pattern encodes information in binary form within the message. (Item 119) Item 114. The method of item 113, wherein the light pattern includes one or more colors that further encode the message of the communication. (Item 120) 1. A light emitting user input device, comprising: 1. A touchpad assembly configured to receive user input, the touchpad assembly comprising: A touch surface and a touch sensor coupled to the touch surface, at least a portion of the touch sensor underlying the touch surface and configured to detect actuation of the touch surface; a light emitting assembly coupled to the touch sensor and the touch surface, the light emitting assembly including an optical diffusing element and a plurality of light illuminating elements, the light emitting assembly configured to display a plurality of light patterns; a printed circuit board coupled to the light emitting assembly and the touch sensor; a touchpad assembly comprising: a body for supporting the touchpad assembly; 1. A light emitting user input device comprising: (Item 121) Item 121. The luminous user input device of item 120, wherein at least a portion of the optical diffusing element is overlaid on the touch sensor, and the touch sensor is further configured to detect actuation of the optical diffusing element. (Item 122) Item 121. The light-emitting user input device of item 120, wherein the optical diffusing element comprises an optical waveguide. (Item 123) Item 121. The luminous user input device of item 120, wherein the optical diffusing element substantially surrounds the touch surface. (Item 124) Item 121. The illuminating user input device of item 120, wherein the touchpad assembly further comprises an armature for holding the light emitting assembly, the touch surface, and the touch sensor. (Item 125) Item 121. The light emitting user input device of item 120, wherein the plurality of light illumination elements comprise light emitting diodes. (Item 126) Item 121. The luminous user input device of item 120, further comprising a connection interface configured to establish a wireless connection with a wearable device configured to present mixed reality content. (Item 127) Item 121. The illuminated user input device of item 120, wherein the body has an upper portion for supporting the touchpad assembly and a bottom portion configured to be removably mounted to a base. (Item 128) Item 121. The luminous user input device of item 120, wherein the body further includes at least one of a trigger, a bumper, or a home button for user interaction. (Item 129) Item 121. The light emitting user input device of item 120, further comprising a hardware processor programmed to control the illumination of the plurality of light illumination elements.

[0006] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario with a virtual reality object and a physical object viewed by a person.

[0008] [Figure 2] FIG. 2 illustrates diagrammatically an example of a wearable system.

[0009] [Figure 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0010] [Figure 4] FIG. 4 illustrates diagrammatically an embodiment of a waveguide stack for outputting image information to a user.

[0011] [Figure 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0012] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, image, or bright field.

[0013] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0014] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0015] [Figure 9] FIG. 9 is a block diagram of another embodiment of a wearable system.

[0016] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0017] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0018] [Figure 12A] FIG. 12A illustrates a side and front (user-facing) view of an embodiment of a totem.

[0019] [Figure 12B] FIG. 12B illustrates a top view of another embodiment of a totem.

[0020] [Figure 13A] FIG. 13A illustrates a cross-sectional view of an example touchpad of the totem.

[0021] [Figure 13B] FIG. 13B illustrates an example of touchscreen technology.

[0022] [Figure 13C] 13C and 13D illustrate additional cross-sectional views of an exemplary touchpad of the totem. [Figure 13D] 13C and 13D illustrate additional cross-sectional views of an exemplary touchpad of the totem.

[0023] [Figure 13E] FIG. 13E illustrates a bottom view of an exemplary touchpad.

[0024] [Figure 14A] FIG. 14A illustrates a top view of an example touchpad of the totem.

[0025] [Figure 14B] FIG. 14B illustrates an overview of an example layout of LEDs associated with a touchpad.

[0026] [Figure 15]FIG. 15 illustrates an example LED layout or the pattern of light from an LED layout.

[0027] [Figure 16A] 16A and 16B illustrate exemplary locations or movement patterns of the totem's light emissions. [Figure 16B] 16A and 16B illustrate exemplary locations or movement patterns of the totem's light emissions.

[0028] [Figure 17A] FIG. 17A is a block diagram illustrating exemplary components of a totem.

[0029] [Figure 17B] FIG. 17B is a side cross-sectional view illustrating components of another exemplary totem.

[0030] [Figure 18A] 18A-18D illustrate an example programming interface for configuring location or movement patterns for light emissions from a totem's halo. [Figure 18B] 18A-18D illustrate an example programming interface for configuring location or movement patterns for light emissions from a totem's halo. [Figure 18C] 18A-18D illustrate an example programming interface for configuring location or movement patterns for light emissions from a totem's halo. [Figure 18D] 18A-18D illustrate an example programming interface for configuring location or movement patterns for light emissions from a totem's halo.

[0031] [Figure 19A] 19A-19C illustrate an example of totem calibration using the halo of light emission from the totem. [Figure 19B]19A-19C illustrate an example of totem calibration using the halo of light emission from the totem. [Figure 19C] 19A-19C illustrate an example of totem calibration using the halo of light emission from the totem.

[0032] [Figure 19D] 19D and 19E illustrate an example of totem calibration using a light pattern associated with a halo. [Figure 19E] 19D and 19E illustrate an example of totem calibration using a light pattern associated with a halo.

[0033] [Figure 20A] 20A and 20B illustrate an example showing the wireless pairing process between a mixed reality device and a totem with a halo. [Figure 20B] 20A and 20B illustrate an example showing the wireless pairing process between a mixed reality device and a totem with a halo.

[0034] [Figure 20C] FIG. 20C illustrates an exemplary process of device pairing with a halo.

[0035] [Figure 20D] FIG. 20D illustrates another exemplary process of device pairing with a halo.

[0036] [Figure 21-1] FIG. 21A illustrates an example showing the status of a totem.

[0037] [Figure 21-2] FIG. 21B illustrates an example of a light location or movement pattern during the power on and off process.

[0038] [Figure 21-3]Figure 21C illustrates an example of a light location or movement pattern indicating battery charge status. Figure 21D illustrates an example light pattern when the totem enters sleep mode.

[0039] [Figure 21-4] FIG. 21E illustrates an exemplary process for indicating the status of a totem based on light location or movement patterns.

[0040] [Figure 22A] 22A and 22B illustrate exemplary light locations or movement patterns used as cues for user interaction. [Figure 22B] 22A and 22B illustrate exemplary light locations or movement patterns used as cues for user interaction.

[0041] [Figure 22C] FIG. 22C illustrates another example of using light patterns to provide an indication of available user interface actions.

[0042] [Figure 23] FIG. 23 illustrates an example of using light patterns as an alert to indicate incorrect or inappropriate user interaction.

[0043] [Figure 24A] FIG. 24A illustrates an example light pattern for a swipe gesture.

[0044] [Figure 24B] FIG. 24B illustrates an example light pattern for a touch gesture.

[0045] [Figure 24C] FIG. 24C illustrates an example process for providing cues for user interaction on a totem.

[0046] [Figure 25A] FIG. 25A illustrates an exemplary interactive use of an optical waveguide.

[0047] [Figure 25B] FIG. 25B illustrates an exemplary interactive use of a totem with two interactable areas.

[0048] [Figure 25C] FIG. 25C illustrates an exemplary process for interacting with a totem.

[0049] [Figure 26A] 26A and 26B illustrate an example of using a totem to interact with a physical object. [Figure 26B] 26A and 26B illustrate an example of using a totem to interact with a physical object.

[0050] [Figure 27] FIG. 27 illustrates an example of using a six degree of freedom (6DOF) totem to move a virtual object.

[0051] [Figure 28A] 28A and 28B illustrate an example that provides information about an object through the location and movement of a light pattern. [Figure 28B] 28A and 28B illustrate an example that provides information about an object through the location and movement of a light pattern.

[0052] [Figure 28C] FIG. 28C illustrates an exemplary process for providing information associated with an object using light patterns.

[0053] [Figure 29A] FIG. 29A illustrates an exemplary light location or movement pattern indicating receipt of a notification.

[0054] [Figure 29B] FIG. 29B illustrates an example process for providing notification using light patterns on a totem.

[0055] [Figure 30] FIG. 30 illustrates an example light pattern that may be used to inform people in the user's environment of the user's current interaction.

[0056] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0057] Overview of Illuminated User Input Devices Touch-sensitive user input devices can support user inputs such as swipes, taps, clicks, and presses. For example, when a user uses a touchpad to browse a website, the user can use one finger (e.g., thumb) to swipe left and right to move the web page left and right, or tap up and down to move the web page up and down. To achieve more user interface functionality, such as snapping content to a desired location, scrolling, or resizing content, touch-sensitive user input devices often require multiple fingers. For example, a user may use two fingers to zoom in on a web page and one finger to navigate within the web page. However, when the touchpad is part of a handheld user input device for a wearable device (which may include an AR / VR / MR wearable display), the user may not have as many fingers available to interact with the touchpad. For example, a user may use their thumb to interact with the touchpad while using another finger to hold a totem. As a result, the types of user interface functionality that can be achieved with touchpads on conventional handheld devices may be significantly reduced.

[0058] To ameliorate this problem, the touchpad of the totem embodiments described herein can be divided into multiple interactable regions, and each region may be mapped to one or more types of user interface interactions. For example, the touchpad may include a touch surface near the center of the touchpad and an outer region at least partially surrounding the touch surface. The outer region may include a light guide configured to output a light pattern (e.g., light location, illumination, color, and / or movement) that assists the user in interacting with the totem. The light pattern output by the light guide may sometimes be referred to herein as a "halo" because the light pattern may appear to surround the central touch-sensitive portion of the totem. The light guide may be above the touch sensor so that the user can interact with the light guide and provide touch-sensitive input to the totem via the light guide region. When the user activates the touch surface, the totem may simulate a cursor action (e.g., moving forward or backward on a browser). However, if the user desires to perform other types of user interface actions (e.g., scrolling a web page, etc.), the user may activate the light guide. In some embodiments, the light guide may not be touch-sensitive. The user may activate an area near the light guide (e.g., an area on the touchpad that may be surrounded by the light guide) to activate the totem.

[0059] As another example, a touch surface can be divided into two interactable regions, one region supporting user touch actions (e.g., simulating the functionality of a multi-degree-of-freedom (DOF) directional D-pad) while the other region supports user swipe actions. A touch surface with two interactable regions may include concentric rings, with one inner ring (as the first interactable region) and one outer ring (as the second interactable region). In this example, optical waveguides surrounding the touch surface may be interactable or not, but may provide visual feedback related to the user's interaction or related to the wearable system. As will be further described herein, the type of user interaction supported by the interactable regions may be dynamically changed based on events in the virtual environment or objects the user interacts with. For example, the outer region may be used as a D-pad when the user is browsing the web, while the same region may support swipe interactions (e.g., circular swipes) when the user is playing a virtual game.

[0060] In some embodiments, the totem's light guide can provide cues for user interaction. For example, a halo can be used to inform the user of the type and location of available user interactions or to indicate a current user interaction. As an example, a light-emitting diode (LED) below the light guide may illuminate to indicate to the user that the user can touch or tap a portion of the touch surface, where the LED will illuminate and select a virtual element on the wearable display. The LED may also be used in conjunction with haptic feedback (e.g., provided by a haptic actuator in the totem) or audio feedback (e.g., provided by a speaker in the wearable device) to provide an indication of or guide user interaction.

[0061] When a totem is used in conjunction with a wearable device, the user interface experience can be extended to the 3D environment surrounding the user. However, the user's field of view (FOV) perceived through the wearable display may be smaller than the natural FOV of the human eye or smaller than the entire environment surrounding the user. Thus, there may be physical or virtual objects in the user's environment that are initially outside the FOV of the augmented reality display but may subsequently move into the FOV of the wearable display (e.g., objects that may move relative to the user) or may subsequently become perceptible if the user's body, head, or eye posture changes (which would change the user's FOV). For example, in a gaming context, a user may be trying to find a robot avatar. If the robot is just outside the user's current FOV, the user may not receive a cue from the wearable display that the robot is nearby. If the user moves their head slightly, the robot may suddenly enter the user's FOV, which may startle the user. Furthermore, if the user's FOV through the wearable display is relatively small, it may be difficult for the user to see the robot unless the user turns their head or looks directly at the robot.

[0062] To improve the user interface experience, a totem may provide information about objects outside the user's FOV. For example, a totem may provide a visual halo (e.g., emitted via a light guide) on an outer region of the touchpad for a corresponding object outside the user's current FOV. The light location or movement pattern of the halo may be used to indicate information associated with the object; for example, a brighter or larger halo may indicate that the object is closer to the FOV, while a dimmer or smaller halo may indicate that the object is farther from the FOV. Similarly, the color of the halo may be used to indicate the type of object. For example, a competitor avatar (in a virtual game) may be associated with a red halo, while a companion avatar (in a virtual game) may be associated with a green halo. As another example, a flashing rainbow halo may indicate a system notification or warning. The light pattern of the halo may change as objects in the user's environment change or as the user changes posture.

[0063] Additionally or alternatively, the light pattern of the halo may be used to indicate the progress of a process. For example, while the totem is charging, the totem may display a halo corresponding to the percentage of battery charge. For example, when the battery is only 25% charged, the totem may display a quarter of the halo (e.g., a 90-degree arc). When the battery is 100% charged, the totem may display the full halo. The light location or movement pattern of the halo may also provide an indication of the user's interaction to persons in the user's environment. For example, when a user is recording video using an augmented reality device, the LED halo may flash red to let nearby others know that the display is in recording mode, so that they will not accidentally interrupt or interfere with the user's recording session.

[0064] The totem embodiments described herein may be programmable. For example, the location or movement of the light pattern may be customized by an application developer or by the user in various embodiments. The halo may be customized (e.g., via an application programming interface (API)) based on the type of application the user is interacting with. As an example, the light guide of a touchpad may be mapped to a four-way D-pad (corresponding to up, down, left, and right user interface actions) when the user is using a browser. As another example, the outer area of ​​the touch surface may be mapped to a three-way D-pad when the user is playing a racing game, where the three-way D-pad may correspond to turn left, turn right, and brake. The user may also customize the light location or movement pattern of the halo. For example, the user can turn the halo off or change the color of the halo associated with receiving an email notification. The user can customize the light pattern associated with the halo using the wearable display or by activating the totem. Detailed examples of totems, halos, and user interface interactions using the totems and wearable devices are described below.

[0065] Although the exemplary totem is described as being used in conjunction with a wearable system (or any type of AR, MR, or VR device), the exemplary totem and techniques described herein can also be used in conjunction with other systems. For example, the totem may be used to interact with a projector or display (e.g., a television or computer display), a gaming system, an audio system, an Internet of Things (IoT) connectable device, or another computing device.

[0066] Example of a 3D display for a wearable system A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. A wearable system may include a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. A wearable device may include, for example, a wearable display device such as a head-mounted display (HMD). A wearable device may also include a belt pack, which may include a central processing unit that handles some of the data processing for the wearable device, a battery, etc. In some circumstances, a wearable device may be used synonymously with an augmented reality device (ARD).

[0067] Figure 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, a building in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives as "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying, cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0068] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it is desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by convergence and stereoscopic binocular depth cues, the human eye may experience accommodation conflict, resulting in unstable imaging, adverse eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0069] VR, AR, and MR experiences can be provided by a display system having a display that provides a viewer with images corresponding to multiple depth planes. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of the scene located on different depth planes and / or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0070] 2 illustrates an example of a wearable system 200. The wearable system 200 may include a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 may be positioned directly in front of the eyes of the user 210. A portion of the wearable system (such as the display 220) may be worn on the user's head.

[0071] In FIG. 2 , speaker 240 is coupled to frame 230 and positioned adjacent the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). Wearable system 200 may also include an outward-facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. Wearable system 100 may also include an inward-facing imaging system 462 (shown in FIG. 4 ) that may track the user's eye movements. The inward-facing imaging system may track either one eye movement or both eyes movement. The inward-facing imaging system may be attached to frame 230 and may be in electrical communication with processing modules 260 and / or 270 that may process image information obtained by the inward-facing imaging system and determine, for example, pupil diameter and / or orientation of user's 210's eyes, eye movement, or eye posture.

[0072] As an example, wearable system 200 can obtain images of a user's posture using outward-facing imaging system 464 and / or inward-facing imaging system 462. The posture may be used to determine the user's movement or to synthesize an image of the user. Images obtained by outward-facing imaging system 464 and / or inward-facing imaging system 462 may be communicated to a second user in a telepresence session to create a tangible sense of the user's presence within the second user's environment.

[0073] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired or wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0074] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., which may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in the inward-facing and / or outward-facing imaging systems), microphones, inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, and / or gyroscopes), and / or b) data obtained and / or processed using the remote processing module 270 and / or the remote data repository 280, possibly for passage to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to the remote processing module 270 and / or the remote data repository 280 by communication links 262 and / or 264, such as via wired or wireless communication links, so that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other. The local processing and data module 260, the remote processing module 270, and the remote data repository 280 may each include a network interface to provide for communication via the communication links 262, 264.

[0075] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0076] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that viewers of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence of the two eyes relative to one another (i.e., pupil rotation such that the pupils move toward or away from one another, converging the eyes' lines of sight and fixating on an object) is closely linked to the focusing of the eye's lenses (or "accommodation"). Under normal conditions, a change in the focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a coordinated change in vergence at the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a coordinated change in accommodation under normal conditions. Display systems that provide better alignment between accommodation and vergence-divergence movements may produce more realistic and comfortable simulations of three-dimensional images.

[0077] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3 , objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume particular accommodated states, focusing objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length such that an object or portion of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different representations of an image for each of the eyes 302 and 304, and by providing different representations of an image corresponding to each of the depth planes. While shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Additionally, while shown as flat for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular state of accommodation. Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes.

[0078] Waveguide Stack Assembly FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4 diagrammatically illustrates several portions of wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated into display 220 of FIG. 2.

[0079] 4, the waveguide assembly 480 may also include multiple features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers and / or structures to form air gaps).

[0080] Waveguides 432b, 434b, 436b, 438b, 440b and / or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits the output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide, outputting an entire field of cloned collimated beams directed toward eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0081] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0082] A controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 460 may, in some embodiments, be part of processing modules 260 and / or 270 (shown in FIG. 2 ).

[0083] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguides by redirecting the light, causing it to propagate within each individual waveguide, and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective and / or diffractive optical feature. While shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments, the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material attached to a transparent substrate and forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic pieces of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the material pieces.

[0084] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to create a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next upper waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another, increasing amount of wavefront curvature so that the eye / brain interprets the light emerging from the third upper waveguide 436b as originating from a second focal plane closer inward toward the person from optical infinity, which was the light from the next upper waveguide 434b.

[0085] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the collective power of the lower lens stacks 458, 456, 454, 452. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0086] Continuing with reference to FIG. 4 , light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with that waveguide. As a result, waveguides with different associated depth planes may have differently configured light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that can be configured to output light at specific angles. For example, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated herein by reference in its entirety.

[0087] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related output beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of output emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0088] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0089] In some embodiments, the number and / or distribution of depth planes or depths of field may be dynamically varied based on the pupil size and / or orientation of the viewer's eyes. The depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's pupil decreases, the depth of field increases so that a plane that is indistinguishable because its location exceeds the eye's depth of focus may become distinguishable and appear more focused with a corresponding decrease in pupil size and an increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a decreased pupil size. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting their eye's accommodation from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus on the user at different pupil sizes without changing accommodation.

[0090] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size and / or orientation or in response to receiving an electrical signal indicating a particular pupil size and / or orientation. For example, if a user's eye is unable to distinguish between two depth planes associated with two waveguides, controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing system responsiveness. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0091] In some embodiments, it may be desirable to have the output beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the output beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the output beam may also decrease. In some embodiments, the output beam size may be varied using a variable aperture.

[0092] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV), and the imaging system 464 is sometimes referred to as an FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400. In some implementations of the wearable system 400, the FOR may include substantially all of the solid angle around the user of the display system 400, as the user can move their head or eyes to see objects around them (in front of, behind, above, below, or to the sides of the user). Images obtained from the outward-facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0093] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements, such as eye and facial movements. The inward-facing imaging system 466 may be used to capture images of the eyes 410 and determine the size and / or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size and / or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter and / or orientation of only one eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye posture and / or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU (e.g., accelerometer, gyroscope, etc.).

[0094] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, haptic device, totem, smartphone, smartwatch, tablet, etc., combinations thereof, or the like. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A user can interact with the user input device 466 or objects (e.g., virtual or physical objects) in its environment by, for example, clicking a mouse, tapping a touchpad, swiping a touchscreen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a five-way D-pad), pointing a joystick, wand, or totem toward an object, pressing a button on a remote control, or other interaction with the user input device. Actuation of the user input device 466 may cause the wearable system to perform user interface operations, such as displaying a virtual user interface menu associated with the object, animating the user's avatar in a game, etc. As described herein, the user input device 466 may be configured to emit light. The light patterns may represent information associated with an object in the user's environment, the user's interaction with the user input device 466 or the wearable device, etc.

[0095] In some cases, a user may use a finger (e.g., a thumb) to press or swipe a touch-sensitive input device to provide input to the wearable system 400 (e.g., provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 via wired or wireless communication. The user input device 466 may comprise an embodiment of a totem described herein. The totem may include a touch surface that may allow a user to activate the totem, such as by swiping along a trajectory or tapping.

[0096] FIG. 5 shows an example of an output beam output by a waveguide. While one waveguide is illustrated, it should be understood that other waveguides in waveguide assembly 480 may function similarly, and that waveguide assembly 480 may include multiple waveguides. Light 520 is launched into waveguide 432b at input edge 432c of waveguide 432b and propagates within waveguide 432b by TIR. At the point where light 520 impinges on DOE 432a, a portion of the light exits the waveguide as output beam 510. While output beams 510 are illustrated as being approximately parallel, they may also be redirected to propagate to eye 410 at an angle (e.g., divergent output beam formation) depending on the depth plane associated with waveguide 432b. It should be understood that a nearly collimated exit beam may refer to a waveguide with light-extracting optics that outcouples light to form an image that appears to be set at a depth plane at a large distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light-extracting optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0097] FIG. 6 is a schematic diagram illustrating an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, an image, or a bright field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal stereoscopic display, an image, or a bright field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, and 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (the vertical or Y-axis in the illustration of FIG. 6 ) and expand the effective exit pupil of the light along the first axis (e.g., the Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622 b and at least one DOE 622 a (illustrated by a double-dashed line) associated with the dispersive planar waveguide 622 b. The dispersive planar waveguide 622 b may be similar or identical in at least some respects to a primary planar waveguide 632 b having a different orientation therefrom. Similarly, the at least one DOE 622 a may be similar or identical in at least some respects to the DOE 632 a. For example, the dispersive planar waveguide 622 b and / or the DOE 622 a may be made of the same material as the primary planar waveguide 632 b and / or the DOE 632 a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0098] The relayed, exit-pupil-expanded light is optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b preferably relay the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is orthogonal to the first axis. Notably, the second axis can be non-orthogonal to the first axis. The primary planar waveguides 632b expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b can relay and expand the light along the vertical or Y-axis and pass the light to a primary planar waveguide 632b that relays and expands the light along the horizontal or X-axis.

[0099] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as a free-standing, flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0100] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend piezoelectric tube 642 independently in two axes. The protruding distal tip of optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of optical fiber 644. By oscillating piezoelectric tube 642 near the first mechanical resonant mode of fiber cantilever 644, fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0101] By stimulating resonant vibrations in two axes, the tip of fiber cantilever 644 is scanned biaxially within an area filling a two-dimensional (2-D) scan. By modulating the intensity of light source 610 synchronously with the scanning of fiber cantilever 644, light emitted from fiber cantilever 644 forms an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0102] Components of the optical coupler subsystem collimate the light emitted from the scanning fiber cantilever 644. The collimated light is reflected by a mirrored surface 648 into a narrow dispersive planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light propagates perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b by total internal reflection (TIR), and in doing so, repeatedly intersects with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This diffracts a portion of the light (e.g., 10%) toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, allowing a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 622b via TIR.

[0103] At each point of intersection with DOE 622a, additional light is diffracted toward the entrance of primary waveguide 632b. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light is vertically expanded by DOE 4 within dispersive planar waveguide 622b. This vertically expanded light outcoupled from dispersive planar waveguide 622b enters the edge of primary planar waveguide 632b.

[0104] Light entering the primary waveguide 632b propagates horizontally (with respect to the view of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects the DOE 632a at multiple points. The DOE 632a advantageously has a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, and may be designed or configured to produce both deflection and focusing of the light. The DOE 632a advantageously may have a low diffraction efficiency (e.g., 10%) so that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0105] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a level of focus to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles that match the designed level of focus.

[0106] Thus, these different paths can couple light out of the primary planar waveguide 632b by providing different fill patterns at the DOE 632a's multiplicity, focal level, and / or exit pupil at different angles. Different fill patterns at the exit pupil can be advantageously used to generate bright-field displays with multiple depth planes. Each layer in the waveguide assembly or set of layers (e.g., three layers) in the stack may be employed to generate distinct colors (e.g., red, blue, and green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image bright-fields with various focal depths. Other Components of a Wearable System

[0107] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a haptic sensation of pressure and / or texture upon touching virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptic sensation may replicate the sensation of a physical object represented by the virtual object, or may replicate the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the haptic device or component may be held by the user.

[0108] A wearable system may include, for example, one or more physical objects that can be manipulated by a user to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as selections or inputs made via the virtual keyboard and / or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of a totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0109] Examples of haptic devices and totems that can be used in conjunction with the wearable devices, HMDs, ARDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0110] Exemplary Wearable Systems, Environments, and Interfaces The wearable system may employ various mapping-related techniques to achieve a high depth of field within the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this goal, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features in the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. A first user's world model can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0111] 7 is a block diagram of an example MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from a user's wearable system, stationary input 704 such as a room camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from user input device 466) from various user systems 720a, 720b. The user systems 720a, 720b may comprise one or more user wearable systems (e.g., wearable system 200 and / or display system 220) and / or stationary room systems (e.g., room cameras, etc.). The wearable systems can determine the location and various other attributes of the user's environment using various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.). This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives and / or various cues. Image data acquired by a camera (such as an indoor camera or an outward-facing imaging system camera) may be reduced to a set of mapping points.

[0112] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize and / or map the points, tag images, and associate semantic information with objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., a LAN, a WAN, etc.) and accessible to the cloud.

[0113] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0114] Object recognition may be performed using various computer vision techniques. For example, the wearable system may analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms, etc. (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.).

[0115] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variables. In some embodiments, the wearable HMD can be configured to generate models for analysis of aggregated data using multiple techniques. Other techniques may include the use of predefined thresholds or data values.

[0116] The wearable system can also complement recognized objects with semantic information, bringing them to life. For example, if the object recognizer recognizes that a set of points is a door, the system may associate some semantic information (e.g., a door has a hinge and 90-degree movement around the hinge). If the object recognizer recognizes that a set of points is a totem, the wearable system may associate semantic information that the totem can be paired with the wearable system (e.g., via Bluetooth). Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene taking place in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, the object recognizer and other software components can map points collected from various images, recognize objects, etc. so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0117] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 describes how a virtual scene can be presented to a user of an MR system (e.g., a wearable system). The user may be geographically remote from the scene. For example, a user may be in New York but may want to view a scene currently occurring in California, or may want to go for a walk with a friend who resides in California.

[0118] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc., communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used to determine pose data (e.g., head pose, eye pose, body pose, and / or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizer 708a, 708n may crawl through these collected points and recognize one or more objects using the map database in block 830. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user appropriately in block 850. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0119] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 includes a map, which may include map data about the world. The map may reside partially locally on the wearable system and partially in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and moves within its world. The data may include images, data from sensors (such as inertial measurement devices, which generally include accelerometer and gyroscope components), and surface information about objects in the real or virtual environment.

[0120] The sparse point representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is only image / vision) process. The system can be configured to find not only the location of various components in the world, but also what the world is made of. Poses can be building blocks that accomplish many goals, including capturing in and using data from maps.

[0121] In one embodiment, the sparse point locations may not be entirely correct by themselves, and additional information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill in this gap, at least in part. Such information may be calculated from a process referred to as stereoscopic vision 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, and some of this may be summarized using a surface representation. For example, mathematically definable surfaces are efficient (e.g., for large point clouds) and easy-to-summarize inputs to other processing devices, such as game engines. Thus, the outputs of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. Pose may also be an input to this fusion process 930, the output of which is input to the map capture process 920. Sub-surfaces can interconnect to form larger surfaces, such as in topographic mapping, where the map becomes a large-scale hybrid of points and surfaces.

[0122] Various inputs may be utilized to resolve various aspects of the mixed reality process 960. For example, in the embodiment depicted in Figure 9, game parameters may be input to determine that a user of the system is playing a monster battle game with one or more monsters at various locations, that monsters die or flee under various conditions (such as when the user shoots them), walls or other objects at various locations, and the like. A world map may contain information about where such objects are relative to one another, which is another useful input to mixed reality. Attitude relative to the world is likewise an input and plays an important role for nearly any interaction system.

[0123] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, etc. To move around or play a game, for example, the user may need to command the wearable system 900 regarding what they want done. There are various forms of user control that can be utilized beyond just moving around in space. In one embodiment, an object such as a totem, another user input device, or a toy gun may be held by the user and tracked by the system. The system will preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU, which can help determine what is happening even when such activity is not within the field of view of any of the cameras).

[0124] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture for button presses, left or right, stop, grasp, hold, etc. For example, in one configuration, a user may wish to flip through email or calendar in a non-gaming environment or perform a “fist bump” with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as extending the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0125] Eye tracking is another input (e.g., to track where the user is looking and control display technology to render at a specific depth or range). In one embodiment, eye vergence movement may be determined using triangulation, and then accommodation may be determined using a vergence movement / accommodation model developed for that particular person.

[0126] With respect to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a relative wide-FOV or passive SLAM pair of cameras arranged on the sides of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and capture hand gestures and totem / object tracking in front of the user's face. The cameras in the three pairs of cameras may be part of the outward-facing imaging system 464 (shown in FIG. 4). The wearable system 900 may also include an eye-tracking camera (which may be part of the inward-facing imaging system 462 shown in FIG. 4) oriented toward the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0127] 10 is a process flow diagram of an example embodiment of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may designate one totem for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect movement of the totem. Movement of the totem may be recognized through the user's FOV camera or detected through sensors (e.g., haptic gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0128] Based at least in part on the detected gestures, eye postures, head postures, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a frame of reference in block 1020. The frame of reference may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. In block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user's interactions to the frame of reference 1020, the system determines the user input in block 1040.

[0129] For example, a user may move a totem or physical object back and forth to indicate turning a virtual page, advancing to the next page, or moving from one user interface (UI) display screen to another. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, the real or virtual object may be selected as user input. In some implementations, the user's eye vergence / divergence movement can be tracked, and an accommodation / vergence / divergence movement model can be used to determine the user's eye accommodation state, which provides information about the depth plane the user is focusing on. In some implementations, the wearable system can use ray casting techniques to determine real or virtual objects along the direction of the user's head pose or eye pose. In various implementations, ray casting techniques can include casting a narrow beam of light with substantially no lateral width or casting a ray of light with a substantial lateral width (e.g., a cone or frustum).

[0130] The user interface may be projected by a display system (such as display 220 in FIG. 2) as described herein. It may also be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a globe. Interactions with the user interface may be tracked using one or more cameras external to the system or part of the system (e.g., using inward-facing imaging system 462 or outward-facing imaging system 464, etc.).

[0131] 11 is a process flow diagram of an example method 1100 for interacting with a virtual user interface. Method 1100 may be implemented by a wearable system described herein.

[0132] In block 1110, the wearable system may identify a specific UI. The type of UI may be predetermined by the user. The wearable system may identify that a specific UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for the virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. Additionally, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical posture, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through an FOV camera, data received through sensory input, or any other type of collected data.

[0133] In block 1130, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a bright field display can project the virtual UI into one or both of the user's eyes. Once the virtual UI is created, the wearable system may simply wait for a command from the user to generate additional virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (such as a gesture, head or eye movement, input from a user input device, etc.) and, if recognized (block 1160), virtual content associated with the command may be displayed to the user (block 1170).

[0134] Additional examples of wearable systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0135] Overview of exemplary totems As described with reference to Figures 4 and 7-10, a user can use a user input device 466 (e.g., a totem, etc.) to perform various user interface operations on a display (e.g., display 220). Figure 12A illustrates an exemplary embodiment of a totem 1200, illustrating a side view 1210 and a front (facing the user) 1250. The totem 1200 may be the user input device 466 (shown in Figure 4), alone or in combination with other user input devices. The totem 1200 can be sized and shaped to be handheld. Further examples of user input devices (e.g., totems, etc.) in one or more of the various embodiments disclosed herein are shown and described in U.S. Patent Application No. 15 / 683,677, filed August 22, 2017, and have an ornamental appearance identical to or similar to the totem controller in U.S. Design Patent Application No. 29 / 575,031, filed August 22, 2016 (both of the foregoing applications are incorporated herein by reference in their entirety).

[0136] The totem 1200 shown in FIG. 12A can include a body 1214 having a trigger 1212, a home button 1256, and a touchpad 1260, although more or fewer buttons, triggers, or features can be included in other exemplary totems. A light guide 1264 can substantially surround the touchpad 1260. In the example shown in FIGS. 12A and 12B, the light guide 1264 is generally annular (e.g., a circular ring) that substantially surrounds the circular touchpad 1260. In other embodiments, the touchpad 1260 and the light guide 1264 can be shaped differently. For example, the touchpad 1260 can be polygonal (e.g., square, rectangular, hexagonal, etc.) or oval, and the light guide can have a shape (e.g., circular, oval, polygonal, etc.) that substantially surrounds the touchpad.

[0137] The trigger, home button, and touchpad can receive user input (e.g., by being pulled, pressed, or touched, respectively). The light guide 1264 can be illuminated and display light patterns. In some embodiments, the light guide 1264 is touch-sensitive and can receive user input. The totem 1200 can be removably attached to the base 1220 when not being held by a user. The base 1220 may include a power connection (e.g., to a wall socket), which can be used to charge the totem 1200 when the totem is attached to the base 1220.

[0138] Trigger Example The trigger 1212 can be located on an upper portion of the totem body 1214 facing away from the user. The trigger 1212 can include a touch surface 1234 and a touch sensor (not shown in FIG. 12A ) that can receive user input. The touch sensor can sense a user's finger on (or near) the touch surface 1234 and the movement of the user's finger on the touch surface 1234. Additionally or alternatively, the trigger 1212 can include a button 1232 that the user can press. The button 1232 can include a pressure sensor that can detect when the user presses the button 1232. The button 1232 can be pressed 6-8 mm from its rest position. In some embodiments, the trigger 1212 can include multiple buttons, such as select, back, options, etc.

[0139] The trigger 1212 may be implemented using, for example, an Alps force sensor. The trigger 1212 (or button 1232) can also be implemented using an analog or digital button. The trigger 1212 may be associated with a microprocessor. The microprocessor may reserve two general-purpose input / output (GPIO) pins for various functions of the trigger 1212.

[0140] In some implementations, the trigger 1212 can be configured to provide haptic feedback. The trigger 1212 can include a haptic actuator, such as a linear resonant actuator (LRA), an eccentric rotating mass (ERM), a piezoelectric actuator, etc. For example, the trigger 1212 can employ an S-type LRA (e.g., an S-type Alps haptic on the analog button 1232, etc.) to generate vibrations.

[0141] A user can activate the trigger 1212 using various hand gestures. For example, the user can activate the trigger 1212 by touching, swiping, tapping, pressing, etc. The trigger 1212 can provide rich touch features, and various user interface interactions may be associated with different hand gestures used to activate the trigger 1212. For example, a user can switch between an AR interface and a VR interface by pressing the trigger 1212. As another example, a user can swipe a virtual object in and out of the user's FOV by swiping the touch surface 1234 of the trigger. User interface interactions can also be associated with the duration of the actuation. The totem 1200 can record the duration of the press and determine a user interface operation based on the duration of the press. For example, pressing the trigger 1212 for a long duration (e.g., 3-5 seconds) may cause the wearable system to end a program (e.g., a movie), while a quick press of the trigger may cause the wearable system to select a virtual object in the user's line of sight.

[0142] The trigger may be actuated in conjunction with other components of the totem or the user's posture to perform user interface actions. For example, a user may swipe the touchpad 1260 while pressing the trigger 1212 to move a virtual object. As another example, a user may press the trigger 1212 with their right thumb and move their arm to the right, moving a virtual object to the right.

[0143] The trigger 1212 can also be configured to provide an indication to the user that one or more user interface actions are available. For example, when the trigger 1212 is pressed for a long duration, the wearable system may allow the user to change a setting in the game. To indicate to the user that a setting can now be changed, the wearable system may provide haptic feedback (e.g., vibration, etc.) on the trigger 1212 (or the body of the totem). In addition to, or as an alternative to, providing an indication that a user interface action is available, haptic feedback can also be used to inform the user that they have activated the trigger 1212 using a gesture (e.g., tap, press, etc.).

[0144] In other embodiments, some or all of the above-described functionality of the trigger 1212 can be implemented by other buttons or touch-sensitive surfaces of the totem 1200. For example, as described below with reference to FIG. 17B, a button (also referred to as a bumper) can perform some or all of the functionality of the trigger 1212 (or home button 1256).

[0145] Another embodiment of the totem 1200 is described with reference to Figures 17A and 17B.

[0146] Touchpad example The totem may include a touchpad 1260 located on an upper front portion of the totem body 1214 facing toward the user. The touchpad 1260 may include a touch surface 1262 and a light guide 1264. In some embodiments, the structure and / or functionality of the touchpad 1260 may be substantially the same as or similar to that described in U.S. Patent Application No. 15 / 683,677 (as noted above, which is incorporated herein by reference in its entirety). In the illustrated embodiment, the light guide 1264 substantially surrounds the touch surface 1262. The touchpad 1260 may include various interactable areas. For example, a user may activate the touch surface 1262, the light guide 1264, alone or in combination. The touch surface 1262 may further include multiple interactable areas, each mapped to a type of user input (or user interface action).

[0147] The touch surface 1262 may be a circular surface with a diameter in the range of 27 mm to 40 mm. The touch surface 1262 may also be other shapes, for example, oval, rectangular, triangular, diamond, irregular, etc. In some implementations, the touchpad 1260 may have a traffic-to-pilot (T2P) latency of less than 50 ms.

[0148] The touch surface 1262 can be substantially surrounded by a light guide 1264. For example, in various embodiments, the light guide 1264 may span an angular distance around the touch surface of more than 90 degrees, more than 180 degrees, more than 270 degrees, or up to 360 degrees. The light guide 1264 can be illuminated to display a halo with various location and moving light patterns. The light guide 1264 can diffuse light generated by the touchpad's light sources 1330 (e.g., LEDs) so that illumination from the individual discrete light sources blends to display a halo. The light guide 1264 can comprise a diffusing optical element formed in a ring shape, for example, from a plastic or polymer material, which can transmit (and diffuse) light from the light sources 1330 to a viewer of the totem 1200 (see, e.g., FIGS. 13A-14A ). The light guide can be transparent or translucent. In various embodiments, the light guide 1264 can comprise a diffuser sheet, a diffuser film, an etched waveguide, a transmissive optical element with a layer of particles, an irregular surface, a holographic, a white surface, polished glass, polytetrafluoroethylene (PTFE or Teflon), opal glass, gray glass, a colored gel, etc. Light guide embodiments comprised of optically diffusing materials can advantageously diffuse and scatter light from the light sources 1330 such that the light guide appears to have an overall substantially continuous iridescence (when all light sources are illuminated) rather than appearing as discrete individual light sources (when the light guide is substantially transparent).

[0149] The touchpad may include several red, green, and blue (RGB) LEDs (e.g., 6-24 RGB LEDs, etc.). The light location or movement pattern of the halo may provide a visual indication of the user's interaction with the totem or other components of the wearable system, progress status associated with the wearable system, objects in the user's environment, etc. See, for example, LED 1383a in FIG. 13C or LED 1394a in FIG. 13E.

[0150] The light source may (in addition or alternatively) include emitters that irradiate in non-visible portions of the electromagnetic spectrum, e.g., infrared or ultraviolet. In such embodiments, a camera on the wearable device may be sensitive to the corresponding non-visible spectrum and can image the non-visible light displayed by these emitters. Thus, the totem 1200 and the wearable device can exchange information via such non-visible spectrum modalities. For example, the totem 1200 and the HMD can exchange device pairing information using non-visible spectrum modalities.

[0151] The touchpad 1260 can include a haptic component beneath the touch surface 1262 and a portion of the light guide 1264. The haptic component can include a haptic actuator for providing haptic feedback to the user via the touch surface 1262 or the light guide 1264. The haptic feedback can be used alone or in combination with a visual halo to provide an indication of the user's interaction with the totem or an object in the user's environment.

[0152] The haptic component may also include a touch detector for detecting user actuation of the touchpad. The haptic component may be implemented using a tough-type Alps LRA. The haptic component may include a strain gauge with an analog-to-digital converter (ADC).

[0153] The touchpad 1260 can employ hybrid control techniques, such as, for example, a TrackPoint-style hybrid velocity control input technique, a hybrid force and velocity technique, etc. The touchpad 1260 can use an ADC to enable such input techniques. In some implementations, the sensitivity of the touchpad can be less than 5 Newtons. Additional structural examples of the touchpad 1260 are described with further reference to FIGS. 13A and 14E.

[0154] The touchpad 1260 can provide a rich set of touch features for a variety of user interface experiences. For example, a user can use the touchpad 1260 (alone or in conjunction with other components of a totem or wearable system) to move or orient virtual objects. The touchpad 1260 can also be used as a virtual keyboard to enter text.

[0155] Example of a totem body The body 1214 of the totem 1200 may be shaped like a flashlight (see, e.g., FIG. 12A ), with an elongated cylinder that may be angled outward toward the top of the totem to create a more frusto-conical shape (which may be easier or more comfortable to hold in a user's hand). The body 1214 may include indentations for the user's fingers, which may aid in gripping and holding the totem. FIG. 12B is a top view 1270 illustrating another example of the totem 1200. In this embodiment, the body 1214 is oval-shaped so that it may fit better in a user's palm or be more stable when placed on a surface (e.g., a table). In some embodiments, the decorative appearance of the body 1214 of the totem 1200 may be substantially the same as or similar to that of the totem controller shown and described in U.S. Design Patent Application No. 29 / 575,031, filed August 22, 2016 (as noted above, which is incorporated herein by reference in its entirety). Other shapes for the body 1214 are also possible, such as an elongated body having an ovular, polygonal, etc. cross-section. The cross-section of the body 1214 may be substantially symmetrical so that the totem 1200 can be easily held in a user's left hand or a user's right hand.

[0156] The body 1214 can include an upper portion 1242, which can include a trigger 1212 and a touchpad 1260. The body 1214 can also include a bottom portion 1244. The bottom portion 1244 can include an interface such that the totem 1200 can be removably attached to the base 1220. For example, the bottom portion 1244 can be connected to the base 1220 via a connection interface 1222 (e.g., a USB-C connection, etc.) when the totem 1200 is being charged. As another example, the bottom portion 1244 can be connected to another computing device (such as a home computer) via the base 1220. When connected to a computer, a user can configure the light location or movement pattern of the halo (on the touchpad 1260) as appropriate via a programming interface on the computing device (described below with reference to FIGS. 18A-18D ).

[0157] The touchpad 1260 can be angled so that the touch surface 1264 and the light guide 1262 are easily viewable by a user when the totem 1200 is held in the user's hand. The angle of the touchpad 1260 with respect to the horizontal can be in the range of approximately 10 to 60 degrees. Such an angle allows the body 1214 to assume a more frusto-conical shape. As explained above, this shape can make the totem 1200 easier or more comfortable for a user to hold in their hand. In some embodiments, the angle of the touchpad 1260 with respect to the horizontal can be in the range of approximately 15 to 30 degrees. The trigger 1212 can be positioned opposite the touchpad 1260 so that it can be easily depressed with the user's index finger when the totem 1200 is held in the user's hand.

[0158] In some embodiments, the bottom portion 1244 can include an illuminated power indicator. The illuminated power indicator can include one or more RGB LEDs. The light location or movement pattern of the LED power indicator can be used to provide information related to the remaining battery or battery charge status for the LED totem.

[0159] The body 1214 can be configured to provide haptic feedback to the user. The body 1214 can include a haptic actuator 1760 (shown in FIGS. 17A and 17B) for providing the haptic feedback. The haptic actuator 1760 may be implemented with a tough-type LRA or ERM haptic. Details of the components associated with the body 1214 are described with reference to FIGS. 17A and 17B.

[0160] Home button example The totem 1200 may include a home button 1256. The home button 1256 may be located on the body 1214 of the totem 1200. When a user activates the home button 1256, the wearable system may perform various user interface actions, such as returning to / opening a main menu of an application or display, exiting a program, selecting an application, etc. In some embodiments, when a user activates the home button 1256, the wearable system may enter or wake up from a sleep mode, turn virtual content on the display on or off, etc.

[0161] The wearable system may perform an action based on how the user activates the home button 1256. For example, if the user activates the home button 1256 for a long duration, the display may be turned off. As another example, if the user presses the home button 1256 twice, the display may present the main page of the display, while if the user presses the home button once, the display may present the main menu of the gaming application with which the user is currently interacting. Other functionality can also be implemented by the home button 1256; for example, a quick tap of the button can invoke an application launcher user interface; within an application, a quick tap may invoke the application's menu, or a long press may pause the application and invoke the application launcher user interface.

[0162] The home button 1256 may include an RGB LED indicator that may be used to indicate the current status of the totem or wearable display (such as whether it is turned on / off), the battery status of the totem, the wearable display, or other components of the wearable system.

[0163] In various embodiments, the totem 1200 includes both the home button 1256 and the trigger 1212, or includes only one of these user input elements (or an additional button, such as the bumper 1266 described with reference to FIG. 17B). The totem 1200 can be configured or shaped differently than shown. For example, the home button 1256 may be implemented as part of the trigger (rather than as part of the body 1214). As yet another example, the touchpad 1260 or the home button 1256 can provide a visual indication as to the progress of charging the totem 1200. Many variations are possible, and the accompanying figures illustrate example embodiments and are not intended to limit the scope of the present disclosure.

[0164] Example of base As described with reference to the totem's body 1214, the body 1214 of the totem 1200 may be removably attached to the base 1220. An interface between the base and the totem can allow charging, access data within the totem, provide software updates, etc. For example, the base 1220 can include a Universal Serial Bus (USB) plug, which can be plugged into a USB port on the bottom portion 1244 of the body 1214. The base 1220 can further be connected to an electrical outlet using a power cord to charge the totem. The base can also connect to other computing devices, such as personal computers, televisions, projectors, etc., using wired or wireless communication channels.

[0165] Example of activating a totem The totem 1200 can support a variety of input techniques, such as touch, swipe, scroll, dial rotate, click, press, D-pad emulation, etc. In some embodiments, the totem 1200 can support inertial swipes. For example, as a user swipes along a trajectory on the touchpad 1260, a virtual object may continue to move along the trajectory even when the user's finger stops swiping. The totem 1200 can also support edge swipes, such as a swipe on a horizontal or vertical edge of the touchpad 1260 or light guide 1264.

[0166] A user can actuate the totem 1200 using various hand gestures and body postures. For example, the user can swipe or touch the touchpad 1260 or trigger 1212. In some embodiments (such as embodiments 1210 and 1250), the totem 1200 can support 6 DOF of user interaction. For example, the user can rotate the totem 1200 and rotate a game avatar. As further described with reference to FIGS. 17A and 17B, the totem 1200 can detect user movement using environmental sensors 1710 (such as an IMU). In other embodiments (such as the example totem 1200 shown in FIG. 12B), the totem can support 3 DOF (rather than 6 DOF). In these embodiments, the totem can detect and translate the user's forward, backward, sway, and heave movements, but not their pitch, yaw, and roll movements, or vice versa.

[0167] Although the examples are described with reference to handheld devices, similar techniques may also be applied to user input devices designed for seated and tabletop control.

[0168] Exemplary Touchpad Structures Touchpad cross section 13A illustrates a cross-sectional view of an example touchpad 1260 of the totem 1200. The example touchpad 1260 may be part of the totem 1200 described with reference to FIGS. 12A and 12B. The touchpad 1260 may include three layers: a top layer 1312, a middle layer 1314, and a bottom layer 1316.

[0169] The bottom layer 1316 can include a printed circuit board (PCB) 1340. The PCB 1340 can mechanically support and electronically connect to one or more components on the middle layer 1314, such as light sources 1330 (e.g., LEDs) and touch sensors 1350. The PCB 1340 can also support the armature 1320. Additionally or alternatively, the PCB 1340 can electronically connect to other components of the totem, such as, for example, haptic actuators (e.g., actuators for the touchpad 1260, trigger 1212, or body 1214, etc.), a microprocessor, etc.

[0170] The central layer 1314 of the touchpad 1260 can include a touch sensor 1350. The touch sensor 1350 may be an example of the haptic component described with reference to FIG. 12A . The touch sensor 1350 can use various touchscreen technologies (e.g., resistive or capacitive sensor technologies, etc.) to determine user actuation of the touchpad 1260 or to provide tactile feedback. The touch sensor 1350 can be configured to enable multi-touch technologies that can distinguish between different levels of force applied to the touch surface 1262 (and optical waveguide 1264). The touch sensor 1350 can provide sensitivity in three orthogonal directions (e.g., X, Y, Z). Exemplary touchscreen technologies that can be employed by the touch sensor 1350 are illustrated in FIG. 13B .

[0171] In FIG. 13A, the LEDs 1330 can be positioned between the touch sensor 1350 and the armature 1320. The center layer 1314 can include 6 to 12 LEDs, although other numbers (e.g., 3, 15, 18, 20, etc.) are also possible. The LEDs can be positioned substantially uniformly around the circumference of the light guide (see, e.g., the top views in FIGS. 13E and 14A), or some LEDs may be positioned closer to each other rather than uniformly spaced. The LEDs 1330 may comprise, alone or in combination, monochromatic LEDs (e.g., blue-violet LEDs), bichromatic LEDs, RGB LEDs (or other multicolor LEDs), white LEDs, organic LEDs (OLEDs), quantum dot LEDs, infrared LEDs, ultraviolet LEDs, etc. The LEDs 1330 may be controlled by a microprocessor (e.g., on a PCB board 1340). For example, by using a microprocessor to independently adjust each color (red, green, and blue) of the RGB LED, a wide range of colors (e.g., a large color gamut) can be produced by the RGB LED. The light patterns produced by the LEDs can be associated with user interaction with the totem or wearable system and with objects in the user's environment. In some examples, the light patterns can also be referred to as halo light locations or movement patterns.

[0172] The armature 1320 may straddle the top layer 1312 and the middle layer 1314. The armature 1320 may hold the middle layer 1314 and the top layer 1312 together. A portion of the armature may touch a top portion of the LED 1330.

[0173] The top layer may include a portion of the armature 1320 as well as a light guide 1264 and a touch surface 1262. The touch surface 1262 may be surrounded by the light guide 1264 and may sit on top of the touch sensor 1350. A user can activate the touch surface 1262 using hand gestures. The touch surface 1262 may also provide tactile feedback to the user. A portion of the light guide 1264 may be overlaid on top of the touch sensor 1350, allowing the user to activate the light guide 1264 accordingly. In some embodiments, the light guide may not be touchable (e.g., because the touch sensor 1350 is not below the light guide 1264). However, the light guide may still provide visual or tactile feedback.

[0174] The light guide 1264 may be between the armature 1320 and the touch surface 1262 on the top layer 1312. The light guide 1264 may include a first portion 1362 and a second portion 1364. The first portion 1362 of the light guide 1264 may be overlaid on top of the LEDs 1330. The light guide 1264 can diffuse the light generated by the LEDs 1330, distributing the light over a larger area of ​​the light guide 1264 than an individual LED package, which may provide a more appealing visual appearance. The light patterns can provide visual feedback associated with a user's interaction with the wearable system or touchpad 1260. The light patterns can also provide information associated with objects in the user's environment, such as the relative position between the user and the object.

[0175] A second portion 1364 of the light guide 1264 may be positioned over the touch sensor 1350. For example, the light guide 1264 may be positioned 0.75 mm to 2 mm inset over the touch sensor 1350. A portion of the touch sensor 1350 may extend below the light guide 1264 so that the light guide 1264 can not only provide visual and tactile feedback, but also be activated by a user. As shown by finger 1310, a user can activate the light guide 1264 by, for example, touching, tapping, swiping, or depressing the light guide 1264. Thus, when a user activates the light guide 1264, the totem can simulate a user interface experience, such as selectively tapping an illuminated “button” or “arc” of light on the light guide.

[0176] The light guide 1264 does not need to be positioned over the touch sensor 1350. As a result, a user may not be able to activate the light guide 1264 and interact with the totem or wearable device. In some implementations, the second portion 1364 is positioned over the touch sensor 1350, but the processor 1770 or local processing and data module 260 may be configured not to recognize a user's input on the light guide 1264. As a result, the light guide 1264 may be used to provide visual or tactile feedback, but may not be responsive when a user activates the light guide 1264.

[0177] 13B illustrates examples of touchscreen technology. In these examples, touch sensors 1350 (e.g., 1350a, 1350b, and 1350b) are sandwiched between touch surface 1262 and PCB 1340. Touchscreen technology embodiment 1362 illustrates an example of force under touch technology. In this example, touch sensor 1350a can comprise a pressure-sensitive sensor that can detect multiple levels of force applied on touch surface 1262. In some embodiments, touch sensor 1350 can also be configured to relay tactile feedback to touch surface 1262.

[0178] In example 1364, touch sensor 1350b may include a strain gauge. The strain gauge may be located under the edge of touch surface 1262. When a user activates touch surface 1262, touch sensor 1350b can determine the user's gesture (e.g., a press or tap gesture) by measuring the strain on touch surface 1262.

[0179] Example 1366 illustrates an interpolated force-sensitive resistive touch sensor technology (IFSR). The touch sensor 1350c can use force-sensitive resistors, whose level of sensitivity can change depending on the level of pressure. Thus, the touch sensor 1350c can detect multiple levels of force applied on the touch surface 1262.

[0180] 13C and 13D illustrate additional cross-sectional views of an exemplary touchpad of the totem. Cross-section 1380 shows top cover 1381a and top housing 1381b below top cover 1381a. Either top cover 1381a or top housing 1381b, or a combination thereof, may be part of the armature 1320 shown in FIG. 13A. The top housing may be attached to the light guide 1264 (e.g., light pipe 1381c) with an adhesive. The light pipe can be transparent or translucent. In various embodiments, the light pipe 1381c can comprise a diffuser sheet, a diffuser film, an etched waveguide, a transmissive optical element comprising a layer of particles, an irregular surface, a holographic, a white surface, ground glass, polytetrafluoroethylene (PTFE or Teflon), opal glass, gray glass, a colored gel, etc. Light pipe embodiments made of optically diffusing material may advantageously diffuse and scatter the light from LED 1383a so that the light output in direction 1384 appears to have an overall substantially continuous glow (when all LEDs are illuminated) rather than appearing as discrete individual LEDs (if the light pipe were substantially transparent).

[0181] The top housing 1381b can also be attached to white Mylar 1381c (or other reflective sheeting) using adhesive. The top housing can encapsulate one or more LEDs 1383a. For example, the top housing may include RGB LEDs, which include packages of red, green, and blue LEDs. Thus, if a touchpad includes 12 RGB LEDs, the touchpad may include 12 top housing structures, each including an RGB LED. As another example, the top housing may include all 12 RGB LEDs for the touchpad.

[0182] The top cover is adjacent to a touch cover 1383d. The touch cover may be an embodiment of the touch surface 1262 shown in FIG. 13A. The touch cover may be attached to a touch board 1383 using touch cover adhesive 1383c. The touch board includes an embodiment of the haptic component described with reference to FIG. 12A. As further described with reference to FIG. 13D, an adhesive stack 1386 may be interposed between a portion of the touch board and a portion of the top housing. The adhesive stack 1386 may be used to securely attach the portion of the touch board and the top portion of the top housing.

[0183] The adhesive stack 1386 may include two layers: a shock-absorbing foam pad (e.g., Poron, available from Rogers Corp., Rogers, CT) and a flexible and adhesive layer. The top portion of the Poron pad 1387a may be attached to the bottom portion of the touchboard, and the bottom portion of the Poron pad may be attached to the flexible and adhesive layer 1387b. The flexible and adhesive layer may be attached to the upper housing. The Poron pad may consist of two 0.2 mm thick adhesives and one 1.00 mm thick adhesive. The one 1.00 mm thick adhesive may be positioned between the two 0.2 mm thick adhesives. Alternatively, one of the two 0.2 mm thick adhesives may be between the 1.00 mm thick adhesive and the other 0.2 mm thick adhesive.

[0184] Referring back to FIG. 13C , one or more LEDs 1383 a may be mounted on the touchboard. The LEDs may emit light in direction 1382. The light may reach a light pipe, which, through total internal reflection (TIR), may transmit the light to light output direction 1384. The bottom portion of the light pipe may also be mounted on a section of white Mylar. The white Mylar (or other reflective sheeting) may reflect some of the light emitted by the LEDs, increasing the amount of light output in light output direction 1384. The light pipe may diffuse the light from the LEDs so that the light output appears as more of an angular continuum rather than individual, discrete light sources.

[0185] Touchpad bottom view FIG. 13E illustrates a bottom view of an exemplary touchpad. Touchpad 1390 may be an embodiment of touchpad 1260. Touchpad 1390 may include 12 LEDs 1394a, although other numbers of LEDs are also possible. The 12 LEDs 1394a are equally spaced circumferentially in this example, although in other examples (such as the examples described with reference to FIGS. 14B and 15), the spacing between the LEDs may not necessarily be equal. The 12 LEDs may be located on the bottom of the touchpad. The touchpad can selectively illuminate the LEDs to create a light pattern, which may appear as a halo around the touchpad.

[0186] The LEDs can be mounted in a housing 1392. The housing can have two alignment pin holes and three snap gaps 1394e. The two alignment pin holes can be on opposite ends of the touchpad diameter. The three snap gaps can be equally spaced. The alignment pin holes 1394d and snap gaps 1394e can be used to orient and position the touchpad relative to the totem body. The snap gaps 1394e can also be used to secure the touchpad 1390 to the totem body.

[0187] The touchpad 1390 can employ touchscreen technology such as, for example, a force flex stack 1394b. A portion of the force flex stack 1394b can be overlaid on top of the touchboard component area 1394c. The force flex stack 1394b can include the touch surface 1262, either alone or in combination with the touch sensor 1350. The force flex stack can detect user inputs on the touchpad 1390, such as presses, swipes, taps, touches, etc.

[0188] In one embodiment, the touch board component area may include PCB 1340. The touch board component area may include haptic components (e.g., haptic actuators, etc.), a microprocessor, one or more communication units (e.g., ADCs, etc.), etc.

[0189] Touchpad top view Figure 14A illustrates a top view of an example touchpad of the totem. The touchpad 1260 in Figure 14A can include an armature 1320, a light guide 1264, and a touch surface 1262 (which can be further divided into interactable areas 1262a and 1262b). The light guide 1264 can be positioned on top of a light source (e.g., an LED) 1330 (shown in dashed lines in Figure 14A).

[0190] Advantageously, in this embodiment, the touchpad 1260 can be divided into three interactable areas: a light guide 1264, a first portion 1262a of the touch surface, and a second portion 1262b of the touch surface. As described with reference to Figures 12 and 13A, each interactable area may be mapped to a type of user interaction. For example, a user can touch the light guide 1264 and move it back and forth on a browser while swiping the second portion 1262b of the touch surface to move web content up and down. A user can also swipe in a circular motion on the first portion 1262a of the touch surface to move a virtual object closer to (or farther away from) the user.

[0191] While this example includes three interactable areas, the totem may include more or fewer interactable areas in other embodiments. For example, rather than dividing the touch surface into two interactable areas (corresponding to first portion 1262a and second portion 1262b), the touch surface may be divided into four interactable areas, each occupying a quadrant of the touch surface. The touch surface may also include only one interactable area. Additionally or alternatively, the light guide 1264 may not be interactable because, for example, the processor 1770 or another module of the wearable system may not be programmed to recognize user input from the light guide 1264 or the touch sensor 1350 does not extend below the light guide 1264.

[0192] The type of interaction mapped to the touch area may also be customized based on the type of user interaction. For example, when a user is watching television, the touchpad may simulate a four-way D-pad (up, down, left, right) on light guide 1264. However, when a user is browsing a web page, light guide 1264 may support back / front user interface actions.

[0193] The interactable areas can also provide visual or tactile feedback to the user. For example, the touch surface 1262 can be configured to provide tactile feedback, while the light guide 1264 can provide both tactile and visual feedback (e.g., via the light location or moving pattern of an LED halo).

[0194] Light source layout FIG. 14B illustrates an example layout of light sources (e.g., LEDs) associated with a touchpad. FIG. 14B shows a touchpad 1450 having a visible circular surface 1452 with 12 LEDs (illustrated as LED0, LED2...LED11) on its periphery. These 12 LEDs can be used to represent specific information when they are illuminated in a specific sequence. For example, each LED may be assigned a bit, thus resulting in a 12-bit word. As described with reference to FIGS. 24B, 20A, and 20B, an outward-facing imaging system 464 of the wearable device can capture the display sequence of the LEDs. The wearable device can then analyze the image and extract information from the display sequence. The extracted information may be used, for example, to pair a totem with the wearable device, calibrate the totem, etc. For example, when LEDs 0, 3, 6, and 9 are illuminated, the wearable device may recognize that pairing between the wearable device and the totem has begun. Thus, the wearable device can search for wireless (e.g., Bluetooth®) devices in its environment and determine whether any new devices (such as a totem associated with an LED) are present. When the LEDs are multi-LEDs (e.g., RGB LEDs), the amount of information represented by the LED light pattern can increase (e.g., greater than a 12-bit word). For example, a blue light on LEDs 6, 7, 8 can indicate that the totem is in pairing mode, while a yellow light on LEDs 6, 7, 8 can indicate that the totem is in charging mode.

[0195] In some embodiments, the LEDs may be divided (e.g., multiplexed) into groups that are illuminated separately. Each group may be used to represent specific information. FIG. 14B shows four example groups (illustrated as MUX0 (1454a), MUX1 (1454b), MUX2 (1454c), and MUX3 (1454d)), each group containing three LEDs. For example, when the LEDs in MUX1 are illuminated (e.g., LEDs 3, 4, and 5), the wearable device may recognize that the totem is currently indicating an upward movement of the totem, while when the LEDs in MUX0 are illuminated, the wearable device may recognize that the totem is being rotated left, and so on.

[0196] In other embodiments, different numbers or layouts of light sources or multiplexed groups can be utilized, and different amounts of information (e.g., word lengths different from 12 bits) may be represented by illuminated groups or sequences of LEDs.

[0197] As previously mentioned, with reference to FIG. 12A, the light source can emit light in the non-visible spectrum (e.g., infrared or ultraviolet light). Such light can be captured by the wearable device, and light patterns in the non-visible spectrum can also be used to communicate information from the totem to the wearable device. The information may include the totem's device information (e.g., for pairing the totem with the wearable device) or other types of information, such as the totem's status (e.g., whether it is low on battery, whether a connection has been established, the device currently paired with the totem, etc.).

[0198] FIG. 15 illustrates an example LED layout or light pattern from an LED layout. In this example, the LEDs are essentially installed with reference to, for example, a north-east-south-west (NESW) coordinate system 1592, such that when the totem is held in a user's hand, north may point away from the user, south may point toward the user, east may point toward the user's right, and west may point toward the user's left. The coordinate system 1592 may be rotated to some extent. For example, the layout in column 1550 is essentially installed with reference to a neutral position, and the coordinate system 1592 is not rotated. In contrast, the layouts in columns 1570 and 1580 are rotated 15 degrees clockwise from the base position shown in the NESW coordinate system 1592.

[0199] The LED layout may be user-adjustable. For example, the touchpad (or light guide) on the totem may be mechanically rotatable. As an example, the LED layout of a totem may initially include a 15-degree clockwise rotation for a right-handed user. However, when a left-handed user uses the same totem, the left-handed user may rotate the touchpad 30 degrees counterclockwise, allowing them to better interact with the touchpad using their thumb. An exemplary layout in columns 1560 and 1580 may consist of 12 LEDs. These LEDs may be placed adjacent to each other. For example, 12 LEDs may be placed next to each other with no spacing between them to form a circular layout, as illustrated in Pattern I.

[0200] In some embodiments, two adjacent LEDs may be placed with a space between them (as illustrated by the patterns in columns 1550 and 1570). The space may be approximately the size of one LED. For example, pattern D shown in column 1550 and row 1540 may include six LEDs, placed approximately the same distance from each other. However, in some situations, to produce the patterns in columns 1550 and 1570, the totem may selectively turn off or not include one or more LEDs. For example, pattern D may include 12 LEDs (rather than 6 LEDs), and 6 of the 12 LEDs may not be illuminated in pattern D. In contrast, when a user changes user interface actions, more or fewer LEDs than 6 as shown in pattern D may be illuminated. Advantageously, in some embodiments, the totem can conserve battery consumption because fewer LEDs are used (with spacing between them) or because LEDs can be selectively turned off to achieve a desired lighting pattern (such as that shown in FIG. 15 ). Additionally, by spacing the illuminated LEDs apart from one another (either physically or by not illuminating some intermediate LEDs), the user may not need to move their thumb to a precise location to perform certain user interface actions, which can reduce user fatigue in some situations.

[0201] The LEDs may be divided into multiple groups (see also the exemplary multiplexed groups described with reference to FIG. 14B). Rows 1510, 1520, 1530, and 1540 of FIG. 15 illustrate exemplary ways of grouping LEDs. For example, layouts A and G in row 1510 include two groups of LEDs, each group represented by a different color. Layouts B and H in row 1520 include three groups, layouts C, D, and F in row 1530 include four groups, and layouts D and I include six groups. The LEDs in each group may have similar characteristics, such as similar light patterns, similar responses to user interaction (e.g., all illuminating simultaneously), etc. Additionally or alternatively, as described with reference to FIG. 14B, each group of LEDs may be used to represent specific information. For example, a wearable device may determine that a user may enter video recording mode when its outward-facing imaging system captures that one or more LEDs associated with the upper arc in pattern B are illuminating.

[0202] Advantageously, the LED layout may be aligned with the gripping direction of the totem. Layout AI (highlighted in black) is such an example. In these layouts, when a user grips the totem, they can reach groups of LEDs without necessarily having to adjust their hands significantly.

[0203] In some implementations, only a portion of the LEDs are user-interactable. LEDs may be interactable if the user can activate them (or the light guides associated with them). For example, pattern E includes four LED groups: an upper arc, a lower arc, a left arc, and a right arc, each group having three LEDs. The user may be able to activate the upper and lower arcs (e.g., by pressing the light guides above these arcs), but not the left and right arcs. However, the left and right arcs can still provide visual or tactile feedback, either alone or in combination with the upper and lower arcs. Similar considerations applied to other layouts are shown in FIG. 15.

[0204] In addition to, or as an alternative to, the physical locations of the LEDs, the patterns in FIG. 15 can be light patterns illuminated by the LEDs. For example, a totem can present pattern E by illuminating a ring of 12 LEDs, while presenting pattern D by illuminating every other LED in the 12-LED ring. As another example, rather than a user rotating a touchpad or light guide, a totem can be programmed to adjust the location of the halo based on whether the user is left-handed or right-handed. The totem may display an arc at the 1 o'clock position for a left-handed user, but the same arc at the 11 o'clock position for a right-handed user. The arc may also be associated with a user interface action, such as scrolling up a web page.

[0205] While many embodiments of user input devices are described in terms of the totem 1200, this is for purposes of illustration and not a limitation on the types of user input devices that can be used with the wearable system 200. For example, in other embodiments, a display screen (or variations thereof) may be used to display lighting shapes and patterns to the user. For example, in some of these embodiments, a smartphone may be utilized as the totem and may display various shapes and patterns on its screen. In some examples, the totem 1200 may include a display screen, which may be touch-sensitive (similar to the surface of a smartphone). In some such embodiments that include a display screen, the use of the illuminated light guide 1264 may be optional, as the display screen may be used to display the lighting patterns and transition sequences described herein.

[0206] Example of location and movement of light patterns 16A and 16B illustrate example locations or movement patterns of light emission from the halo of a totem (or other type of light-emitting user input device). As explained above, the halo may constitute an illuminated portion of the totem that surrounds the touchpad area. As illustrated in the embodiment shown in FIGS. 16A and 16B, the light guide substantially diffuses the light emitted by the individual light sources so that the halo appears as a ring or an arcuate portion of a ring, rather than as individual discrete light sources.

[0207] The location and movement of the light pattern that may be projected by the halo may include one or more characteristics, such as, for example, shape, color, brightness, position, size, movement, animation, other visual effects (e.g., blinking or flashing, fade-in, fade-out), etc. For example, FIG. 16A illustrates four example patterns. In pattern 1612, the halo is shown as light blue with a light pink at the 2 o'clock position. The halo in pattern 1614 has four colored arcs, namely, yellow, green, blue, and red, which correspond to the top, bottom, left, and right positions of the touchpad (respectively). Pattern 1616 shows a dark blue halo, and pattern 1618 shows a dark blue halo with a light blue arc at the top.

[0208] The light location or movement pattern of the halo can provide visual feedback to the user and persons in the user's environment, and may be determined based on contextual information, such as the user's environment, characteristics of the user, information associated with objects, processes, components of the wearable system, and the user's interactions with the wearable system.

[0209] As an example of generating a halo based on a user's environment, the environmental sensor 1710 (described below with reference to FIGS. 17A and 17B ) in the totem 1200 can be a light sensor that can detect whether the user is in a bright or dark environment. If the user is in a dark environment, the totem 1200 can display the halo in a lighter color (such as white) to help the user perceive the halo. On the other hand, if the user is in a bright environment, the totem 1200 can display a darker color to help the user distinguish the halo from the ambient light in the user's environment. As another example, in a dark room, the intensity of the light in the halo may be reduced (e.g., "night mode") because it is easier to perceive an illuminated halo in a dark environment. Conversely, in a bright environment (e.g., outdoors in sunlight), the intensity of the light in the halo may be increased so that the halo pattern is visible in a bright environment.

[0210] The pattern in the halo can also be based on user characteristics, such as the user's physiological data, demographic information (e.g., age, location, occupation, preferences, etc.). For example, when a user is playing a racing game, the totem can display a halo that corresponds to the user's driving direction. The color of the halo may initially be red. However, if the wearable system (e.g., environmental sensors on the totem or HMD, etc.) detects that the user's heart rate or breathing rate exceeds a threshold condition, the wearable system may determine that the user is in an emotionally elevated state. Thus, to calm the user's emotional state, the wearable system may change the color of the halo from red to blue (e.g., pattern 1618, where a light blue arc illustrates the driving direction).

[0211] Information associated with the object may include, for example, notifications or alerts associated with the object, characteristics of the object (e.g., function, type, location, shape, orientation, etc.), etc. As an example, the object may be a virtual email application displayed to the user by the HMD. A new email received by the virtual email application may be indicated to the user by a halo having a rainbow light pattern on the totem. As another example, the halo may be an indication of whether the object is perceptible in the user's environment. For example, in a treasure hunt game, the halo may correspond to the location of the treasure. If the treasure is located in a position directly in front of the user, the totem may display a halo at the 1 o'clock position of the light guide (e.g., see pattern 1612). As yet another example, if the object is not interactable by the user, the wearable system may be configured not to display a halo.

[0212] As described with reference to Figures 20A-21E, the halo can also be used to guide a process or indicate the status of a process. For example, the wearable system can apply computer vision algorithms to analyze the location of the halo in order to pair the totem with another device. The wearable system can also use computer vision algorithms to analyze the appearance of the light pattern and calibrate the totem in the user's environment. Once pairing or calibration status is complete, the totem may display pattern 1616 to indicate to the user that the process is complete.

[0213] The halo can also be used to indicate information associated with components of the wearable system. For example, the totem may be used to locate a battery pack used to provide power to the wearable device. In this example, the battery pack may include an electromagnetic emitter, while the totem may include an electromagnetic sensor. The electromagnetic sensor can detect the magnetic field generated by the electromagnetic emitter and, accordingly, calculate the relative position between the totem and the battery pack. The halo, as displayed by the totem, may correspond to the relative position between the totem and the battery pack (helping a user find the battery pack). For example, a halo with a large arc at the 3 o'clock position may indicate that the battery pack is near and to the right of the user. However, a halo with a small arc at the 3 o'clock position may indicate that the battery pack is farther away from the user but still to the right of the user. As another example, when the wearable device is low on power, the totem may display a halo with a small arc to indicate that the battery is running low. Although described in terms of a battery pack, any component of the wearable system or other (e.g., the user's car keys) can contain an electromagnetic emitter that can be tracked by the totem, which illuminates a halo and assists the user in finding the component.

[0214] The halo can also provide an indication of the user's interaction with the wearable system. Advantageously, this indication can inform people in the user's environment of the user's current interaction. For example, when the user is recording video, the halo may flicker red. Thus, someone next to the user would see the flickering red light pattern and know not to interrupt the user's video recording experience. As another example, when a user completes a level in a game, the halo may light up in the same color as the game level, indicating to the user's friends that the user has completed the level.

[0215] In addition to, or as an alternative to, visual feedback, the halo can guide user interaction. For example, the halo may show four arcs (up, down, left, and right) in different colors (as illustrated by effect 1614) to indicate that the user can use the totem's touchpad as a D-pad. As another example, while the user is watching a television program using display 220, the totem may present halo pattern 1662 in FIG. 16B. Pattern 1662 includes a red arc and a blue arc. The arc lengths for the red and blue arcs may be approximately 0.25π. However, when the user activates the totem (such as by tapping the totem's touch surface), the totem may present an arc fade-in effect (as illustrated by process 1 in FIG. 16B). As a result of the fade-in effect, the lengths of the red and blue arcs may increase from approximately 0.25π to 0.75π, as shown in pattern 1664. The brightness of the red and blue arcs may also increase due to a fade-in effect. This increased brightness and arc length may provide the user with a clearer indication that they can swipe left or right on the touch surface. However, if the totem detects that the user has not interacted with the totem for a threshold duration, the totem may present a fade-out effect, as illustrated by process 2 in FIG. 16B. Thus, the halo is changed from pattern 1664 to 1662. As a result, the size of the arc and the brightness of the arc will decrease. Advantageously, by reducing the size and brightness of the arc, the totem may reduce battery consumption when the user is not interacting with the totem.

[0216] In some embodiments, the totem 1200 may present a halo, provide contextual and associated information, or guide user interaction, in conjunction with tactile, auditory, visual, or other effects.

[0217] Other Components of the Totem 17A and 17B illustrate example components of a totem 1200. The example totem 1200 may include a touchpad 1260 (which may include a touch surface 1262 and a light guide 1264), a trigger 1212, and a totem body 1214, as described with reference to FIGS. 12A and 12B. The light guide 1264 may comprise a user-interactive area (e.g., touch-sensitive) and may at least partially or completely surround the touch surface 1262. The totem 1200 may also include various components, at least some of which may be disposed inside the totem's body 1214. These components are described further below and may include an environmental sensor 1710, a battery 1720, a near-field communication (NFC) interface 1732, a network interface 1734, a haptic actuator 1760, and a processor 1770. The connection interface 1222 is located on the bottom of the body 1214 and allows the totem 1200 to be removably attached to a base, for example. The connection interface 1222 can be used to provide power, charge the battery 1720, and provide a communications link between the components of the totem and an external device (e.g., a computing device).

[0218] Bumper Totem 1200, in the embodiment shown in FIG. 17B, may include a button 1266 (referred to as a bumper) located on the front edge of the totem, above trigger 1212 and below touchpad 1260. Bumper 1266 may provide an ergonomically comfortable place for a user to rest their index finger. Bumper 1266 may comprise a touch-sensitive surface, implemented, for example, as a depressible button, a capacitive touch sensor, a haptic element, or the like.

[0219] In the example shown in FIG. 17B , a user can primarily use three fingers to activate the totem 1200, e.g., using the thumb to activate the home button 1256 or touchpad 1260, the index finger to activate the bumper 1266, and the middle finger to activate the trigger 1212. Such a three-finger-activatable totem can enable a user to quickly and efficiently provide user input without excessive and fatiguing use of only one finger (as can occur with a mouse configuration for a desktop computer). The different buttons 1212, 1256, 1266 can provide different functionality depending, for example, on whether the user is working within an application, scrolling through an application launcher, selecting an object in the environment, etc. In some cases, a user may simply use their index finger to switch back and forth between pressing the bumper or pulling the trigger.

[0220] As examples of bumper functionality, tapping the bumper while the user is viewing an application can invoke an options menu for that application, while long pressing the bumper can activate a manipulation sequence for a virtual object in the environment. For example, long holding the bumper can grasp the object, and long pressing the bumper while pointing the totem 1200 toward the object can activate direct manipulation of the object (e.g., moving or reorienting the object). Tapping the bumper (or pulling the trigger) during a manipulation sequence can end the sequence.

[0221] Environmental Sensors The environmental sensors 1710 can be configured to detect objects, stimuli, people, animals, places, or other aspects of the user's surrounding environment. The environmental sensors may include image capture devices (e.g., cameras), microphones, IMUs, accelerometers, compasses, global positioning system (GPS) units, wireless devices, gyroscopes, altimeters, barometers, chemical sensors, humidity sensors, temperature sensors, external microphones, light sensors (e.g., light meters), timing devices (e.g., clocks or calendars), or any combination or subcombination thereof. In some embodiments, the environmental sensors may also include various physiological sensors. These sensors can measure or estimate physiological parameters of the user, such as heart rate, respiratory rate, galvanic skin response, blood pressure, brainwave state, etc. The environmental sensors may further include emitting devices configured to receive signals, such as lasers, visible light, invisible wavelengths of light, or sound (e.g., audible, ultrasonic, or other frequencies). In some embodiments, one or more environmental sensors (e.g., cameras or light sensors) may be configured to measure the ambient light (e.g., brightness) of the environment (e.g., to capture the lighting conditions of the environment). Physical contact sensors, such as touch sensors, strain gauges, curb detectors, or the like, may also be included as environmental sensors.

[0222] Information obtained by the environmental sensor 1710 may be used to determine the light location or movement pattern of the halo displayed on the totem. For example, the environmental sensor may use a GPS sensor or an electromagnetic sensor (to detect electromagnetic signals associated with the physical object) to determine the relative position between the user and a physical object in the user's environment. The totem may present a halo, the location of which may correspond to the location of the physical object. For example, if the object is directly in front of the user, the totem may present the halo at the 12 o'clock direction on the light guide 1264.

[0223] Additionally or alternatively, information obtained by the environmental sensors 1710 can be used for one or more user interface operations. For example, the totem 1200 can use an IMU to detect 6DOF movement of the totem. For example, as a user rotates the totem 1200 while playing a game, an avatar (or other virtual object) controlled by the user (and displayed to the user via a wearable device) can rotate accordingly based on the movement data obtained by the IMU. In addition to, or as an alternative to, moving or rotating the totem 1200, the user can provide input on the touchpad 1260. For example, movement of the user's thumb (e.g., along the long axis of the totem) toward or away from the user can cause a virtual object to move toward or away from the user. Moving the user's thumb back and forth laterally on the touchpad 1260 can scale the size of the virtual object (e.g., from a larger size to a smaller size, or vice versa), or rotating the user's thumb around the touchpad can rotate the virtual object.

[0224] In this example, the environmental sensors are located on the totem 1200, but in some embodiments, the environmental sensors may be located on other components of the wearable systems described herein. For example, the environmental sensors (such as cameras or physiological sensors) may be part of the display 220 of the wearable system 200 (shown in FIG. 2).

[0225] Battery The battery 1720 stores power for the totem 1200. The totem can use the processor 1770 to determine the current status of the battery 1720. For example, the processor 1770 can measure and calculate the amount of power remaining in the battery 1720, whether the battery 1720 is currently being used, and the remaining life of the battery 1720 (such as when the battery 1720 will need to be replaced). As further described in FIGS. 21A-21E, the current status of the battery 1720 may be visualized via visual feedback presented on the light guide 1264, the home button 1256, an LED power indicator located on the bottom portion 1244 of the totem body 1214 (e.g., shown in FIG. 12A), or a display.

[0226] The totem 1200 can also use the processor 1770 to determine the totem's power consumption level. For example, the totem can estimate power consumption based on an estimated current required to output a certain light signature. A light signature may include a haloed light location or movement pattern associated with an indication of a process, status, object, or user interaction. As an example of estimated current associated with a light signature, the totem 1200 may require 5.6 mA to perform a 3 or 6 DOF calibration, 48 mA to locate or indicate direction to an object in the user's environment, and 54 mA to perform wireless (e.g., Bluetooth) pairing. As another example of estimated current for the light signature, 14.85 mA of current may be consumed to provide an indication that the battery is below 15%, 21.6 mA may be used to indicate that the totem is in sleep mode or idle, and 72.4 mA of current may be supplied to provide incoming notification (e.g., new messages, etc.).

[0227] These estimated currents are examples only. The specific amount of power required may depend on various factors, such as the type of LEDs used, the color of the LEDs, and the location and movement of the halo light pattern. For example, the totem battery 1720 may be depleted as much as two hours faster if all RGB LEDs in the touchpad 1260 are set to constantly display white at 100% brightness instead of green at 100% brightness. This is because when the RGB LEDs display green, only one-third of the LEDs (e.g., the green LEDs in the RGB LEDs) are utilized, but when the RGB LEDs display white, all of the LEDs in the RGB LEDs are utilized.

[0228] To reduce overall power consumption, LEDs (particularly interactable LEDs) may be spaced to reduce the total number of LEDs on the touchpad. For example, a totem may employ the layout shown in columns 1550 and 1570 in FIG. 15. Advantageously, in some embodiments, by spacing out the LEDs, power consumption may be reduced by up to 50%. The specific amount of power consumption reduction may depend on the LED layout. For example, pattern D in FIG. 15 may have lower power consumption than pattern A because pattern D has fewer LEDs illuminated. As another example, to reduce power consumption, a totem may activate the EDs at a lower brightness (e.g., <40% brightness, etc.).

[0229] The totem can manage power consumption based on the object the user interacts with. The totem can turn off LEDs if it does not have an active user interface option. For example, the totem may simulate a D-pad using LED layout E shown in FIG. 15. However, during gaming, the down button may be disabled at some level. Thus, the totem may be configured not to display a down arrow. Additionally or alternatively, the totem can manage power consumption based on the status of the totem, the display, or other components of the wearable system. For example, the totem can turn off the halo after detecting a certain amount of inactivity. The totem can make such detection by calculating the amount of totem movement data obtained by the IMU.

[0230] In some embodiments, the wearable system may include a battery pack, which may be attached to the user (e.g., around the user's waist, etc.). The battery pack may be connected to and provide power to the totem or the wearable device. Battery 1720 may be part of the battery pack or may be used in connection with the battery pack to provide electricity to the totem 1200.

[0231] Near Field Communication (NFC) Interface and Network Interface The NFC interface 1732 and network interface 1734 can be configured to enable the totem to pair with or communicate with a target object, such as another totem (or other user input device), a wearable display, a personal computer, headphones, keys, a server, or another physical object. The NFC interface 1732 can be used for short-range wireless communication (such as when the totem is within 10 cm or less of the target object). In some embodiments, NFC employs electromagnetic induction between two loop antennas when NFC-enabled devices, such as a totem and an HMD, are positioned in close proximity to each other (within approximately 10 cm) and exchange information wirelessly. This information exchange may be operated within the globally available unlicensed radio frequency ISM band of 13.56 MHz at rates ranging from 106 to 424 Kbit / s via the ISO / IEC 18000-3 air interface. Various communication protocols and data exchange formats may be used for NFC. Some non-limiting examples include ISO / IEC 14443 and Felicia, ISO / ICE 18092, which are standards defined by the NFC Forum and the GSMA Group, among others.

[0232] The network interface 1734 can be configured to establish a connection and communicate with the target object over a network, which may be a LAN, a WAN, a peer-to-peer network, radio frequency, Bluetooth, Wi-Fi, a cloud-based network, or any other type of communication network.

[0233] For example, the totem and wearable device may be paired using a wireless communication protocol such as Bluetooth® Low Energy (BLE). BLE may be advantageous in some embodiments because it can preserve bandwidth during BLE audio streaming. As further described with reference to FIGS. 20A-20C, various techniques may be used to pair the wearable device with the totem. For example, the wearable system may detect certain light patterns presented by the totem by analyzing images of the totem acquired by an outward-facing camera. Detection of the light pattern may trigger the wearable system to begin the pairing process. During the pairing process, a user may interact with the wearable display via dialog boxes or a user interface (UI) wizard to define settings and parameters associated with the totem, the wearable display, the type of information to be shared between the totem and the wearable display, the type of communication channel, etc.

[0234] Although in this example, NFC interface 1732 and network interface 1734 are illustrated as separate components, in some embodiments, NFC interface 1732 and network interface 1734 may be part of the same communication device or system.

[0235] tactile actuator The totem 1200 may include a haptic actuator 1760. As described with reference to Figure 12A, the haptic actuator 1760 may provide haptic feedback to the user. One or more haptic actuators 1760 may be implemented within the totem 1200. The haptic actuator may be located in the trigger 1212, the touchpad 1260, or the totem body 1214.

[0236] Haptic feedback may be provided based on user interaction, contextual information associated with the object or user, the status of the totem or other components of the wearable device, and the like.

[0237] Processor The hardware processor 1770 can be configured to process data obtained by the environmental sensors 1710. The processor 1770 can also receive and transmit data from another device (such as a wearable display or another paired device) via the NFC interface 1732 or the network interface 1734. The processor 1770 can analyze these data to determine the location or movement pattern of the halo at a given time. In some embodiments, the processor 1770 can collaborate with and analyze another computing device (such as a wearable device, a remote server, or a personal computer). For example, the processor 1770 can detect the user's movement using the environmental sensors 1710. The processor 1770 can pass the user's movement data to the local processing and data module 260 or the remote processing module 270 for further analysis. The processor 1770 can receive the results of the analysis from the local processing and data module 260 or the remote processing module 270. For example, the processor 1770 can receive information regarding the location and movement trajectory of a competitor in a game. Based on this information, the processor 1770 can command the LEDs to emit light to display a halo corresponding to the competitor's position and movement trajectory.

[0238] The hardware processor 1770 can process various user inputs from a user's actuation of the totem 1200. For example, the hardware processor 1770 can process user inputs on the touchpad 1260, trigger 1212, home button 1256, bumper 1266, etc., as described with reference to FIGS. 12A-17B. As an example, the hardware processor can detect a user's hand gestures on the touchpad 1260 by processing signals from a touch sensor. As described herein, the hardware processor 1770 can process the user's inputs and instruct other components of the totem 1200 (e.g., LEDs or actuators 1760, etc.) to provide an indication of the user's actuation of the totem 1200.

[0239] Connection Interface 12A , the connection interface 1222 may be used to establish a connection with another device. The connection may be a wired connection. For example, the connection interface 1222 may include a USB connector. The connection interface 1222 may also include a USB port, such as a USB-B type port, a USB-A type port, a micro USB, or a USB Type-C port. The connection interface 1222 may also include a power cord for connecting to a power source for charging.

[0240] 17A and 17B are non-limiting examples, and totem 1200 may include fewer, more, or different components than those shown. For example, totem 1200 may not have an NFC interface 1732. As another example, totem 1200 may include an optical emitter configured to emit light (such as infrared light), an electromagnetic emitter configured to generate or sense a magnetic field (e.g., used for electromagnetic tracking), or the like. As yet another example, NFC interface 1732 and network interface 1734 may be part of the same communication system.

[0241] Example of configuring the location and movement of the halo light pattern The illuminated halos described herein can be customized by a user of the wearable system, a developer, or another entity. For example, a user of the wearable system can set preferences associated with the location and movement of the halo's light pattern. The preferences may be associated with contextual information as described with reference to FIGS. 16A and 16B . As an example, a user can set the color of the halo associated with system notifications (such as a low battery status) to blue, while setting the color associated with game objects to red. The user can set their preferences via the wearable device, alone or in combination with a totem. Based on the user's preferences, the totem can automatically present the illuminated halo when the user interacts with the wearable system. In some situations, the user can turn off the illuminated halo by activating the totem (such as trigger 1212 shown in FIG. 12A ) or via posture. For example, the outward-facing imaging system 464 can capture a user's hand gesture associated with turning off the halo. The wearable system can use computer vision algorithms to recognize hand gestures from images acquired by the outward-facing imaging system and, appropriately, command the totem to turn off the illuminated halo.

[0242] The location and movement of the halo light pattern can also be configured using an application programming interface (API). Figures 18A-18D illustrate an example programming interface for configuring the halo light location or movement pattern. The API can be interacted with by a user or developer to set the totem's halo light pattern. Figure 18A shows the API, including a totem visualization 1802. The totem visualization 1802 includes an image of a halo 1810 displayed on the visualization touchpad 1804. In this example, the length of the halo 1810 is set to 0.25, which may be approximately one-quarter of the top circumference of the visualization touchpad 1804. The endpoints of the halo 1810 are at the 12 o'clock and 3 o'clock positions.

[0243] The programming interface 1800 may include an application bar 1830. The application bar 1830 may include the type of application for which the halo light pattern is to be configured. For example, in FIG. 18A , a user of the programming interface 1800 is configuring a light pattern associated with a gaming application. The user may also configure light patterns associated with web applications, system applications, scenes (e.g., the location of objects in an environment, etc.), user characteristics (e.g., associating a halo pattern based on whether the user is right-handed or left-handed, etc.), etc.

[0244] The programming interface 1800 may also include an arc configuration tab 1820. The arc configuration tab 1820 may include various options for configuring the shape (e.g., arc, ring, etc.), size (e.g., angular arc length), color, visual effect, or orientation (e.g., N, S, E, or W) of the halo. Visual effects may include fade-in, fade-out, flashing, blinking, clockwise or counterclockwise rotation, etc. For example, the programming interface 1800 may associate a fade-in or fade-out effect with the halo 1810 (or a portion of the halo). The programming interface 1800 may also vary the arc length of the halo, e.g., from ¼ of a circumference to ¾ of a circumference. The rotation of the halo 1800 (e.g., from clockwise to counterclockwise) or the location of the halo 1800 (e.g., from the top right corner to the bottom right corner of the touch surface) may also be varied.

[0245] As an example of configuring the light location or movement pattern of the halo using the arc configuration tab 1820, FIG. 18B shows an example halo 1812 when the arc length is set to -0.25. In this example, the arc length of the halo 1812 is approximately 3 / 4 of the circumference of the touch surface. The endpoints of the halo 1812 are at the 9 o'clock and 6 o'clock positions.

[0246] The programming interface 1800 may also include a color selection tool 1830, shown in FIG. 18C . The color selection tool 1830 may include a search bar 1832, allowing a user of the programming interface to enter a halo color. The halo color may be entered in the format of a HyperText Markup Language (HTML) color code. The color selection tool 1830 may also include a color adjustment panel 1834, allowing a user to adjust the saturation, brightness, and hue of the color. The color selection tool 1830 may also include an RGB tool 1836, which allows a user to select a color based on the relative ratio between RGB and black / white. A user may use a pointing device (e.g., a mouse) to select a color, saturation, brightness, hue, etc. from the color selection tool panel.

[0247] 18C , the user has set the color of the halo 1814 to blue. The user can also use the color selection tool 1830 to change the color of the halo 1814 or the color adjustment panel 1834 to adjust the saturation, brightness, or hue of the selected color. In some embodiments, the color of the halo may be preset, for example, based on a theme or style sheet. The wearable system may preset the color based on a system default color or based on contextual information. A user of the programming interface 1800 can change or adjust the preset color using the color selection tool 1830.

[0248] FIG. 18D illustrates another portion of the programming interface 1800 , which includes a visualization tab 1842 , a source code tab 1844 , and a pattern adjustment tool 1850 .

[0249] The pattern adjustment tool 1850 can be used to configure the halo. For example, a user of the programming tool can adjust sliders associated with arc length, rotation, and color to provide various visual effects for the halo. The user can also adjust the position and rotation of the halo by setting x, y, and z coordinate values ​​for the halo (as shown in the transform tab 1852). The user can also adjust lighting and movement effects associated with the totem, such as fade-in / fade-out effects.

[0250] Visualization tab 1842 may be used to visualize one or more effects of the halo created by pattern adjustment tool 1850, arc configuration tab 1820, or color selection tool 1830. As an example, visualization tab 1842 may be associated with totem visualization 1802. When a user of programming tool 1800 selects visualization tab 1842, programming tool 1800 may show totem visualization 1802 so that the user may directly perceive the effect of updates to the halo.

[0251] The source code tab 1844 contains source code associated with the location and movement of the totem's light pattern. For example, when a user of the programming tool 1800 configures the halo using the arc configuration tab 1820, color selection tool 1830, or pattern adjustment tool 1850, the source code may be automatically updated based on the user's selection. The source code (or executable file version) can be communicated to the totem using a wired connection (e.g., via connection interface 1222) or a wireless connection (e.g., via network interface 1734). Additionally or alternatively, the source code (or executable file) can be stored within the totem or a head-mounted device (e.g., local processing and data module 260, etc.), which can be used to generate analog signals and control the light pattern of the LEDs 1330.

[0252] In some embodiments, the halo may be composed of multiple arcs (see example halo pattern 1614 in FIG. 16A). The programming tool 1800 can allow a user to configure each arc using the techniques described herein. The pattern adjustment tool 1850 shown in FIG. 18D can be used alone or in combination with the arc configuration tab 1820 and color selection tool 1830 to configure the halo.

[0253] Although the examples in Figures 18A-18D are described with reference to configuring the light location or moving pattern of a halo, similar techniques can also be used to configure tactile, auditory, visual, or other effects.

[0254] Example of totem calibration using halo User input devices (such as the totems described herein) may require calibration to minimize errors and improve the user interaction experience. For example, the totem's IMU may need to be calibrated to accurately detect and respond to a user's movements of the totem. Calibration is typically performed at the time of manufacture (e.g., when the totem or IMU is manufactured). Post-manufacturing calibration can be difficult because users may not have the control and precision of a calibration system that a totem manufacturer has. However, performing post-manufacturing calibration can be beneficial to users because user mobility can vary. For example, one user may move their arm to a greater extent than another user when they bob and jerk the totem. Additionally, it can be advantageous to provide a customized user experience so that the totem can have variable responsiveness tailored to different users. Some users may prefer a more sensitive totem (which may respond to slight movements), while other users may prefer a less sensitive totem (which may be configured not to perform user interface actions when slight movements are present).

[0255] The wearable system and totem described herein can advantageously support post-manufacturing calibration without requiring the user to use the manufacturer's calibration system. The totem can display a halo. As the user moves the totem 1200, the wearable system's outward-facing imaging system 464 can capture images of the totem and the illuminated halo. The shape of the halo as captured by the outward-facing imaging system 464 can vary based on the totem's position and orientation. The wearable system can use the object organizer 708 to identify the shape of the halo and determine the totem's position and orientation accordingly. To calibrate the totem's IMU, for example, the totem's position and orientation as determined based on images acquired by the outward-facing imaging system 464 can be compared to the position and orientation captured by the IMU. The wearable system can correct the totem's IMU based on the comparison.

[0256] FIG. 19A illustrates an example of totem calibration using an illuminated halo. While the example in FIG. 19A is described with reference to calibrating a totem 1900, this is for purposes of illustration and not limitation. The totem 1900, in this example, may have three degrees of freedom. The 3DOF totem 1900 may include a circular touch surface 1962 and a circular light guide 1964 surrounding the touch surface 1962. The light guide 1964 may display a circular halo. The 3DOF totem 1900 may be an example of the totem 1200 described herein. Thus, the touch surface 1962 may be an example of the touch surface 1262, and the light guide 1264 may be an example of the light guide 1264.

[0257] In this exemplary FIG. 19A, various poses of the totem are shown (e.g., poses 1910, 1920, 1930, and 1940). The poses of the totem can include the orientation and position of the totem. For example, pose 1920 indicates that the user is pointing the totem downward. pose 1930 indicates that the totem is facing right, and pose 1940 indicates that the totem is facing an upper right position. pose 1910 can be a base position, which is associated with the totem's natural resting position.

[0258] During the calibration process for the totem's IMU, the wearable system can capture images of the user's environment using its outward-facing imaging system 464. The images of the user's environment can include an image of the totem. The wearable system's object recognizer 708 can apply computer vision algorithms to identify the totem in the captured images. The object recognizer 708 can also apply computer vision algorithms to identify the shape of the halo in the images. The wearable system can determine the totem's position and orientation based on the shape of the halo, alone or in combination with an image of the totem body. For example, the wearable system can identify a shape associated with a base position and calculate changes to the shape associated with the base position. Based on the changes to the shape associated with the base position, the wearable system can determine changes to the totem's position and orientation based on the base position. For example, a halo having a circular shape can be changed to an elliptical shape if the normal to the circular shape points in a direction that is not directly toward the imaging camera. The wearable system can thus determine the current position and orientation of the totem based on changes to the base position.

[0259] As an example, the illuminated halo indicated by light guide 1964 may have a circular shape when totem 1900 is in the home position (as shown by pose 1910). Additionally, outward-facing imaging system 464 may not perceive the body of totem 1900 when the totem is in the home position. Referring to pose 1940, the wearable system may identify, based on the image obtained by outward-facing imaging system 464, that a portion of the totem's body is in a diagonal position. The wearable system may also identify that the illuminated halo in totem pose 1940 appears to have an elliptical shape, rather than a circular shape. Based on observation of the totem body and halo (among other possible information), the wearable system may determine that the totem is tilted to the right and facing up. As another example, the shape of the halo in poses 1920 and 1930 may both be elliptical. However, the long axis of the halo in pose 1930 is vertical (and the short axis in pose 1930 is horizontal), while the long axis of the halo in pose 1920 is horizontal (and the long axis in pose 1920 is vertical). Thus, the wearable system may determine that in pose 1930 the totem is positioned horizontally (e.g., facing left or right), and that in pose 1920 the totem is positioned vertically (e.g., facing up or down).

[0260] The origin position may be defined by the wearable system. In this example, the origin position is defined to be associated with pose 1910. In other examples, the origin position may be associated with other poses of the totem (poses 1920, 1930, 1940, etc.). In this example, the halo appears to be circular at the origin position, but the totem can illuminate other light patterns for totem calibration, which the wearable system can detect and analyze. For example, the totem can illuminate a square shape, a triangle (e.g., with three arcs surrounding the totem's touch surface), a pattern of overlapping line segments (e.g., having cross-shaped arcs or "x"-shaped arcs), etc. In some embodiments, the light pattern may also include arcs with multiple colors (which may be useful, for example, for calibrating totems with more degrees of freedom, such as a 6DOF totem). For example, if the light pattern includes a cross-shaped arc, the totem can illuminate the light pattern as illustrated in FIG. 25B. The outward-facing imaging system 464 and object recognizer 702 can use similar techniques to determine the totem's position and orientation regardless of the halo light pattern.

[0261] The wearable system can also receive data related to the totem's position and orientation from the IMU. For example, the wearable system can calculate the totem's orientation based on data obtained from a gyroscope within the totem. The position and orientation data obtained by the IMU can be compared to that calculated based on an image of the totem. The wearable system can calibrate the IMU based on this comparison. For example, the wearable system may determine that the totem 1900 is at attitude 1940 based on data obtained by the IMU. However, the position of the totem 1900 as observed by the wearable system may actually be at attitude 1930. Thus, the wearable system can correlate the position and orientation observed by the IMU with attitude 1930 (as observed by the outward-facing imaging system 464) and calibrate the IMU. In some embodiments, in addition to or as an alternative to calibrating the IMU, the wearable system may store a calibration transformation (e.g., a lookup table) to correct the IMU readings to represent calibrated (as opposed to raw) readings.

[0262] To facilitate the calibration process, the halo may be used to guide the user to move the totem into a certain position (e.g., position or orientation). The wearable system can then obtain the totem's position or orientation using an outward-facing imaging system and IMU. FIG. 19B illustrates an example of totem calibration, where light patterns are used as a guide for user movement. For example, the totem may exhibit patterns 1972, 1974, 1976, and 1978. These patterns may correspond to MUX2, 0, 1, and 4 shown in FIG. 14B. For example, LED3, LED4, and LED5 in MUX1 may illuminate and exhibit pattern 1976.

[0263] During the calibration process, the totem presents pattern 1972, indicating to the user to move the totem to the right, and pattern 1974, indicating to the user to move the totem to the left. When the totem presents pattern 1976, the user may move the totem upward, while when the totem presents pattern 1978, the user may move the totem downward. Once the totem (or IMU) is calibrated, the totem may present pattern 1980, which displays a solid blue color and forms a circular shape around the totem's touch surface. In addition to, or as an alternative to, pattern 1980, the totem may also employ other indications to indicate that the totem (or IMU) has been calibrated. For example, the totem may provide a vibration (e.g., via actuator 1760) or a sound (e.g., via speaker 240). As another example, the totem may provide other visual indications, such as a fade-in or fade-out effect, a portion of a circular halo, or other visual scheme. In some embodiments, the totem (e.g., processor 1770) may command the LED to stop illuminating once calibration is complete.

[0264] Although FIG. 19A is described with reference to calibrating an IMU in a 3DOF totem 1900, similar techniques can also be used to calibrate a 6DOF totem or other type of user input device.

[0265] Process 1982 in Figure 19C illustrates another example of a calibration process. Patterns 1984a, 1984b, 1984c, and 1984d indicate the current position / movement to which the totem is currently calibrating. For example, pattern 1984a (showing an arc at the top of the circular touchpad) may indicate that the totem is currently calibrating itself in association with upward movement, while pattern 1984b (showing an arc at the bottom of the circular touchpad) may indicate that the totem is currently calibrating itself in association with downward movement. As another example, pattern 1984c (showing an arc to the right of the circular touchpad) may indicate that the totem is currently calibrated for right movement, and pattern 1984d (showing an arc to the left of the circular touchpad) may indicate that the totem is currently calibrated for left movement.

[0266] In some embodiments, patterns 1984a, 1984b, 1984c, and 1984d may be triggered due to user interaction. For example, if a user moves the totem right, the totem may display halo 1984d and, if appropriate, calibrate the totem in association with this right movement. Additionally or alternatively, patterns 1984a, 1984b, 1984c, and 1984d may automatically illuminate to guide the user in moving the totem into a certain pose, as described with reference to FIG. 19B .

[0267] Once the totem is calibrated in all four directions (left, right, up, and down), the totem can present a confirmation pulse. The confirmation pulse may include alternating between displaying patterns 1986a and 1986b. For example, the totem can change its halo from pattern 1986a to pattern 1986b (and vice versa) every 0.1 seconds.

[0268] Although FIGS. 19A and 19B are described with reference to calibrating an IMU, similar techniques can also be used to calibrate other environmental sensors 1710.

[0269] In addition to, or as an alternative to, guiding a user through a totem calibration process, similar techniques can be used to calibrate a user's interactions with tangible objects. For example, when a user begins playing a game, the light location or movement pattern of a halo may be used to guide the user through a tutorial process. As an example, when the totem exhibits pattern 1984c, the user can move the totem to the left, and the display can therefore indicate a left movement of the avatar. The user can also associate the movement of the totem with a particular user interface action. For example, after an avatar is created, the user can perceive the avatar as being in a running action. The user can move the totem forward (away from the user) and set the totem's forward movement to be associated with the running action.

[0270] In some embodiments, the light location or movement pattern of the halo may differ based on the type of calibration. For example, the halo may be blue when the IMU is being calibrated, while it may be green when the user configures the totem's movement for a game.

[0271] 19D illustrates an example process 1990 for totem calibration using a light pattern associated with a halo. The example process 1990 can be performed by a wearable system (e.g., local processing and data module 260 or remote processing module 270, etc.) alone or in combination with a totem.

[0272] In block 1992a of the example process 1990 in Figure 19D, the wearable system can receive images of the user's environment. The images may be obtained by the outward-facing imaging system 464 shown in Figure 4.

[0273] In block 1992b, the wearable system can analyze the image and identify a totem (or a halo) within the image. The totem can include a body and a light guide. The light guide may be configured to display a halo. The wearable system can use the object recognizer 708 described with reference to FIG. 7 to identify the totem, the totem's body, the halo, or the light guide. The object recognizer 708 may use computer vision algorithms to perform such identification.

[0274] In block 1992c, the wearable system may determine a first position of the totem. The first position may be a base position. For example, the base position, such as the totem's natural resting position, may be predefined by the wearable system. The base position may also be identified or changed by the user. For example, the user may define the base position to be a position associated with posture 1910. The wearable system may determine a first light pattern of the halo associated with the base position. The pattern associated with the base position may be pre-stored by the wearable system. In some embodiments, once the wearable system identifies the base position, the wearable system may prompt the user with a notification requesting that the user hold the totem at the base position, and the wearable system may optionally capture an image of the totem at the base position.

[0275] At block 1992d, the wearable system may identify a second pattern of the halo based at least in part on analysis of the image. The wearable system may analyze the image and identify the pattern using the object recognizer 708.

[0276] In block 1992e, the wearable system may determine a first change to the totem's position and orientation relative to the first position. For example, the wearable system may analyze changes to the halo pattern based on a pattern associated with the home position when the totem is in the updated position and a pattern acquired by the outward-facing imaging system. In addition to image analysis, the wearable system may also identify a second pattern of the halo or calculate a first change to the totem's position and orientation based on other information, such as data from uncalibrated, user interaction, or one or more environmental sensors. For example, a user may provide an indication to indicate a change to the totem's position and orientation. The indication may indicate that the totem has been moved forward from its original position via posture or a user input device 466, such as pointing a finger forward.

[0277] In block 1994a, the wearable system may also receive movement data associated with the totem from environmental sensors that need to be calibrated. Optionally, the wearable system may calculate a second change to the totem's position and orientation based on the data received from the environmental sensors in block 1994b.

[0278] The wearable system may calculate the difference between the first change and the second change. In block 1992f, if the wearable system determines that the difference between the first change and the second change reaches a threshold condition, the wearable system may calibrate the environmental sensor in block 1992g. Otherwise, the wearable system may determine that the environmental sensor does not need to be calibrated and provide an indication, as shown in block 1992h. The indication may include, for example, a location and movement of a halo pattern or tactile feedback, such as a confirmation pulse shown in FIG. 19C. Additionally, the indication may be presented via the display 220, a speaker, or the like.

[0279] In some situations, the wearable system may collect data associated with multiple positions and orientations of the totem, and based on the collected data, the wearable system may determine whether or how environmental sensors need to be calibrated. For example, the wearable system may determine that the IMU does not need to be calibrated when the totem moves left and right, but may need to be calibrated for forward / backward movement. However, to calibrate for forward / backward movement, the wearable system may also need to adjust the IMU to detect left / right movement. Thus, the wearable system may calculate the amount of adjustment needed for forward / backward and left / right movement, even if the IMU is already capable of detecting left / right movement with reasonable accuracy.

[0280] 19E illustrates another example process 1996 of totem calibration using a light pattern associated with a halo. The example process 1996 can be performed by a totem or wearable system described herein.

[0281] In block 1998a, the totem can provide a first indication of the totem's posture. For example, the totem's light guide may display a halo with an arc located on the left side of the light guide. This arc can provide an indication to the user to move the totem to the left. In some embodiments, the indication can also be provided by a wearable display or via tactile or auditory feedback.

[0282] In block 1998b, the totem can obtain movement data associated with the pose. For example, as a user moves the totem to the left, the totem's IMU can detect the user's movement and communicate the movement data to the processor.

[0283] In block 1998c, the totem can be calibrated based on the movement data. For example, the totem can correlate left movement with movement detected by the IMU.

[0284] In block 1998d, it may be determined whether the totem has been calibrated. For example, if the totem's IMU needs to be calibrated, the totem may determine whether the totem has been calibrated for 6 DOF. Once the totem has been calibrated, in block 1998f, the totem may provide a second indication to signal that the totem has been calibrated. For example, the totem may provide a flashing green halo on the light guide to indicate that the totem has been calibrated.

[0285] If the calibration process is unsuccessful, the process 1996 may return to block 1998a to repeat the calibration.

[0286] Example of device pairing using halo Two devices may be required to establish a wireless communication link before being paired (e.g., via Bluetooth). When devices pair, they may share information such as the address, name, and profile of the device's user or the device itself, or share information stored in their respective memories. Once two devices are paired, they may share a common secret key, which may allow them to be reconnected for future information sharing.

[0287] Device pairing can be tedious and sometimes require the user to key in unique information about the device. For example, the user may manually enter the device's information after the device (or the user's other device) completes an initial search for nearby devices. Devices can use the NFC protocol to share information between two devices for the pairing process, but NFC components are often relatively large and costly.

[0288] To reduce or minimize user interaction and simplify the pairing process between the totem and the wearable device, the totem can present a light pattern that can be analyzed and used for pairing. The light pattern can include a message used for pairing between the totem and the wearable device. The light pattern can be presented by a halo illuminated on the totem's LED. The message can include an indication to initiate the pairing process. For example, the light pattern can encode a trigger message that causes the wearable device to search for nearby devices or broadcast its device information to nearby devices. As another example, the light pattern can encode the totem's device information. The device information may be encrypted. The device information can include a device identifier (ID), operating system information (e.g., operating system version), firmware information, the type of communication channel used for pairing (e.g., Bluetooth, wireless channel), a digital key shared between the totem and the wearable display, etc. In some implementations, the device ID may be provided or generated by a manufacturer, distributor, developer, or any suitable entity. Examples of device identifiers may include an Android identifier (ID), an iPhone® unique identifier (UDID), an iPhone® advertising identifier (IFA or IDFA), a cookie ID, a login ID, an Internet Protocol (IP) address, a Media Access Control (MAC) address, a hash of any of the above, a combination of any of the above, or equivalents. In some cases, a device identifier may be derived based on one or more hardware and / or software parameters of the device identified by the device identifier. For example, a device identifier may be derived from the device's IP address, operating system version, and locale setting.In some embodiments, a device identifier may be used to identify the source or origin of a transaction, request, or network event. For example, a device identifier may include a user identifier, an account identifier, and the like. Additionally or alternatively, a device ID may be associated with an entity (e.g., a user) using the device.

[0289] As an example, a totem may include a ring of LEDs on its perimeter (e.g., 12 LEDs arranged circumferentially around the touchpad). These 12 LEDs can be used to represent certain device information when illuminated in a specific temporal sequence, spatial sequence, or combination thereof. Each LED may be assigned a bit, and the ring of LEDs can represent a 12-bit word in this example. One or more of the LEDs can be displayed in sequence to indicate information needed for pairing. For example, LED lights can be illuminated in sequence to indicate the totem's device identifier (ID). Assuming the device ID is 2 and the binary representation of the number 2 is 10, the totem may illuminate LED0 (shown in FIG. 14B) to represent the binary number 2, while the other LEDs (e.g., LEDs 1-11) remain unilluminated.

[0290] In some implementations, the color of one or more LED lights can also represent information used for pairing. For example, red may represent one manufacturer or model of the totem, while blue may represent another manufacturer or model. The totem can also use LED lights to indicate the totem's manufacturer and model combination. For example, the totem can indicate manufacturer A and model M by illuminating LED1 blue while illuminating LED3 pink.

[0291] Furthermore, if each LED is assigned a bit, the color of the LED can expand the amount of information the LED can represent. For example, instead of one LED being assigned one bit, an LED with a certain color may be assigned one unit of information. Thus, if an LED light can illuminate nine colors, the LED can be used to indicate the numbers 0-9 based on the illuminated color. For example, if an LED is not illuminated, the wearable device can interpret it as 0, while purple can represent the number 1, green can represent the number 2, etc. As another example, LEDs can also represent letters in the alphabet (e.g., red is A, white is B, etc.).

[0292] The wearable device may be associated with a display and a camera (e.g., an outward-facing imaging system 464, etc.). The camera can be used to capture the light sequence, then demodulate it and extract usable information for automatic pairing. For example, images captured by the camera can be analyzed using various computer vision algorithms described herein to extract a halo light pattern. The light pattern may be converted into numbers, letters, words, etc., which can represent the totem's device information. The wearable device can then use this device information for pairing with the totem.

[0293] Two totems may also be paired. For example, two friends may wish to pair their totems and share game control while playing a video game using their individual wearable devices. In this example, the outward-facing imaging system of a first wearable device can capture the light sequence on a second friend's totem (or vice versa). The first wearable device can extract the device information of the second friend's totem and pass that information to the first friend's totem for pairing. In some implementations, the totem may also have a camera. Thus, a totem can capture an image of another totem's light pattern, achieving pairing of the two totems.

[0294] As another example, a wearable device (or HMD) may be paired with two totems. An outward-facing imaging system can obtain an image of the light pattern for one totem. Once the wearable device extracts one totem's device information from the subsequent light, it can pass the device information (of the wearable device or another totem) to the other totem for pairing.

[0295] In addition to or as an alternative to device pairing, similar techniques for communicating information by a totem can also be used to communicate other types of information. For example, the totem may communicate user profile information or the totem's status to the wearable device using a light pattern illuminated by the totem.

[0296] Advantageously, in some embodiments, the totem's light location or movement pattern can be used to indicate the progress of wireless pairing between a totem and a wearable device or between two totems. Figures 20A and 20B illustrate examples showing a wireless (e.g., Bluetooth or WiFi) pairing process using a halo. The light patterns illustrated in these two figures can be used to indicate the progress of the device pairing process via wireless communication (e.g., a totem or device wirelessly broadcasting its device information to other devices) or through the light patterns described with reference to Figure 14B.

[0297] 20A shows a totem 2020, which may have a touchpad 2022 and a display area 2026. Totem 2020 may be an example of totem 1200, and touchpad 2022 may be an embodiment of touchpad 1260. Display area 2026 may include a ring of LEDs and a light guide that diffuses the light emitted by the LEDs. In some embodiments, a user can activate display area 2026.

[0298] The display area 2026 may display a halo 2024. A portion of the halo may be dark blue. As shown in process 2012, the totem may play a clockwise spinning animation of the dark blue portion. Once the totem is paired, the totem may present a pulsating animation of the dark blue halo. An example animation is described with reference to FIG. 20B. The totem may initiate the spinning animation by displaying pattern 2062. While wireless pairing is in progress, the totem may sequentially present patterns 2064, 2066, and 2068, as illustrated by arrows 1, 2, and 3. The totem may play pattern 2062 again after pattern 2068 if pairing is still ongoing. If pairing is successfully completed, the totem may present pattern 2070, which may provide a visual indication to the user.

[0299] Although these examples are described with reference to pairing a totem, similar techniques can also be used to pair other user input devices 466 or devices with display or lighting components. For example, the techniques can be applied to pairing two or more of a smartwatch, smartphone, wearable device, totem, tablet, computer, television, camera, peripheral device, home appliance, or other device with image recognition and processing capabilities. As an example of pairing between a smartphone and a totem, the smartphone can use its camera to capture and analyze lighting patterns on the totem. As another example, a wearable device may be paired with a smartwatch, and the wearable device can capture and analyze patterns displayed by the smartwatch (e.g., on the smartwatch's display screen). The techniques described herein further apply to Bluetooth pairing, WiFi pairing (e.g., Wi-Fi Direct), and other methods of pairing. TM), or other types of wireless pairing. As an example of Bluetooth® pairing, when the totem is in discoverable mode, the totem may display a light pattern to indicate that it is in discoverable mode. The wearable device may capture this light pattern and complete pairing when the wearable device is also in discoverable mode. When the light pattern encodes a trigger message, the wearable device may automatically enter discoverable mode in response to detecting the light pattern. As an example of wireless pairing, the totem may illuminate a light pattern to indicate that a connection may be made to the totem. In response to detecting the light pattern, the wearable device may receive an invitation and connect to the totem. In addition to these examples of wireless pairing, the pairing techniques may also be applied to types of pairing other than wireless pairing. For example, the techniques may also be used to share files among a group of user input devices in a home via an Ethernet connection.

[0300] Example Process for Device Pairing Using Halo 20C illustrates an example process 2080 for device pairing using a halo. Pairing may occur between a wearable device and another device (e.g., a totem, a smartphone, a television, a smartwatch, etc.). The example process 2080 can be performed by a wearable device having an HMD and an outward-facing imaging system 464. The process 2080 can be used to assist in pairing a totem with another device, an HMD, etc.

[0301] In block 2082, the wearable device may receive a request from the totem to establish pairing. For example, the totem may broadcast a signal to nearby devices to establish communication. The wearable device may receive the broadcast signal and identify the totem that communicated the signal. As another example, the request may be a trigger message encoded in the totem's light pattern.

[0302] In block 2084, the wearable device may receive an image of the halo presented by the totem. The image may be captured by the outward-facing imaging system 464. As described with reference to Figures 20A and 20B, the totem may use the halo to communicate device information needed to establish pairing via the light location or movement pattern of the halo. For example, the halo may encode the device information in binary form.

[0303] In block 2086, the wearable device may analyze the image of the halo and extract device information. For example, the wearable device may use the object recognizer 708n to identify the halo in the captured image. If the halo encodes the device information in binary form, the wearable device may convert the binary representation of the device information to a decimal or alphabetic representation.

[0304] In block 2088, based on the extracted device information, the wearable device may pair the wearable device with the totem. For example, the wearable device may add the totem's device ID to a whitelist of devices with which the wearable device may share its information.

[0305] Optionally, in block 2089, the wearable device may receive an indication from the totem that pairing was successful. The indication may be an electromagnetic signal. Additionally or alternatively, the totem may display a halo with a location and movement to indicate that pairing is complete. For example, the totem may display a pulsating animation of a circular blue halo. The outward-facing imaging system 464 may capture this halo in an image. Once the wearable device recognizes the light location or movement pattern of the halo, the wearable device may determine that pairing is complete.

[0306] In some embodiments, the totem also provides a visual indication that pairing is in progress. For example, the totem may present a halo with a rotating arc during pairing. The rotating arc may be presented after the totem provides device information to the wearable device.

[0307] Although examples are described herein with reference to a wearable device, in some embodiments, similar techniques can also be applied to other computing devices. For example, the exemplary process 2080 can be used to pair a totem with a mobile phone configured to provide AR / VR / MR features.

[0308] 20D illustrates another exemplary process 2090 for device pairing using a halo. Pairing may occur between a totem and a computing device. The computing device may be a wearable device or other computing device (e.g., a television, a smartwatch, a smartphone, a tablet, another totem, etc.) having a wearable display and an outward-facing imaging system 464. The exemplary process 2090 may be performed by the totem 1200.

[0309] In block 2092, the totem may begin a pairing process with a computing device. For example, the totem may search for nearby devices that may be paired with the totem. The totem may also broadcast a signal and identify available devices for pairing based on responses to the signal received from other devices. A user may activate the totem or the computing device, causing the totem to begin the pairing process. For example, a user may press the home button 1256 for an extended period of time, causing the totem to enter the pairing process. In some implementations, the totem may present a light pattern that indicates that the totem is now in pairing mode. The light pattern may be captured by the computing device, causing the computing device to enter the pairing process with the totem.

[0310] At block 2094, the totem may illuminate a first light pattern that encodes information for the pairing process 2094. The encoded information may include device information for the totem. The encoded information may also include a key for the computing device to connect with the totem. For example, the computing device may extract the key based on the light pattern extracted from the key image and transmit the key back to the totem over a wireless channel to achieve pairing. In some embodiments, block 2092 may be optional. The computing device may automatically initiate the pairing process at block 2094 in response to detecting the first light pattern.

[0311] At block 2096, the totem may receive a response from the computing device. The response may be device information for the computing device. In some circumstances, the response may include a request for more information from the totem. The computing device may also provide other information in the response, such as network communication information (e.g., network port, encryption standard, or communication protocol) that the totem may use to communicate with the computing device. In some circumstances, the response may also include an indication that the computing device has added the totem to a list of approved devices.

[0312] Optionally, at block 2098, once the pairing process is completed, the totem may illuminate a second light pattern to indicate successful pairing. The second light pattern may be pattern 2070 shown in FIG. 20B. In some embodiments, the totem may cease illumination to indicate that pairing is complete. While example process 2090 is described above as being performed by a totem, it should be understood that example process 2090 may also be performed by another device (e.g., a smartphone, a smartwatch, a computer, etc.).

[0313] An embodiment using halo light patterns to provide cues about the status of an object The location and movement of the halo light pattern can be used as a cue to indicate the status of the object. For example, a certain light location or movement pattern can be used to indicate whether the user has turned the totem on or off, whether the totem is in sleep mode, or the totem's battery charge status. Non-limiting illustrative examples are described below.

[0314] 21A illustrates an example showing the status of a totem. As shown in process 2110, when the power button is turned from off to on, the totem may present a portion of its halo and gradually fill the remainder of that portion until the totem is fully on. The totem may also rotate its halo clockwise through one cycle during the power-on process.

[0315] When the power button is toggled from on to off, the totem may present the full halo and rotate the halo counterclockwise through one cycle (as shown in process 2120). The totem may also gradually obscure more and more of the halo until the totem is completely powered off (when the totem will no longer present the halo).

[0316] The halo color palette can be selected to be visually appealing to the user. Different color palettes can be used for different system functions, such as a startup color palette when the system is powered on (e.g., see process 2110 in FIG. 21A ) and a shutdown color palette when the system is powered off (e.g., see process 2120 in FIG. 21A ). For example, the startup color palette can represent colors typically seen at sunrise, and the shutdown color palette can represent colors typically seen at sunset. Other color palettes can also be used for other system functions (e.g., charging or battery status or sleep idle mode shown in FIGS. 21C and 21D ). Color palettes can include color, grayscale, illumination level, etc. The illumination pattern can include illumination information communicated to a light source controller (which can be a controller located on the touch board 1394 c) used to control the brightness of the totem light sources (e.g., LEDs 1383 a, 1394 a). The lighting information may include a color palette, the timing (or rate of change), duration, and brightness level at which each of the LEDs (including the individual colors that can be produced by the LEDs) is illuminated. Some LEDs may include additional or different control functionality, for example, RGBA, which includes an RGB plus alpha channel that indicates the opacity of each pixel and allows the totem to perform alpha compositing (enabling the appearance of partial or full transparency). The lighting information may include such additional or different control functionality depending on the light source selection used in the totem.

[0317] There can be a vast variety of color palettes that can be displayed by an RGB LED (e.g., typically, each color can be set to 256 values ​​from 0 to 255). The color palette can be selected for one or more of the halos described herein based on any of a variety of different factors, such as user characteristics, contextual information, device usage or status, system functionality, etc.

[0318] The halo lighting pattern can include gradients in color, shade, hue, etc. For example, due to the limited number of light sources in a totem (e.g., 12 RGB LEDs), variations in the brightness of the colored light sources can be used to provide the desired gradient (color or intensity). The gradient can provide the visual illusion of an intermediate end or fade-out of the lighting pattern. The gradient can be provided by dynamically adjusting the brightness of the light sources (time-varying lighting pattern). As an example, for a totem with 12 LEDs positioned at the hours of a clock, a lighting pattern starting at 4 o'clock and extending 45 degrees toward 6 o'clock, with the gradient ending at 6 o'clock, may have LED brightness values ​​set as follows: LED1-3, brightness 0.0; LED4, brightness 1.0; LED5, brightness 1.0; LED6, brightness 0.5; LED7-12, brightness 0.0.

[0319] Other shutdown processes may involve user interaction with a totem to confirm that the user desires to perform a shutdown operation (e.g., to shut down, suspend, or resume the wearable device 200). For example, the display 220 of the wearable system 200 may display a shutdown user interface to the user once shutdown is selected by the user. In one example, the shutdown user interface displays a circular ring (representing the circular light guide 1264 on the totem 1200) and a text indication that the user should move their thumb over the totem's touchpad 1260 to confirm the shutdown operation. For example, the user interface may display "Complete the circle to power off," and the circular ring may include an animation indicating that the user should move their thumb (for example) clockwise around the touchpad 1260. If the user presses the home button 1256, the shutdown operation may be aborted, and the shutdown user interface may fade out.

[0320] When the user desires to complete the shutdown operation, the user can follow the displayed direction and move their thumb in a circular clockwise path around the touchpad 1260. As the user does so, the light guide 1264 can illuminate and create an arc that follows the thumb's movement (and optionally, the display 220 can display an animated graphic indicating progress around the circular ring). When the user's finger completes a full circle (confirming the shutdown operation), a haptic "bump" can be implemented by the totem 1200 to provide the user with tactile feedback that the shutdown operation has begun (optionally, the system 200 can output an audio sound). The display 220 of the wearable system 200 can display a shutdown screen to the user and begin the shutdown operation. The light guide 1264 of the totem 1200 can display a shutdown illumination pattern (e.g., as shown in FIG. 21B ).

[0321] 21B illustrates an example of the light location or movement pattern of the halo during a power-on and power-off process. During the power-on process, the halo pattern can move from pattern 2122 to pattern 2124 and then to pattern 2126, as shown by arrows 1 and 2. During this process, the arc of the halo (which may be illuminated in a color, e.g., yellow) can become gradually longer in length and brighter in color.

[0322] On the other hand, during the power-off process, the halo pattern may move from pattern 2126 to 2124 and then to 2122, as shown by arrows 3 and 4. In this process, the shape of the halo may gradually reduce from a full ring to a small arc.

[0323] Additionally or alternatively, the halo can be configured to indicate the charge status of the totem's battery 1720, where the amount of battery may correspond to the size of the portion of the halo that is displayed. FIG. 21C illustrates an example of the location and movement of a light pattern, which indicates the battery charge status. Process 2132 illustrates an animated sequence of the halo as the battery 1720 is charging. For example, the halo may be divided into four quadrants. Each quadrant may be associated with one or more LED lights. The totem can gradually fill all four quadrants by illuminating the LEDs in each quadrant. In some implementations, rather than filling one quadrant at a time, the totem can gradually illuminate the LEDs and track the percentage of battery remaining. For example, when the battery is at 30%, the totem can illuminate 30% of the LEDs.

[0324] When charging begins, if the battery is below 15% (for example), the totem may present a halo with a first color (for example, orange) in one of the four quadrants. Once the battery reaches 25% (for example), the totem may change the color of the halo from the first color to a second color (for example, orange to green). Once the battery reaches 50% (for example), the totem may illuminate the portion of the halo associated with the next quadrant in a clockwise direction (for example, with the second color). The totem may continue fading in other quadrants as the totem charges. Once the battery is fully charged, the totem may present the halo in the second color and all four quadrants may be illuminated (as shown in process 2134).

[0325] While the battery is being consumed, the shape of the halo may initially be a full circle. The totem may gradually fade the halo in a counterclockwise direction as the amount of power decreases, as illustrated in process 2136. In some embodiments, when the battery is only 15% (for example), the totem may change the remaining portion of the halo to a third color (e.g., red) and play a pulsating animation (the totem may play a halo fade-out animation) until no power remains.

[0326] The halo can also be used to indicate whether the totem has entered sleep mode. Figure 21D illustrates an example halo pattern when the totem has entered sleep mode. As shown in process 2142, when the totem becomes idle because no user interaction has been detected within a threshold time period, the totem can present the halo in the form of a trail (e.g., a white trail). The totem can also repeat the trail in a clockwise cycle.

[0327] An exemplary process for indicating totem status using a halo 21E illustrates an example process for indicating the status of a totem based on light location or movement patterns. Process 2150 in FIG. 21E can be performed by one or more components of the wearable system described herein. For example, process 2150 can be performed by the totem or a wearable device.

[0328] In block 2152, the wearable system can determine the status of the totem. The totem can include a group of light sources (e.g., LEDs) configured to display a halo. The halo can be used to indicate the status of the totem. The status can include, for example, whether the totem is powered on / off, whether the totem is in sleep mode, or the totem's battery status, such as the progress of battery charge or consumption.

[0329] In block 2154, the wearable system can access a first light pattern for the halo. The first light pattern may define a location or movement pattern for the halo. The first light pattern can be associated with a current status of the totem. For example, when the battery is currently low, the light location or movement pattern for the halo can be a red arc around the 6 o'clock position.

[0330] In block 2156, the wearable system can instruct the group of LEDs to display a halo according to the first light pattern. Continuing with the previous example, the wearable system can generate and send instructions to illuminate a red light to LEDs located near the 6 o'clock position (such as LED9, LED10, and LED11 in MUX3 shown in FIG. 14B).

[0331] The wearable system can continuously monitor the status of the totem. Optionally, in block 2158, the wearable system can determine whether there are any updates to the status. For example, the totem's battery may have previously been below 15%, but the user plugged the totem into a power source to charge. Thus, the battery level may gradually increase.

[0332] In optional block 2160, in response to the update, the wearable system can instruct the group of LEDs to display a halo according to a light pattern. A second light pattern is associated with the update. For example, once the totem enters charging mode, the color of the halo may change from red to green.

[0333] Although examples herein are described with reference to indicating the status of a totem, similar techniques can also be used to indicate the status of other components of a totem, such as a wearable display or a battery pack.

[0334] An embodiment of using light patterns as cues for user interaction In addition to, or as an alternative to, providing information about device status, halos can also be used as cues to indicate or guide a current user interaction. Figures 22A and 22B illustrate example light location or movement patterns used as cues for user interaction. Figure 22A displays halo patterns 2210 and 2220. The halo patterns may be displayed by a light guide around the totem's touch surface. A user can activate the touch surface using, for example, their thumb 2230. Halo pattern 2210 may include an arc 2222 (which may be illuminated in a pinkish-purple color) on the left side of the light guide. The length and brightness of arc 2222 may reflect the distance between the user's finger and the portion of the light guide displaying arc 2222.

[0335] As an example showing a user's current interaction with the totem, when the user's thumb is at position 2212, halo pattern 2210 may be shown. The portion of arc 2222 closest to thumb 2230 may be brightest. The brightness of other portions of the arc may gradually decrease as they move farther away from thumb 2230, eventually transitioning to black (e.g., the corresponding LED is turned off). In this example, a user may activate the touch surface using a swipe gesture. For example, a user may swipe their thumb to the left. As the user moves their thumb 2230 left toward arc 2222 (illustrated by arrow 2214), the length of arc 2222 may increase from less than half a circle to more than half a circle. The brightness of arc 2222 may also increase due to the reduced distance between thumb 2230 and arc 2222.

[0336] 22B, the halo associated with pattern 2250 may include three arcs 2252, 2254, and 2256. Arc 2252 is illustrated in a first pattern corresponding to a first color (which may be, for example, red), arc 2254 is illustrated in a second pattern corresponding to a second color (which may be, for example, blue), and arc 2256 is illustrated in a third pattern corresponding to a third color (which may be, for example, yellow). Referring to FIG. 14B, arc 2252 may be illuminated by LED1, LED2, and LED3, arc 2254 may be illuminated by LED5, LED6, and LED7, and arc 2256 may be illuminated by LED9, LED10, and LED11.

[0337] Pattern 2250 may be associated with a three-way D-pad, with each arc associated with a user interface action. For example, a user may tap an area near arc 2252 to move the virtual object left and tap an area near arc 2254 to move the virtual object right. A user may also tap an area near arc 2256 to select or release a virtual object. When a user's thumb 2230 is near an arc, the length of the arc may decrease while the brightness of the arc may increase. For example, when a user's thumb 2230 is near arc 2254, the totem may turn off LED5 and LED7 (shown in FIG. 14B ) and reduce the size of arc 2254. This change to the light location or movement pattern of the halo can provide an indication that the user has begun (or is about to begin) moving the virtual object left. As the user's thumb moves away from arc 2254, the totem may re-illuminate LED5 and LED7 and increase the size of arc 2254.

[0338] The totem can simulate the functionality of various user input devices, such as a D-pad, touch device, 6DOF controller, etc. The light location or movement pattern of the halo can be used to indicate the type of user input device the totem is currently using or to indicate available user interface interactions. For example, while a user is browsing a web page using a 3D display, the totem can be used as a touch device that supports left / right swipe gestures. To provide an indication to the user that swipe gestures are available, the totem can display the pattern shown in FIG. 22A.

[0339] In some implementations, the wearable system can automatically select a preferred user input device configuration based on context information. For example, a user may want to use a 3-way D-pad to move virtual objects in and out of the FOV and select virtual objects. Thus, when the wearable device detects that the user is interacting with a virtual object in its environment, the wearable device can instruct the totem to activate the 3-way D-pad and present the halo pattern 2250 to inform the user that the 3-way D-pad is available.

[0340] FIG. 22C illustrates another example of using a light pattern associated with a halo to provide an indication of available user interface actions. The totem in FIG. 22C can be configured to act as a star-shaped D-pad. To inform the user that the totem is currently being used as a star-shaped D-pad, the totem may illuminate LED2, LED4, LED6, LED8, LED10, and LED0 in FIG. 14B. As an example, when a user selects a text box on a web page, the totem may present pattern 2260, indicating that the user can use the star-shaped D-pad for text entry. When a user activates an area near an LED, the LED may display a brighter color to indicate that the user is activating that area. For example, when a user activates an area near LED0 (as shown in pattern 2270), the totem may command LED0 to emit a brighter light.

[0341] In some embodiments, the totem may be used to inform the user that they are performing an incorrect user interaction or to provide an indication of a correct / recommended user interaction. FIG. 23 illustrates an example using a light pattern as an alert to indicate an incorrect or inappropriate user action. A user's thumb 2230 may initially be at position 2312 on the touch surface. The touch surface may be surrounded by a light guide, which may display a halo 2222. When the user's thumb is at position 2312, the light guide may display a halo pattern 2310. The user may need to move the object to the left by swiping left (e.g., moving their thumb 2230 closer to the halo 2222). However, the user actually swiped right (as indicated by arrow 2232), which is the incorrect direction to perform the desired action of moving the object to the left. As a result, the user's thumb is now at position 2314. To remind the user that they need to swipe left and move the object appropriately, the light guide may show a halo pattern 2320, where the halo 2222 appears longer and brighter in color, which can attract the user's attention and ensure that they move their thumb 2230 in the correct direction.

[0342] The pattern of the halo can vary depending on the type of user interaction. Figures 24A and 24B illustrate example patterns for swipe and touch gestures, respectively. The touchpad in Figures 24A and 24B can include a light guide, which displays the halo. The touchpad can also include a touch surface, which is surrounded by the light guide. The touch surface can receive user input, such as a swipe gesture or a touch gesture.

[0343] 24A and 24B may have three arcs: 2422 (illuminated a first color, e.g., red), 2424 (illuminated a second color, e.g., blue), and 2426 (illuminated a third color, e.g., yellow). Referring to FIG. 14B, arc 2422 may correspond to LED1, LED2, and LED3; arc 2424 may correspond to LED5, LED6, and LED7; and arc 2426 may correspond to LED9, LED10, and LED11.

[0344] A user's thumb 2230 may initially be at position 2412 on the touch surface. The light guide can thus display pattern 2402 for the halo. The user can swipe downward toward arc 2426, as indicated by downward arrow 2450. For example, the user can move their thumb 2230 from position 2412 to position 2414 (as indicated by arrow 1). In response to detecting the user's movement, the totem can dim the display of arcs 2422 and 2424, as indicated in pattern 2404. The user can further move their thumb downward, as indicated by arrow 2. In pattern 2406, when the user's thumb reaches position 2416, the totem can increase the brightness and length of arc 2426. For example, the brightness of LED9, LED10, and LED11 can be increased. The totem may also illuminate LED0 and LED8 (in a third color), increasing the length of arc 2426. The totem may further illuminate LED7 in a third color (rather than the second color, as shown in arc 2424) and LED1 in a third color (rather than the first color, as shown in arc 2422). As the user moves their thumb 2230 further downward, as shown by arrow 3, all LEDs (except LED4, opposite LED10) may illuminate in a third color, indicating the expansion of arc 2426 in the third color. On the other hand, as the user's thumb 2230 moves away from arc 2426, the totem may display a pattern according to arrows 4, 5, and 6. Under this sequence, the size of arc 2426 may gradually shrink, and the other arcs 2422 and 2424 may gradually fade in.

[0345] A user can also activate the touchpad using a touch gesture in FIG. 24A . FIG. 24B illustrates the halo pattern when a user activates the touchpad using a touch gesture. The totem may initially display three arcs 2422, 2424, and 2426, as shown in pattern 2462. When the user touches an area proximate to yellow arc 2426, the length of arc 2426 may increase. When the user activates the totem using a touch gesture at location 2418, as shown in pattern 2464, LEDs 0 and 8 may be illuminated (in a third color) to indicate increased length. However, arc 2426 will not expand to encompass other arcs (e.g., arcs 2422 and 2424) compared to pattern 2408 (which may correspond to a swipe gesture), even if the user's thumb is in the same location 2418.

[0346] An exemplary process for providing cues for user interaction on a totem 24C illustrates an example process for providing cues for user interaction on a totem. The example process 2480 may be performed by, for example, a component of a wearable system, such as a totem or a wearable device.

[0347] In block 2482, the wearable system can identify available user interface actions based on the context information. For example, the wearable system can determine available types of user interactions and map these types of user interactions to interactable areas of a totem. As an example, a browser can allow a user to scroll up / down, zoom in / out the size of a web page, select items on the browser, and move a cursor within the browser. The wearable system can map cursor movement and item selection to a first interactable area of ​​the totem, allowing the user to swipe the touch surface, move the cursor, click the touch surface, and select items. The wearable system can also map scrolling and resizing actions to a second interactable area. The second interactable area may surround the first interactable area. The second interactable area may simulate the functionality of a D-pad, allowing the user to tap the up / down keys to scroll a web page, tap the left / right keys, and resize a web page.

[0348] In block 2484, the wearable system may determine the location or movement pattern of the halo associated with the available interface actions. Continuing with the example above, the wearable system may display a halo with four arcs (as shown in FIG. 25B) to indicate that the totem has a second interactable area that the user may use as a D-pad.

[0349] In block 2486, the wearable system may generate and transmit instructions to the totem's LEDs (shown in FIG. 14B) to display the halo according to the light pattern. For example, if the halo has four arcs, the wearable system may instruct LED1 to illuminate blue, LED4 to illuminate yellow, LED7 to illuminate red, and LED10 to illuminate green.

[0350] In block 2488, the wearable system can receive user input on the totem. For example, the totem can detect a swipe or touch gesture in a first interactive area or a tap gesture in a second interactive area. In some embodiments, the user input can also include activating other components of the totem, such as the trigger 1212 or home button 1256 (shown in FIG. 12), alone or in combination with a user posture (e.g., head posture or other body posture).

[0351] Optionally, in block 2492, the wearable system may determine whether the user input is a correct input. The wearable system may make the determination based on contextual information of the object with which the user interacts. The wearable system may determine whether the user input belongs to a type of user input supported by the object. For example, a video recording application may allow user input via posture, while a web browser may support a combination of swipe and touch gestures. Thus, when a user is browsing a web page, the wearable system may not perform user interface actions based on changes in the user's foot posture. As another example, a web page may allow a user to view web content but not allow the user to enter comments on it. Thus, if a user attempts to type a phrase, the wearable system may determine that the user's input is a correct input because it is not supported by the web page.

[0352] If the user input is not a correct input, optionally, in block 2494, the wearable system can provide an indication that the user's input is incorrect. The indication can be provided in the form of visual, auditory, tactile, or other feedback. For example, the totem can highlight the correct input by providing a brighter color near the interactable area associated with the correct input. As another example, the totem can provide vibrations via a haptic actuator within the body of the totem. As yet another example, the wearable system can provide an alert message via the HMD or via a speaker.

[0353] If the user input is correct, then in block 2496 the wearable system can update the location and movement of the light pattern to reflect the user input. For example, if the user taps the left side of the totem's touch surface, the totem may illuminate an LED in the vicinity of the user's tap. In some embodiments, once the totem receives the user's input, the totem can update the displacement and movement of the light pattern associated with the halo to reflect the user input, in addition to or as an alternative to blocks 2492 and 2494.

[0354] Exemplary interaction with virtual objects using totems A totem can include multiple interactable areas, each of which can be mapped to one or more types of user interaction or types of user input devices. As described with reference to FIGS. 12A and 13A , in some embodiments, a totem has a first interactable area constituting a touch surface that can support touch events and gestures, such as swipe gestures, and a second interactable area constituting a light guide. The light guide can illuminate a halo at four points around the touchpad or using four arcs, each with a different color. This illumination pattern can indicate that the light guide is being used as a D-pad. For example, a user can tap or press on the illuminated points to move a virtual object. The same user gesture can cause different user interface actions to be performed depending on whether the gesture is applied to the light guide or the touch surface (e.g., within the main center area of ​​the touchpad). For example, when a user taps the light guide, the user can move a virtual object, but when the user taps the touch surface, the user can select a virtual object.

[0355] FIG. 25A illustrates an example interactive use of a light guide. The totem 2502 in FIG. 25A may be an example of the totem 1200 described herein. The totem 2502 may include a touchpad 2506 and may include a touch surface and a light guide that may substantially surround the touch surface. The light guide may diffuse light (e.g., emitted by an LED below the light guide) and exhibit a halo, which may include one or more arcs of the halo. In the example in FIG. 25A, the halo may include four arcs 2504a, 2504b, 2504c, and 2504d illuminated in four regions around the touch surface. The light guide may be interactive (e.g., a user may press a portion of the light guide) to receive user input. For example, the light guide may simulate the functionality of a D-pad. The light guide can include an up key 2508a, a right key 2508c, a down key 2508d, and a left key 2508b. The totem can provide visual indications on the locations of these keys. For example, arc 2504a can correspond to up key 2508a, arc 2504b can correspond to left key 2508b, arc 2504c can correspond to right key 2508c, and arc 2504d can correspond to down key 2508d.

[0356] A user can activate regions adjacent to individual keys to implement the user interface functions associated with those keys. For example, a user can activate region 2518a, activate up key 2508a, activate region 2518b, activate left key 2508b, activate region 2518c, activate right key 2508c, activate region 2518d, and activate down key 2508d.

[0357] As an example of activating a light guide, a user can move virtual objects in and out of their FOV by activating the light guide. Scene 2500 in FIG. 25A diagrammatically illustrates an example of interacting with FOV 2550 and virtual objects within FOR 2502. As described with reference to FIG. 4, the FOR can include a portion of the user's surrounding environment that can be perceived by the user via the wearable system. FOR 2502 can include a group of virtual objects (e.g., 2510, 2520, 2530, 2542, and 2544), which can be perceived by the user via display 220. Within FOR 2502, the portion of the world that the user perceives at a given time can be referred to as FOV 2550 (e.g., FOV 2550 can encompass the portion of FOR that the user is currently viewing). In scene 2500, FOV 2550 is diagrammatically illustrated by dashed line 2552. A user of the display can perceive multiple objects within FOV 2550, such as object 2542, object 2544, and a portion of object 2530. FOV 2550 may correspond to the solid angle perceivable by a user when looking through a display such as, for example, stacked waveguide assembly 480 ( FIG. 4 ) or planar waveguide 600 ( FIG. 6 ). As the user's posture (e.g., head posture or eye posture) changes, FOV 2550 changes correspondingly, and the objects within FOV 2550 may also change. For example, virtual object 2510 is initially outside the user's FOV 2550. When the user is looking at virtual object 2510, virtual object 2510 may move into the user's FOV 2550, and virtual object 2530 may move out of the user's FOV 2550.

[0358] A user can actuate the light guide to move virtual objects in and out of FOV 2550 without changing their body posture. For example, a user can actuate key 2508b to move the virtual object to the left within FOV 2550. Thus, virtual object 2530 can be moved entirely within FOV 2550. As another example, a user can actuate key 2508c to move the virtual object to the right, such that virtual object 2510 can be moved into FOV 2550 and virtual object 2530 can be moved out of the FOV.

[0359] Additionally or alternatively, the wearable system can assign a focus indicator to a virtual object within the FOV 2550 based on the user's gaze direction or the location of the virtual object. For example, the wearable system can assign a focus indicator to a virtual object that intersects the user's gaze direction or the virtual object closest to the center of the FOV. The focus indicator can comprise a halo (e.g., around the virtual object), a change in color, perceived size or depth (e.g., making the virtual object appear closer and / or larger when selected), or other visual effects that attract the user's attention. The focus indicator can also include audible or tactile effects, such as vibration, ringing, beeping, etc. For example, the wearable system may initially assign a focus indicator to virtual object 2544. When the user taps on region 2518c, the user can activate key 2508c. Thus, the wearable system may move the virtual object to the right and shift the focus indicator from object 2544 to object 2542, and the wearable system may move object 2510 into the user's FOV 2550 while moving object 2530 out of the user's FOV.

[0360] Figure 25B illustrates an example interactive use of a totem with two interactable areas. As described with reference to Figure 25A, the totem 2502 can include a touch surface 2506 and a light guide. The light guide can simulate the functionality of a D-pad, as described with reference to Figure 25A, and can display a halo having four arcs 2514a, 2514b, 2514c, and 2514d, each having a different color.

[0361] A user can actuate the light guide or touch surface 2506 to activate a totem. For example, in a browser window, a user can actuate the touch surface 2506 to move a cursor. A left swipe on the touch surface 2506 can be mapped to a back function, while a forward swipe on the touch surface 2506 can be mapped to a forward function for the browser page. A tap (illustrated by position 2518) at the 6 o'clock position (illuminated by arc 2514d) can cause the browser to scroll down, and a tap at the 12 o'clock position (illuminated by arc 2514a) can cause the browser to scroll up. Holding at the 6 o'clock or 12 o'clock positions can rapidly scroll pages.

[0362] The totem can provide visual feedback when a user activates the totem. For example, if a user taps at the 6 o'clock position near arc 2514d, the totem can increase the brightness of arc 2514d. Haptic feedback can be used to distinguish whether the user activated the touch surface 2506 or the light guide. For example, when a user taps the light guide, an actuator under the touchpad may be activated and provide a vibration.

[0363] 25C illustrates an example process for interacting with a totem. Process 2570 may be performed by a wearable system (such as a totem or wearable device) described herein.

[0364] In block 2572, the wearable system can receive user input on the totem. User input may include hand gestures such as swipes, taps, touches, presses, etc. The totem can include multiple interactable areas. For example, the totem can include a touchpad, which can include a touch surface and an optical waveguide, both of which may be interactable.

[0365] In block 2574, the wearable system can determine an interactable area associated with the user input from the plurality of interactable areas. For example, the totem's touch sensor 1350 can determine whether the user activated the light guide or the touch surface. The touch sensor 1350 can also detect a user gesture used to activate the totem. For example, the touch sensor can detect whether the user activated the totem using a swipe gesture or a tap gesture.

[0366] In block 2576, the wearable system can access a light pattern associated with the user input detected within the interactable area. As described with reference to FIGS. 24A and 24B, the light pattern can differ based on the type of user input (e.g., touch gesture vs. tap gesture). The light pattern can also differ depending on the interactable area that is actuated. For example, if the user actuates the touch surface, the light pattern may be that illustrated in FIG. 24A, while if the user actuates the light guide, the light pattern may include that shown in FIG. 24B.

[0367] In block 2578, the wearable system can instruct the totem's light sources (e.g., LEDs) to present a halo with a light pattern. For example, the wearable system can provide instructions regarding which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, a movement pattern associated with the light emitted by the LEDs, etc.

[0368] Exemplary interactions with physical objects using totems 26A and 26B illustrate examples of using totems to interact with physical objects. FIG. 26A illustrates an example of a physical environment 2600, which may be a living room in a user's home. The environment 2600 has physical objects such as a television (TV) 5110, a physical remote control 5120 (sometimes simply referred to as a remote), a TV stand 5130, and a window 5140. A user can perceive and interact with the physical objects. For example, a user may watch a TV 5110 and use the remote 5120 to control the TV. The user can control the remote 5120 to turn the TV 5110 on / off or change the channel or volume of the TV 5110.

[0369] A user can also interact with the TV 5110 using a totem 2602. The totem 2602 can be an embodiment of the totem 1200, which can include multiple interactable areas. The totem 2602 can be paired with the TV 5110 or a remote 5120 and simulate the functionality of the remote 5120. The functionality of the remote 5120 can be mapped to one or more interactable areas associated with the totem's touchpad, trigger, or home button. The touchpad can include a touch surface 2634 and a light guide 2630, and the touch surface 2634 and the light guide 2630 can be mapped to different types of user interactions. For example, the touch surface 2634 can be used to change channels via a swipe gesture. The light guide 2630 can be used as a D-pad to adjust volume or fast forward / rewind via a tap gesture.

[0370] In FIG. 26B , the light guide can exhibit a light pattern 2622 for a halo having four arcs: arc 2632a (which may be a first color, e.g., blue), arc 2632b (which may be a second color, e.g., yellow), arc 2632c (which may be a third color, e.g., red), and arc 2632d (which may be a fourth color, e.g., green). The arcs are positioned to form an x-shape. When user 2602 touches an area on the light guide near arc 2632c, as illustrated by arrow 1, the user can fast-forward a program playing on TV 5110. A totem can exhibit pattern 2624 to indicate that the user tapped near arc 2632c. Arc 2632c appears brighter and longer, while the other arcs 2632a, 2632b, and 2632d are obscured in pattern 2624.

[0371] The user can also change to the next channel by swiping right (as illustrated by arrow 2). The totem can illuminate light pattern 2626 to indicate that the user activated the totem with a swipe gesture rather than a tap gesture. Even though the user's 2602 finger is in approximately the same position in patterns 2626 and 2624, the illuminated light pattern in these two situations is different because of the different gestures used to activate the totem. For example, the brightness and length of arc 2632c has increased, but the other arcs 2632a, 2632b, and 2632d are not obscured in pattern 2626. Rather, the totem simply reduces the brightness of the other arcs.

[0372] 26A and 26B are described without a wearable device, in some embodiments, a user can use a totem to control physical objects while wearing the wearable device. For example, a user can perceive physical objects through the wearable device and perceive virtual objects projected by the wearable device. The user can use the totem to interact with the physical objects and the virtual objects.

[0373] Exemplary 6DOF User Experience with Totem FIG. 27 illustrates an example of using a multi-DOF (e.g., 3DOF or 6DOF) totem to move a virtual object. A user can change the totem's position by moving it forward / backward (yaw), up / down (bob), or left / right (yaw). A user can also change the totem's orientation by tilting it sideways (roll), tilting it forward / backward (pitch), or turning it left and right (yaw). By changing the totem's position and orientation, a user can perform various interface actions on the virtual object (e.g., by translating or rotating it, etc.).

[0374] A user may use the totem 2602 to select a virtual object, hold and move the virtual object, and physically move the totem 2602 to move the virtual object. For example, the totem 2602 may initially be at a first location 5100a. The user 5300 may select a target virtual object 5400 located at a first location 5100b by activating the totem 504 (e.g., by activating a touch-sensitive pad on the totem). The target virtual object 5400 can be any type of virtual object that can be displayed and moved by the wearable system. For example, the virtual object may be an avatar, a user interface element (e.g., a virtual display), or any type of graphical element displayed by the wearable system. The user 5300 can move the totem 2602 along a trajectory 5500a to move the target virtual object from the first location 5100b to the second location 5200b along a trajectory 5500b.

[0375] As another example, instead of moving virtual object 5400 from 5100b to 5200b, user 5300 may want to move virtual object 5400 closer to him / her. Thus, the user can move the totem closer to him / her, bringing virtual object 5400 closer.

[0376] As yet another example, totem 2602 may be at position 5200a. The user may rotate the totem 180 degrees clockwise, and virtual object 5400 may therefore be rotated 180 degrees at position 5200b.

[0377] The totem can emit a light pattern to indicate the type of user interaction. For example, a ring of LEDs may illuminate blue when the user moves the totem, while illuminating green when the user rotates the totem. As another example, the totem's light movement pattern may correspond to the totem's rotation. For example, the light pattern may include an arc. As the user rotates the totem, the arc may also rotate in the direction of the user's rotation.

[0378] An example of decorative and informative visual feedback on a totem Some wearable devices may have a limited FOV. Advantageously, in some embodiments, the totem can provide information associated with objects outside the FOV using light patterns illuminated by the totem.

[0379] 28A and 28B illustrate examples of providing object information through the location and movement of light patterns. FIG. 28A illustrates a person's FOV 2820 and FOR 2810. FOR 2810 can contain a group of objects (e.g., objects 2812, 2822, 6050b, 6050c) that can be perceived by a user wearing a wearable system. FOV 2820 can contain multiple objects (e.g., objects 6050, 2822). The FOV can depend on the size or optical characteristics of the wearable system, for example, the effective aperture size of a transparent window or lens in a head-mounted display through which light passes from the real world in front of the user to the user's eyes. In some embodiments, as the posture of the user 6100 (e.g., head posture, body posture, and / or eye posture) changes, FOV 2820 can change correspondingly, and the objects within FOV 2820 can also change. As described herein, the wearable system may include sensors, such as cameras, that monitor or image objects within the FOR 2810 and objects within the FOV 2820. In such embodiments, the wearable system may alert the user (e.g., via a light pattern on a totem) to unnoticed objects or events occurring within the user's FOV 2820 or occurring outside the user's FOV but within the FOR 2810. In some embodiments, the wearable system can also distinguish between objects to which the user 6100 is or is not paying attention.

[0380] Objects within the FOV or FOR may be virtual or physical objects. Virtual objects may include, for example, operating system objects, such as a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. Virtual objects may also include objects within an application, such as avatars, virtual objects within a game, graphics, or images. Some virtual objects can be both operating system objects and objects within an application. A wearable system can add virtual elements, viewed through the transparent optics of the head-mounted display, to existing physical objects, thereby enabling user interaction with the physical objects. For example, a wearable system may add a virtual menu associated with a medical monitor in a room, and the virtual menu may give the user the option of using the wearable system to turn on or adjust medical imaging equipment or medication controls. Thus, a wearable system may present additional virtual image content to the wearer in addition to objects in the user's environment.

[0381] The totem can provide information about objects outside the FOV through the location and movement of the light pattern. The location and movement of the light pattern can be calculated and determined by various components of the wearable system, such as the totem or the wearable device. The wearable system can determine the light pattern using contextual information associated with the user or object. For example, the wearable system can employ factors such as the object's location (including the object's proximity to the user or the user's FOV), the object's urgency, the object's type (e.g., whether the object is interactable, the type of interaction supported by the object, whether the object is physical or virtual, etc.), the object's nature (e.g., a competitor's avatar vs. a friend's avatar, etc.), the amount of information (e.g., the number of notifications, etc.), user preferences, etc. As shown in scene 2834 in FIG. 28B , because object 2840 is farther from the user 6100 than scene 2832, the halo 2844 on the totem 2502 may appear smaller (e.g., have a shorter angular range) than halo 2842 (which has a larger angular range). As another example, a halo may have a larger and brighter appearance because the object associated with the halo is more urgent or closer to the user's FOV.

[0382] The wearable system may assign a color to the halo based on the characteristics of the associated object. For example, the wearable system may assign red to halo 6051b because object 6050b is associated with the color red. Similarly, the wearable system may assign blue to object 6051a because it is an operating system object and the wearable system assigns blue to all operating system objects.

[0383] The wearable system may assign colors to the halos based on the characteristics of the associated objects. For example, the wearable system may assign green to halos 2842 and 2844 because object 2840 is associated with the color green. Similarly, the wearable system may assign red to object 6050b because it is an operating system object and the wearable system assigns red to all operating system objects. As a result, halos associated with operating system objects (including object 6050b) may be red.

[0384] The appearance of the halo may change over time based on changes in the object associated with the halo. For example, the halo may grow darker as the object associated with the halo receives more messages. As another example, the halo 2844 may grow larger (or brighter) as the object 2840 moves closer to the user, or grow smaller (or dimmer) as the object 2840 moves away from the user. The position of the halo may also change as the object moves. For example, in FIG. 28A , the wearable system may initially show the halo associated with object 6050b on the left side of the totem's light guide because object 6050b is to the left of user 6100. However, if object 6050b moves to the right side of user 6100, the wearable system can update the position of the halo to the right side of the totem's light guide.

[0385] The appearance of the visual halo may also change based on changes in the user's posture. For example, referring to Figure 28A, as user 6100 turns left, object 6050b may move closer to the user's FOV, while object 6050c may move further away from the user's FOV. As a result, the halo associated with object 6050b may become brighter (or larger).

[0386] In some situations, the FOR 2810 and FOV 2820 may contain invisible objects that are not directly visible to the user. For example, a treasure hunt game may contain hidden treasure (e.g., objects 2812, 2822 shown in dashed lines) within the user's FOR 2810. The treasure may be embedded within a game object and therefore may not be directly visible to the user. However, the user can use a virtual tool in the game (e.g., a virtual shovel, etc.) to uncover the hidden object. As another example, the user's physical car keys may be under a pile of papers, and the user may not be able to perceive the physical car keys within the user's FOV. A halo can be used to provide an indication of the location of the hidden car keys. For example, the size and location of the halo arc can indicate the relative distance and orientation between the car keys and the totem. The location of the car key may be determined based on mapping of the user's environment (e.g., based on images previously obtained by an outward-facing imaging system), or, if the key is equipped with wireless or wireless communication capabilities, based on wireless communication between the Totem (or other component of the wearable system) and the car key.

[0387] As yet another example, an object (e.g., object 6050a) may be outside the user's visual FOR but may nevertheless potentially be perceived by a sensor (e.g., an electromagnetic sensor, a radio frequency sensor, a sensor associated with detecting radio signals, or another environmental sensor) on the wearable device or totem. For example, object 6050a may be behind a wall in the user's environment such that object 6050a is not visually perceptible by the user. However, the wearable device or totem may include a sensor that can communicate with object 6050a.

[0388] A totem can provide information about invisible objects through light patterns. The location and movement of the light pattern can be based on contextual information associated with the invisible object or the user. For example, the location and movement of the light pattern on the light guide can act as a compass, directing the user to look in a certain direction or move toward it. As another example, the light pattern may provide a map of invisible objects in the user's environment. For example, a totem can illuminate a 30-degree red arc at the 1 o'clock position to indicate that the car key is facing the user's right, while illuminating a 90-degree green arc at the 6 o'clock position to indicate that the user's cell phone is behind the user. The red and green arcs can also be used to indicate the proximity of the car key and cell phone. For example, the cell phone is closer to the user than the car key, and as a result, the green arc is larger than the red arc. As the user moves, the location (or movement) of the light pattern can also change. For example, as the user moves forward, the size of the green arc can decrease, while the size of the red arc can increase.

[0389] In some implementations, the wearable system may present a mapping of the user's physical or virtual environment, which can indicate the location of invisible objects. For example, the HMD may present a virtual map, which shows in-game objects that the user can find. As another example, the HMD may provide a focus indicator or other type of visual indication of a hidden physical car key (or virtual object) as AR / MR content. The appearance of the focus indicator (e.g., size and direction relative to the user) may change as the user moves around in the environment. In some situations, the HMD may present a route to an invisible object. The map presented by the HMD may be combined with a halo on a totem to guide the user to the invisible object.

[0390] In some embodiments, the totem can present a halo that includes information associated with multiple objects. For example, a user 6100 can have a red object 6050b to their left and an invisible object 2812 behind them. The totem can present a halo that has a red arc on the left side of the light guide and a silver arc at the bottom of the light guide, where the red arc corresponds to the red object 6050b and the silver arc corresponds to the invisible object 2812.

[0391] In addition to, or as an alternative to, providing information about objects using light patterns, the display can also be configured to provide information about objects that are outside the FOV or are invisible. For example, the display can display a visual halo for a corresponding object that is outside the user's current FOV. A portion of the visual halo can be located on the edge of the user's FOV of the display. The location of the halo can be based on contextual information about the object.

[0392] In addition to visual haloes, the wearable system can also alert the user to objects using tactile or auditory effects. For example, in a mixed reality game, the wearable system may notify the user of approaching competitors through vibrations on a totem. The wearable system may provide a strong vibration when a competitor is in close proximity to the user. In another example, the wearable system may provide location information of virtual objects using an audible sound. The wearable system may also use a high-pitched sound to alert the user to nearby virtual competitors. Tactile, auditory, or visual feedback can be used in combination with, or as an alternative to, alerting the user to surrounding objects.

[0393] 28C illustrates an example process for providing information associated with an object using light patterns. Process 2870 can be implemented by a wearable system (such as a totem or wearable device) described herein.

[0394] In block 2872, the wearable system can identify an object in the user's environment. The object may be a physical or virtual object. The object may be in the user's FOR but outside the user's FOV. The object may also be hidden from the user's view. For example, the object may be behind another object (e.g., a wall) or may only become visible with some user interface interaction (e.g., when the user completes a level in a game).

[0395] In block 2874, the wearable system may access contextual information associated with the object. The contextual information may be used to determine the movement and location of a light pattern associated with the object, as shown in block 2874. For example, the brightness of a halo with the light pattern may be determined based on the proximity or urgency of the object. The color of the halo may also match the color of the object.

[0396] In block 2878, the wearable system can instruct the totem's LEDs to illuminate according to a light pattern. For example, the instructions may include which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, a movement pattern associated with the light emitted by the LEDs, etc.

[0397] 29A illustrates exemplary light locations or movement patterns that indicate receipt of a notification. Patterns 2912 and 2914 include rainbow patterns. A totem can display a rainbow trail in a clockwise cycle (as shown by process 2920) to provide the notification. The notification may be associated with a virtual object, such as an email application, a game, a video application, or the like. The notification may also be associated with the operating system of the wearable device. One exemplary notification may be an error message from the operating system of the wearable device. Additionally, or alternatively, the notification may be associated with a physical object. For example, rainbow patterns 2912 and 2914 may indicate that a coffee machine in a user's kitchen has finished brewing.

[0398] The totem may repeat the rainbow trajectory until a threshold condition is reached, which may be based on duration, user interaction (e.g., activation of trigger 1212, etc.), or other contextual information.

[0399] 29B illustrates an example process for providing notification using a light pattern on a totem. The example process 2970 can be implemented by a wearable system (such as a totem or wearable device) described herein.

[0400] In block 2972, the wearable system may determine the status of the virtual object. For example, the status may include whether the virtual object has received new information (e.g., a new message), whether the virtual object is idle, whether the virtual object has encountered a problem, etc.

[0401] Optionally, at block 2974, the totem can generate an alert based, at least in part, on the status. For example, the virtual object can be a messenger application. When the virtual object receives a new message from a friend of the user, the wearable system can generate an alert indicating that a new message has arrived.

[0402] In block 2976, the wearable system may access a light pattern associated with the status of the virtual object (or optionally, the alert). The location or movement pattern may be determined based on contextual information associated with the virtual object (or alert), such as, for example, the importance of the alert, the urgency of the alert, the location of the virtual object, the type of virtual object, etc.

[0403] In block 2978, the wearable system can instruct the totem's LEDs to illuminate according to a light pattern. For example, the wearable system can provide instructions regarding which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, the movement pattern associated with the light emitted by the LEDs, etc.

[0404] In addition to, or as an alternative to, providing information about objects outside the FOV, light patterns can also be used to provide information about objects that are not necessarily outside the FOV. For example, when a user completes a level in a game, a totem may illuminate with a halo, which has the same color as the game. As another example, referring to FIG. 29A, a virtual application associated with notifications may be within the FOV. A user may receive a notification that a new email has arrived while interacting with an email application, for example, by composing a message.

[0405] The totem can provide visual feedback of the user's current interaction to the user or others in the user's environment. For example, when a user is recording video with their HMD, the totem can illuminate a blinking red light pattern to inform the user and indicate to others in the vicinity that the HMD is in recording mode.

[0406] FIG. 30 illustrates an example light pattern that may be used to inform persons in a user's environment of the user's current interaction. For example, when a user is in a telepresence session, the totem may illuminate a light pattern 3000 (e.g., a green halo with a clockwise movement) via a light guide. The light pattern 3000 may inform people in the user's environment that the user is in telepresence. This may help discourage other people from getting too close to the user, thereby preventing them from disrupting the user's telepresence session. In addition to or as an alternative to light patterns, other types of feedback, such as tactile, auditory, or video feedback, may also be used to indicate the user's current interaction with the wearable system.

[0407] Additional Aspects In a first aspect, a system includes: a light emitting assembly of a user input device configured to illuminate a plurality of light patterns to provide information of objects in an environment; and a hardware processor communicatively coupled to the light emitting assembly and programmed to identify objects in a user's environment, access contextual information associated with the objects, determine characteristics of the light patterns illuminated by the light emitting assembly based, at least in part, on the contextual information, and instruct the light emitting assembly to illuminate according to the light patterns.

[0408] In a second aspect, the system of aspect 1, wherein the object includes at least one of a physical object or a virtual object.

[0409] In a third aspect, the system of aspect 2, wherein the status of the virtual object includes at least one of a current interaction with the virtual object by a user, whether the virtual object has received new information, whether the virtual object is idle, or whether the virtual object is in an error state.

[0410] In a fourth aspect, the system of any one of aspects 1-3, wherein the characteristics of the light pattern include at least one of brightness, position, shape, size, or color.

[0411] In a fifth aspect, the system of any one of aspects 1-4, wherein the contextual information associated with the object includes at least one of the location of the object, the urgency of the object, the type of the object, the nature of the object, the amount of information associated with the object, or a user preference.

[0412] In a sixth aspect, the system of any one of aspects 1-5 further includes a wearable display device, wherein the object is invisible from the user's view or outside the view of the wearable display device, and the hardware processor is programmed to determine at least one of the size, shape, or color of the light pattern and provide cues to the user for locating the object.

[0413] In a seventh aspect, the system described in any one of aspects 1-6, wherein the object is a component of a wearable system for presenting virtual content to a user, and the light pattern indicates a status of the component of the wearable system.

[0414] In an eighth aspect, the system of aspect 7, wherein the component includes at least one of a user input device, a wearable display device, or a battery pack.

[0415] In a ninth aspect, the system of any one of aspects 1-8, wherein the status includes at least one of battery status, power charging status, wireless pairing status between the wearable display device and the user input device, status of a calibration process of the user input device, or status of the wearable display device.

[0416] In a tenth aspect, the system of any one of aspects 1-9, wherein the light pattern encodes an alert or information associated with the object.

[0417] In an eleventh aspect, the system of any one of aspects 1-10, wherein the status includes at least one of a user's current interaction with the object, whether the object has received new information, whether the object is in an idle state, or whether the object is in an error state.

[0418] In a twelfth aspect, the system of any one of aspects 1-11, wherein the characteristics of the light pattern are configurable by a user via an application programming interface.

[0419] In a thirteenth aspect, a light-emitting user input device includes: a touch component configured to receive user input; a light-emitting assembly configured to output a plurality of light patterns, the light-emitting assembly at least partially surrounding the touch component; and a hardware processor communicatively coupled to the touch component and the light-emitting assembly and programmed to: identify user interface operations supported by the touch component based on context information; determine a first light pattern associated with the user interface operation; generate and transmit instructions to the light-emitting assembly to display a halo having the first light pattern; receive user input on the touch component; and update the halo with a second light pattern to reflect the user input.

[0420] In a fourteenth aspect, the luminous user input device of aspect...

Claims

1. 1. A light-emitting user input device for providing user input to a display system including a wearable display device, the light-emitting user input device comprising: a plurality of interactable areas including a central touch surface configured to receive user input; a light emitting assembly including a light guide exterior to and at least partially surrounding the central touch surface; a hardware processor communicatively coupled to the plurality of interactable areas and the light emitting assembly; Equipped with The hardware processor includes: Detecting actuation of the light-emitting user input device by a user of the wearable display device; determining an interactable area among the plurality of interactable areas corresponding to the actuation; and translating the action into a user input for performing a user interface action to interact with a virtual environment presented through the wearable display device, the virtual environment including virtual content generated by an application running on the display system, the user interface action being determined based at least on a type of the action and the interactable area corresponding to the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input, the light pattern being based at least in part on the application running on the display system; and It is programmed to The light emitting user input device, wherein the light guide is touch sensitive to receive user input.

2. the plurality of interactable areas include a first interactable area and a second interactable area; In response to determining that the user has activated the first interactable area, the hardware processor is programmed to cause the light emitting assembly to illuminate a first light pattern associated with the user input within the first interactable area; 10. The light-emitting user input device of claim 1, wherein in response to determining that the user has activated the second interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input within the second interactable area.

3. The luminous user input device of claim 1 , wherein the actuation comprises at least one of a swipe, a tap, a press, or a touch gesture.

4. The luminous user input device of claim 1 , wherein the light pattern comprises an arc-shaped region having one or more of a color, an arc length, or a visual effect.

5. The luminous user input device of claim 1 , wherein the hardware processor is further programmed to communicate with the display system and cause the display system to perform the user interface operation based on the user input.

6. The light emitting user input device of claim 1 , wherein the light guide is substantially circular.

7. The luminous user input device of claim 1 , wherein the central touch surface is substantially circular.

8. The light emitting user input device of claim 1 , wherein the light guide completely surrounds the central touch surface.

9. The light-emitting user input device of claim 1 , wherein the wearable display device is separate from the light-emitting user input device.

10. 1. A method comprising: Under the control of a hardware processor, Detecting actuation of a light-emitting user input device by a user of a wearable display device, the light-emitting user input device configured to provide user input to a display system including the wearable display device, the light-emitting user input device comprising: a plurality of interactable areas including a central touch surface configured to receive input; and a light-emitting assembly comprising an optical waveguide outside of and at least partially surrounding the central touch surface; determining an interactable area among the plurality of interactable areas corresponding to the actuation; and translating the action into a user input for performing a user interface action to interact with a virtual environment presented through the wearable display device, the virtual environment including virtual content generated by an application running on the display system, the user interface action being determined based at least on a type of the action and the interactable area corresponding to the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input, the light pattern being based at least in part on the application running on the display system; and performing The method wherein the light guide is touch sensitive to receive user input.

11. the plurality of interactable areas include a first interactable area and a second interactable area; The method comprises: in response to determining that the user has activated the first interactive area, causing the light emitting assembly to illuminate a first light pattern associated with the user input within the first interactive area; in response to determining that the user has activated the second interactive area, causing the light emitting assembly to illuminate a second light pattern associated with the user input within the second interactive area; The method of claim 10 further comprising:

12. The method of claim 10 , wherein the actuation comprises at least one of a swipe, a tap, a press, or a touch gesture.

13. The method of claim 10 , wherein the light pattern comprises an arc-shaped region having one or more of a color, an arc length, or a visual effect.

14. The method of claim 10 , further comprising causing the display system to perform the user interface action based on the user input.

15. The method of claim 10 , wherein the optical waveguide is substantially circular.

16. The method of claim 10 , wherein the central touch surface is substantially circular.

17. The method of claim 10 , wherein the light guide completely surrounds the central touch surface.

18. The method of claim 10 , wherein the wearable display device is separate from the light-emitting user input device.

Images