Light-emitting user input device

Abstract

To provide a favorable light-emitting user input device.SOLUTION: A light-emitting user input device can include a touch sensitive portion configured to accept user input (e.g., from a user's thumb) and a light emitting portion configured to output a light pattern. The light pattern can be used to assist the user in interacting with the user input device. Examples include the steps of: emulating a multi-degree-of-freedom controller; indicating scrolling or swiping actions; indicating presence of objects nearby the device; indicating receipt of notifications; assisting pairing the user input device with another device; or assisting calibrating the user input device. The light-emitting user input device can be used to provide user input to a wearable device, such as, e.g., a head mounted display device.SELECTED DRAWING: Figure 17B

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 490,863, filed April 27, 2017, and entitled “LIGHT-EMITTING USER INPUT DEVICE,” the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly, to light emitting user input devices associated with imaging and visualization systems. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality or "MR" refers to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to produce VR, AR, or MR technologies that facilitate comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] Examples of light-emitting user input devices are disclosed. The user input device embodiments can be used to provide input to an AR, VR, or MR device. The light-emitting user input device can also provide visual information of events or objects associated with the AR / VR / MR device to a user or people within the user's environment.

[0005] The light-emitting user input device can include a touch-sensitive portion configured to receive user input (e.g., from a user's thumb) and a light-emitting portion configured to output a light pattern. The light pattern can be used to assist a user in interacting with the user input device. Examples include emulating a multi-degree-of-freedom controller, indicating a scroll or swipe action, indicating the presence of an object in the vicinity of the device, indicating receipt of a notification, assisting in the calibration of the user input device, or assisting in pairing the user input device with another device. The light-emitting user input device can be used to provide user input, for example, to a head-mounted display system such as a mixed reality display device. The present invention provides, for example, the following: (Item 1) 1. A system comprising: a light emitting assembly of a user input device, the light emitting assembly being configured to illuminate a plurality of light patterns to provide information of objects in an environment; a hardware processor communicatively coupled to the light emitting assembly, Identifying an object in a user's environment; accessing context information associated with the object; determining a characteristic of a light pattern illuminated by the light emitting assembly based at least in part on the contextual information; instructing the light emitting assembly to illuminate according to the light pattern; a hardware processor programmed to A system comprising: (Item 2) 2. The system of claim 1, wherein the object includes at least one of a physical object or a virtual object. (Item 3) 3. The system of claim 2, wherein the status of the virtual object includes at least one of a user's current interaction with the virtual object, whether the virtual object has received new information, whether the virtual object is idle, or whether the virtual object is in an error state. (Item 4) Item 2. The system of item 1, wherein the characteristics of the light pattern include at least one of brightness, position, shape, size, or color. (Item 5) 2. The system of claim 1, wherein the contextual information associated with the object includes at least one of a location of the object, an urgency of the object, a type of the object, a nature of the object, an amount of information associated with the object, or a preference of the user. (Item 6) Item 1, the system further comprising a wearable display device, the object being invisible from the user's view or external through the wearable display device, and the hardware processor programmed to determine at least one of a size, shape, or color of the light pattern and provide cues to the user for locating the object. (Item 7) 2. The system of claim 1, wherein the object is a component of a wearable system for presenting virtual content to a user, and the light pattern indicates a status of the component of the wearable system. (Item 8) 8. The system of claim 7, wherein the component includes at least one of the user input device, a wearable display device, or a battery pack. (Item 9) 2. The system of claim 1, wherein the status includes at least one of a battery status, a power charging status, a wireless pairing status between the wearable display device and the user input device, a status of a calibration process of the user input device, or a status of the wearable display device. (Item 10) 2. The system of claim 1, wherein the light pattern encodes an alert or information associated with the object. (Item 11) The status is: a current interaction of the user with the object; whether the object has received new information; whether the object is in an idle state, or Whether the object is in an error state Item 1. The system of item 1, comprising at least one of the following: (Item 12) 2. The system of claim 1, wherein the characteristics of the light pattern are configurable by a user via an application programming interface. (Item 13) 1. A light emitting user input device, comprising: A touch component configured to receive user input; a light emitting assembly configured to output a plurality of light patterns, the light emitting assembly at least partially surrounding the touch sensitive component; a hardware processor communicatively coupled to the touch component and the light emitting assembly, Identifying user interface actions supported by the touch component based on context information; determining a first light pattern associated with the user interface action; generating and transmitting instructions to the light emitting assembly to display a halo having the first light pattern; Receiving user input on the touch component; updating the halo with a second light pattern to reflect the user input; and a hardware processor programmed to 1. A light emitting user input device comprising: (Item 14) The context information comprises: the user's environment; the type of input supported by the light emitting user input device; information associated with an object with which the handheld user input device is configured to interact; or A characteristic of a wearable device associated with the handheld user input device. Item 14. The light-emitting user input device of item 13, comprising at least one of: (Item 15) Item 14. The light emitting user input device of item 13, wherein the light emitting assembly comprises a light guide and a ring of LEDs. (Item 16) Item 14. The light-emitting user input device of item 13, wherein the light-emitting assembly is configured to receive a plurality of user inputs, and the hardware processor is further programmed to display a halo based, at least in part, on the plurality of user inputs supported by actuating the light guide. (Item 17) Item 14. The luminous user input device of item 13, wherein the user input includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 18) Item 14. The light-emitting user input device of item 13, wherein the light-emitting user input device comprises at least one of a totem, a smartwatch, or a smartphone. (Item 19) Item 14. The light-emitting user input device of item 13, wherein the first light pattern provides a cue to a user that the user interface action is supported by the light-emitting user input device. (Item 20) The hardware processor further comprises: determining whether the received user input is inappropriate based on the context information; It is programmed to Item 14. The luminous user input device of item 13, wherein the second light pattern, in response to determining that the received user input is inappropriate, provides a visual alert that the received user input is inappropriate. (Item 21) Item 14. The light-emitting user input device of item 13, wherein at least a portion of the halo appears brighter or larger in the second light pattern compared to the first light pattern. (Item 22) Item 14. The luminous user input device of item 13, wherein characteristics of the plurality of light patterns are configurable by a user via an application programming interface. (Item 23) 23. The luminous user input device of claim 22, wherein the characteristics include at least one of an arc location or movement pattern, color, brightness, shape, or size. (Item 24) Item 14. The light-emitting user input device of item 13, wherein the second light pattern indicates at least one of a battery status, a power charging status, a wireless pairing status between the handheld user input device and another computing device, or whether the handheld user input device is idle in response to the user input. (Item 25) 1. A method comprising: Under the control of a hardware processor, identifying a type of user input supported by a light emitting user input device based on the context information, the light emitting user input device comprising a first element for illuminating a plurality of light patterns and a second element for receiving the user input; determining a first light pattern associated with the type of user input supported; generating and transmitting to the first element instructions to illuminate a halo having the first light pattern; determining a second light pattern based on user input on the light emitting user input device; updating the halo to the second light pattern in response to the user input; A method comprising: (Item 26) 26. The method of claim 25, wherein the first element is further configured to receive another user input, and the type of user input for determining the first light pattern is associated with other user inputs supported by the first element. (Item 27) The context information comprises: the user's environment; the type of input supported by the light emitting user input device; information associated with an object with which the handheld user input device is configured to interact; or A characteristic of a wearable device associated with the handheld user input device. 26. The method according to item 25, comprising at least one of the following: (Item 28) 26. The method of claim 25, wherein the type of user input includes at least one of a swipe, a tap, a press, or a touch input. (Item 29) 26. The method of claim 25, wherein the first light pattern provides a cue to a user that the type of user input is supported by the light-emitting user input device. (Item 30) determining whether user input received by the light emitting user input device is inappropriate based on the context information; Further comprising: 26. The method of claim 25, wherein the second light pattern, in response to determining that the user input is incorrect, provides a visual alert that the user input is incorrect. (Item 31) Item 26. The method of item 25, wherein at least a portion of the halo appears brighter or larger in the second light pattern as compared to the first light pattern. (Item 32) 1. A light emitting user input device, comprising: a plurality of interactable areas configured to receive user input, at least one interactable area of ​​the plurality of interactable areas including a portion of a light emitting assembly of the light emitting user input device; 1. A hardware processor comprising: detecting a user actuation of the light emitting user input device; determining an interactable area among the plurality of interactable areas corresponding to a user actuation; translating the user action into a user input for performing a user interface action based at least on a type of the action and the interaction area associated with the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input; a hardware processor programmed to 1. A light emitting user input device comprising: (Item 33) Item 33. The light-emitting user input device of item 32, further comprising a touch surface, the light-emitting assembly comprising an optical waveguide, and the plurality of interactable areas comprising a first interactable area associated with the optical waveguide and a second interactable area associated with the touch surface. (Item 34) Item 34. The light-emitting user input device of item 33, wherein in response to detecting that the user has activated the first interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a first light pattern associated with the user input in the first interactable area, and in response to detecting that the user has activated the second interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input in the second interactable area. (Item 35) Item 33. The light emitting user input device of item 32, wherein the light pattern is determined at least in part based on contextual information associated with the light emitting user input device. (Item 36) Item 33. The luminous user input device of item 32, wherein the user actuation includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 37) Item 33. The light emitting user input device of item 32, wherein the light pattern comprises an arc-shaped region having a color, arc length, or visual effect. (Item 38) 33. The light-emitting user input device of claim 32, wherein the hardware processor is further programmed to communicate with a wearable device and cause the wearable device to perform a user interface operation based on the user input. (Item 39) 1. A method comprising: Under the control of a hardware processor, Detecting a user actuation of a light emitting user input device, the light emitting user input device comprising a plurality of interactable areas; determining an interactable area among the plurality of interactable areas corresponding to a user actuation; translating the user action into a user input for performing a user interface action based at least on a type of the action and the interaction area associated with the action; causing a light emitting assembly of a light emitting user input device to illuminate a light pattern in response to the user input; A method comprising: (Item 40) 40. The method of claim 39, wherein the plurality of interactable areas comprises a first interactable area supporting a first type of user input and a second interactable area supporting a second type of user input. (Item 41) Item 41. The method of item 40, wherein in response to detecting that the user has activated the first interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a first light pattern associated with the user input in the first interactable area, and in response to detecting that the user has activated the second interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input in the second interactable area. (Item 42) Item 41. The method of item 40, wherein at least one of the first type of user input or the second type of user input depends on contextual information associated with a user's interaction with a wearable system, the contextual information including at least one of a type of application with which the user is interacting, available user inputs supported by the multiple interactable areas, or the user's virtual environment. (Item 43) 40. The method of claim 39, wherein the light emitting user input device comprises a touch surface divided into one or more interactable areas. (Item 44) 40. The method of claim 39, wherein the light pattern is determined at least in part based on contextual information associated with the light emitting user input device. (Item 45) 40. The method of claim 39, wherein the plurality of interactable areas comprises a light-emitting assembly of the light-emitting user input device. (Item 46) 40. The method of claim 39, wherein the user actuation includes at least one of a swipe, a tap, a press, or a touch gesture. (Item 47) 40. The method of claim 39, wherein the light pattern comprises an arc-shaped region having a color, arc length, or visual effect. (Item 48) 1. A system for calibrating a light emitting user input device, the system comprising: an outwardly facing imaging system configured to image the environment; a light emitting user input device configured to illuminate a light pattern; a hardware processor in communication with the outwardly facing imaging system and the light emitting user input device, accessing first movement data acquired by a sensor of the light emitting user input device associated with movement of the light emitting user input device from a first pose to a second pose; determining a first image of the light emitting user input device, the first image including a light pattern corresponding to the first orientation of the light emitting user input device; determining a second image of the light emitting user input device acquired by the outwardly facing imaging system, the second image including a light pattern corresponding to the second orientation of the light emitting user input device; analyzing the second image and calculating second movement data associated with movement of the light emitting user input device; Detecting a difference between the first movement data and the second movement data; calibrating a sensor of the light emitting user input device in response to determining that the difference reaches a threshold condition; and a hardware processor programmed to A system comprising: (Item 49) Item 49. The system of item 48, wherein moving the light-emitting user input device from the first pose to the second pose includes a change in at least one of a position or an orientation of the light-emitting user input device. (Item 50) Item 49. The system of item 48, wherein the first pose or the second pose corresponds to a base position of the light emitting user input device. (Item 51) Item 49. The system of item 48, wherein the light pattern is associated with a halo illuminated by a plurality of light emitting diodes surrounding the touchable portion of the light emitting user input device. (Item 52) Item 49. The system of item 48, wherein to calculate the second movement data, the hardware processor is programmed to calculate a change in a shape of a halo in the first image and the second image. (Item 53) Item 49. The system of item 48, wherein the light-emitting user input device is a totem for interaction with an augmented reality device. (Item 54) 54. The system of claim 53, wherein the sensor is part of an inertial measurement unit (IMU) of the totem, and the sensor is calibrated by adjusting at least one of the responsiveness of the totem to a user's movement or a mapping between the user's movement and the IMU measurements. (Item 55) Item 54. The system of item 53, wherein the totem is further configured to provide a visual indication to a user to move the light-emitting user input device from the first position to the second position. (Item 56) Item 54. The system of item 53, wherein the totem has three degrees of freedom. (Item 57) Item 49. The system of item 48, wherein the hardware processor is programmed to apply a computer vision algorithm to analyze the first image and the second image and to identify light patterns within the first image and the second image. (Item 58) Item 49. The system of item 48, wherein the light-emitting user input device is capable of illuminating another light pattern in response to determining that calibration of the light-emitting user input device is complete. (Item 59) Item 49. The system of item 48, wherein the hardware processor is programmed to determine a type of calibration, and a light pattern illuminated by the light emitting user input device corresponds to the type of calibration. (Item 60) 1. A method of calibrating a light emitting user input device, the method comprising: Under the control of a hardware processor, receiving movement data of a light-emitting user input device in a pose, the movement data being obtained by a sensor of the light-emitting user input device; receiving an image of the light emitting user input device in the pose; analyzing the image to identify a shape of a light pattern illuminated by the light emitting user input device; calculating at least one of a first position or a first orientation of the light emitting user input device in the pose based on the movement data; calculating at least one of a second position or a second orientation of the light emitting user input device in the pose based on a shape of the light pattern; determining a difference between the first position and the second position or the first orientation and the second orientation; calibrating a sensor of the light emitting user input device in response to determining that the difference reaches a threshold condition; and A method comprising: (Item 61) Item 61. The method of item 60, wherein the first position, the first orientation, the second position, or the second orientation is calculated with reference to a home pose. (Item 62) Item 61. The method of item 60, wherein the shape of the light pattern in the image is oval, while the shape of the light pattern at the base pose is circular. (Item 63) Item 61. The method of item 60, wherein at least one of the position or movement pattern of the light pattern corresponds to the type of sensor being calibrated. (Item 64) Item 61. The method of item 60, further comprising providing at least one of visual, auditory, or tactile feedback in response to determining that the calibration was successful. (Item 65) Item 61. The method of item 60, wherein calibrating the sensor includes adjusting at least one of a responsiveness of the sensor to user movement or a mapping between the user movement and measurements of the sensor. (Item 66) 1. A system for calibrating a light emitting user input device, the system comprising: an outwardly facing imaging system configured to image the environment; a hardware processor in communication with the outwardly facing imaging system, receiving an image of the environment; analyzing the image to identify a light emitting user input device; determining a first pose of the light emitting user input device and a first appearance of an illuminated halo light pattern; identifying a second appearance of the illuminated halo light pattern based at least in part on analysis of the image; and determining a first change to an attitude of the light emitting user input device; receiving movement data of the light emitting user input device measured by the light emitting user input device; calculating a second change to an attitude of the light emitting user input device based at least in part on analysis of the images; and calculating a difference between the first change and the second change and determining whether the difference reaches a threshold; calibrating a sensor of the light emitting user input device in response to determining that the difference exceeds the threshold condition; and a hardware processor programmed to A system comprising: (Item 67) Item 67. The system of item 66, wherein the attitude of the light-emitting user input device includes a position and an orientation of the light-emitting user input device. (Item 68) Item 67. The system of item 66, wherein the light-emitting user input device includes at least one of a totem, a smartwatch, or a smartphone, and the sensor is an IMU. (Item 69) Item 67. The system of item 66, wherein in response to determining that the difference does not exceed the threshold, the hardware processor is programmed to provide an indication that the sensor is calibrated. (Item 70) 70. The system of claim 69, wherein the indication comprises visual, auditory, or tactile feedback on the light-emitting user input device. (Item 71) Item 67. The system of item 66, wherein the first position is a base position of the light emitting user input device. (Item 72) Item 67. The system of item 66, wherein the light-emitting user input device is further configured to illuminate a series of light patterns to guide a user of the light-emitting user input device to position the user input device in a posture including the first posture and the second posture. (Item 73) Item 67. The system of item 66, wherein the hardware processor is programmed to determine a deformation of a shape of the light pattern in the second appearance relative to a shape of the light pattern in the first appearance to calculate a second change to an attitude of the light-emitting user input device based, at least in part, on analysis of the image. (Item 74) Item 67. The system of item 66, wherein the light-emitting user input device is part of a wearable system further comprising a wearable display for presenting virtual content within an augmented reality, virtual reality, or mixed reality environment. (Item 75) Item 67. The system of item 66, wherein to calibrate the sensor, the hardware processor is programmed to adjust at least one of the responsiveness of the sensor to a user's movement or the mapping between the user's movement and the sensor's measurements. (Item 76) 1. A method for calibrating a light emitting user input device, the method comprising: under control of a light emitting user input device comprising a light emitting assembly and a hardware processor; determining a first pose of the light emitting user input device; causing the light emitting assembly to illuminate a first light pattern to guide a user to move the light emitting user input device to a second position; in response to determining that the light emitting user input device has been moved to the second position; obtaining orientation data for said light emitting user input device; calibrating the light emitting user input device based at least in part on the pose data; and providing an indication to a user that the calibration process is completed; and A method comprising: (Item 77) Item 77. The method of item 76, wherein determining the first pose is based, at least in part, on data obtained from an inertial measurement unit (IMU) of the light emitting user input device. (Item 78) 77. The method of claim 76, wherein the indication includes at least one of an audible, visual, or tactile indication. (Item 79) Item 77. The method of item 76, wherein the indication includes a second light pattern. (Item 80) 77. The method of claim 76, wherein the pose data includes at least one of position or orientation data of the light emitting user input device. (Item 81) 77. The method of claim 76, wherein the light emitting user input device posture data comprises data obtained when the light emitting user input device is in a plurality of postures. (Item 82) Item 77. The method of item 76, further comprising causing the light emitting assembly to illuminate a third light pattern indicating the start of a calibration process. (Item 83) 1. A system for calibrating a light emitting user input device, the system comprising: one or more sensors configured to obtain movement data of the light emitting user input device; a light emitting assembly of the light emitting user input device configured to output a plurality of light patterns; 1. A hardware processor comprising: providing a first indication to a user to position the light emitting user input device in a position; obtaining movement data of the light emitting user input device to the pose; calibrating the light emitting user input device based at least in part on the movement data; and providing an indication to a user that the calibration process is completed; and a hardware processor programmed to A system comprising: (Item 84) Item 84. The system of item 83, wherein the first indication includes a light pattern, the location, movement, or combination of which provides guidance to a user for moving the light-emitting user input device to the certain pose. (Item 85) Item 84. The system of item 83, wherein the light-emitting user input device is part of a wearable system for interacting with an augmented or mixed reality environment, and the first indication includes a virtual image provided by a head-mounted display of the wearable system showing a pose of the totem to be positioned by a user. (Item 86) The hardware processor further comprises: in response to determining that the light emitting user input device has been calibrated, providing a second indication that the light emitting user input device has been calibrated; or In response to determining that the light emitting user input device is not calibrated, continuing to calibrate the light emitting user input device. Item 84. The system of item 83, programmed to: (Item 87) Item 84. The system of item 83, wherein the hardware processor is further programmed to detect a start condition for initiating a calibration process for the light-emitting user input device. (Item 88) Item 84. The system of item 83, wherein to calibrate the light-emitting user input device, the hardware processor is programmed to access an image including the light-emitting user input device in the certain pose, analyze the movement data and the image, and identify a difference in at least one of a position or orientation of the light-emitting user input device between that calculated from the movement data and that determined from the image. (Item 89) Item 84. The system of item 83, wherein the light-emitting user input device comprises a totem having a touch surface, the light-emitting assembly being positioned adjacent to the touch surface. (Item 90) 1. A wearable device for pairing a light emitting user input device, the wearable device comprising: an outwardly facing imaging system configured to image the environment; a hardware processor in communication with the outwardly facing imaging system, receiving an image acquired by the outwardly facing imaging system, the image including a light pattern illuminated by a light emitting user input device; Identifying a light emitting user input device to be paired with the wearable device; analyzing the image to identify the light pattern encoding information associated with the pairing of the light emitting user input device and a wearable device; extracting information encoded within the light pattern for pairing the light emitting user input device with the wearable device; pairing the light emitting user input device with a wearable device based at least in part on the extracted information; and a hardware processor programmed to A wearable device comprising: (Item 91) The hardware processor further comprises: determining whether pairing between the light emitting user input device and the wearable device is successful; and in response to determining that the pairing was successful, commanding the light emitting user input device to illuminate another light pattern indicating successful pairing. Item 91. The wearable device of item 90, programmed to: (Item 92) Item 91. The wearable device of item 90, wherein the hardware processor is further programmed to communicate with the light-emitting user input device via a wireless connection in response to determining that the pairing is successful. (Item 93) Item 93. The wearable device of item 92, wherein the wireless connection includes a Bluetooth® connection. (Item 94) Item 91. The wearable device of item 90, wherein the information encoded by the light pattern includes device information of the light-emitting user input device. (Item 95) Item 91. The wearable device of item 90, wherein the light pattern encodes the information in binary form. (Item 96) Item 91. The wearable device of item 90, wherein the light pattern includes one or more colors that encode information associated with the pairing. (Item 97) Item 91. The wearable device of item 90, wherein a portion of the light pattern includes light in a non-visible portion of the electromagnetic spectrum. (Item 98) Item 91. The wearable system of item 90, wherein the wearable device comprises a head-mounted display for presenting virtual content within a mixed reality environment, and the outwardly facing imaging system comprises a camera mounted on the head-mounted display. (Item 99) 1. A method for pairing light emitting user input devices, the method comprising: Under the control of a hardware processor, initiating a pairing process between the light emitting user input device and the electronic device; accessing an image acquired by a camera, the image including a light pattern illuminated by the light emitting user input device; identifying the light emitting user input device to be paired with the electronic device; analyzing the image to identify the light pattern encoding information associated with the pairing of the light emitting user input device and the electronic device; extracting information encoded within the light pattern for pairing the light emitting user input device with the electronic device; pairing the light emitting user input device with the electronic device based at least in part on the extracted information; and A method comprising: (Item 100) 100. The method of claim 99, wherein the electronic device includes a component of a wearable system for presenting virtual content in a mixed reality environment. (Item 101) Item 101. The method of item 100, wherein the electronic device includes another user input device or a head mounted display. (Item 102) 100. The method of claim 99, further comprising, in response to determining that the pairing process is successful, establishing a wireless connection between the light emitting user input device and the electronic device. (Item 103) Item 99. The method of item 99, wherein the information encoded by the light pattern includes device information for the light-emitting user input device, the device information including at least one of a device identifier, identification information regarding the light-emitting user input device, or a key for pairing the light-emitting user input device with the electronic device. (Item 104) Item 104. The method of item 103, wherein the light pattern encodes the information in binary form. (Item 105) 100. The method of claim 99, wherein the light pattern includes one or more colors that encode information of the pairing process. (Item 106) 100. The method of claim 99, wherein the portion of the light pattern includes light in a non-visible portion of the electromagnetic spectrum. (Item 107) 1. A system for pairing light emitting user input devices, the system comprising: a plurality of light emitting diodes (LEDs) configured to output a light pattern; 1. A hardware processor comprising: Initiating a pairing process with an electronic device; causing one or more LEDs of the plurality of LEDs to illuminate a first light pattern that encodes information regarding the pairing process; in response to determining that the electronic device has been successfully paired, illuminating a second light pattern indicating that the pairing was successful. a hardware processor programmed to A system comprising: (Item 108) Item 108. The system of item 107, wherein the first light pattern encodes at least one of device information associated with the user input device or a trigger message that causes another computing device to initiate a pairing process. (Item 109) The hardware processor includes: receiving a response from the electronic device during the pairing process; causing a second one or more LEDs of the plurality of LEDs to illuminate a third light pattern encoding a return message in response to the reply; Item 108. The system of item 107, programmed to: (Item 110) Item 108. The system of item 107, wherein the illumination of one or more of the plurality of LEDs encodes information of the pairing process in binary form. (Item 111) Item 108. The system of item 107, wherein a color associated with the illumination of the one or more LEDs encodes information about the pairing process. (Item 112) Item 108. The system of item 107, wherein a portion of the first light pattern or the second light pattern includes light in a non-visible portion of the electromagnetic spectrum. (Item 113) 1. A method comprising: Under control of a hardware processor of a first electronic device comprising a light emitting assembly for illuminating a plurality of light patterns, initiating communication between the first electronic device and a second electronic device; causing the first electronic device to illuminate a light pattern encoding a message for the communication; receiving a response from the second electronic device; providing an indication to a user of the first electronic device based, at least in part, on a response from the second electronic device; and A method comprising: (Item 114) Item 114. The method of item 113, wherein the communication includes a pairing process between the first electronic device and the second electronic device. (Item 115) Item 115. The method of item 114, wherein the first light pattern encodes at least one of device information associated with the user input device or a trigger message that causes another computing device to initiate a pairing process. (Item 116) Item 115. The method of item 114, wherein the indication includes a second light pattern illuminated by the first electronic device in response to determining that the first electronic device and the second electronic device have been successfully paired, indicating that pairing between the first electronic device and the second electronic device has been completed. (Item 117) Item 114. The method of item 113, wherein the indication includes a third light pattern illuminated by the first electronic device that encodes a return message in reply to the response. (Item 118) Item 114. The method of item 113, wherein the first light pattern encodes information in binary form within the message. (Item 119) Item 114. The method of item 113, wherein the light pattern includes one or more colors that further encode the message of the communication. (Item 120) 1. A light emitting user input device, comprising: 1. A touch pad assembly configured to receive user input, the touch pad assembly comprising: A touch surface; a touch sensor coupled to the touch surface, at least a portion of the touch sensor underlying the touch surface and configured to detect actuation of the touch surface; a light emitting assembly comprising an optical diffusing element and a plurality of light illuminating elements, the light emitting assembly coupled to the touch sensor and the touch surface and configured to display a plurality of light patterns; a printed circuit board coupled to the light emitting assembly and the touch sensor; a touch pad assembly comprising: a body for supporting the touch pad assembly; 1. A light emitting user input device comprising: (Item 121) Item 121. The luminous user input device of item 120, wherein at least a portion of the optical diffusing element is overlaid on the touch sensor, the touch sensor being further configured to detect actuation of the optical diffusing element. (Item 122) Item 121. The light emitting user input device of item 120, wherein the optical diffusing element comprises an optical waveguide. (Item 123) Item 121. The luminous user input device of item 120, wherein the optical diffusing element substantially surrounds the touch surface. (Item 124) Item 121. The light-emitting user input device of item 120, wherein the touch pad assembly further comprises an armature for holding the light-emitting assembly, the touch surface, and the touch sensor. (Item 125) Item 121. The light emitting user input device of item 120, wherein the plurality of light illumination elements comprise light emitting diodes. (Item 126) Item 121. The light-emitting user input device of item 120, further comprising a connection interface configured to establish a wireless connection with a wearable device configured to present mixed reality content. (Item 127) Item 121. The light-emitting user input device of item 120, wherein the body has an upper portion for supporting the touchpad assembly and a bottom portion configured to be removably mounted to a base. (Item 128) Item 121. The light-emitting user input device of item 120, wherein the body further includes at least one of a trigger, a bumper, or a home button for user interaction. (Item 129) Item 121. The light emitting user input device of item 120, further comprising a hardware processor programmed to control illumination of the plurality of light illumination elements.

[0006] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purport to define or limit the scope of the inventive subject matter. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0008] [Diagram 2] FIG. 2 illustrates diagrammatically an embodiment of a wearable system.

[0009] [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0010] [Figure 4] FIG. 4 diagrammatically illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0011] [Diagram 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0012] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal stereoscopic display, image, or bright field.

[0013] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0014] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0015] [Figure 9] FIG. 9 is a block diagram of another embodiment of a wearable system.

[0016] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0017] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0018] [Figure 12A] FIG. 12A illustrates a side and front (user-facing side) view of an embodiment of a totem.

[0019] [Figure 12B] FIG. 12B illustrates a top view of another embodiment of a totem.

[0020] [Figure 13A] FIG. 13A illustrates a cross-sectional view of an example touchpad of the totem.

[0021] [Figure 13B] FIG. 13B illustrates an example of touch screen technology.

[0022] [Figure 13C] 13C and 13D illustrate additional cross-sectional views of an example touchpad of the totem. [Figure 13D] 13C and 13D illustrate additional cross-sectional views of an example touchpad of the totem.

[0023] [Figure 13E] FIG. 13E illustrates a bottom view of an exemplary touchpad.

[0024] [Figure 14A] FIG. 14A illustrates a top view of an example touchpad of the totem.

[0025] [Figure 14B] FIG. 14B illustrates an overview of an example layout of LEDs associated with a touchpad.

[0026] [Figure 15]FIG. 15 illustrates an example LED layout or a pattern of light from an LED layout.

[0027] [Figure 16A] 16A and 16B illustrate example locations or movement patterns of the totem's light emissions. [Figure 16B] 16A and 16B illustrate example locations or movement patterns of the totem's light emissions.

[0028] [Figure 17A] FIG. 17A is a block diagram illustrating example components of a totem.

[0029] [Figure 17B] FIG. 17B is a side cross-sectional view illustrating components of another exemplary totem.

[0030] [Figure 18A] 18A-18D illustrate an example programming interface for configuring a location or movement pattern for light emission from a totem's halo. [Figure 18B] 18A-18D illustrate an example programming interface for configuring a location or movement pattern for light emission from a totem's halo. [Figure 18C] 18A-18D illustrate an example programming interface for configuring a location or movement pattern for light emission from a totem's halo. [Figure 18D] 18A-18D illustrate an example programming interface for configuring a location or movement pattern for light emission from a totem's halo.

[0031] [Figure 19A] 19A-19C illustrate an example of totem calibration using the halo of light emission from the totem. [Figure 19B]19A-19C illustrate an example of totem calibration using the halo of light emission from the totem. [Figure 19C] 19A-19C illustrate an example of totem calibration using the halo of light emission from the totem.

[0032] [Figure 19D] 19D and 19E illustrate an example of totem calibration using a light pattern associated with a halo. [Figure 19E] 19D and 19E illustrate an example of totem calibration using a light pattern associated with a halo.

[0033] [Figure 20A] 20A and 20B illustrate an example showing a wireless pairing process between a mixed reality device and a totem with a halo. [Figure 20B] 20A and 20B illustrate an example showing a wireless pairing process between a mixed reality device and a totem with a halo.

[0034] [Figure 20C] FIG. 20C illustrates an example process of device pairing with a halo.

[0035] [Figure 20D] FIG. 20D illustrates another example process of device pairing with a halo.

[0036] [Figure 21-1] FIG. 21A illustrates an example showing the status of a totem.

[0037] [Figure 21-2] FIG. 21B illustrates an example of a light location or movement pattern during the power on and off process.

[0038] [Figure 21-3]Figure 21C illustrates an example of a light location or movement pattern indicating battery charge status. Figure 21D illustrates an example light pattern when the totem goes into sleep mode.

[0039] [Figure 21-4] FIG. 21E illustrates an example process for indicating a totem's status based on light location or movement patterns.

[0040] [Figure 22A] 22A and 22B illustrate example light locations or movement patterns used as cues for user interaction. [Figure 22B] 22A and 22B illustrate example light locations or movement patterns used as cues for user interaction.

[0041] [Figure 22C] FIG. 22C illustrates another example embodiment in which light patterns are used to provide an indication of available user interface actions.

[0042] [Figure 23] FIG. 23 illustrates an example of using light patterns as an alert to indicate incorrect or inappropriate user interaction.

[0043] [Figure 24A] FIG. 24A illustrates an example light pattern for a swipe gesture.

[0044] [Figure 24B] FIG. 24B illustrates example light patterns for touch gestures.

[0045] [Figure 24C] FIG. 24C illustrates an example process for providing cues for user interaction on a totem.

[0046] [Figure 25A] FIG. 25A illustrates an exemplary interactive use of an optical waveguide.

[0047] [Figure 25B] FIG. 25B illustrates an example interactive use of a totem with two interactable areas.

[0048] [Figure 25C] FIG. 25C illustrates an exemplary process for interacting with a totem.

[0049] [Figure 26A] 26A and 26B illustrate an example of using a totem to interact with a physical object. [Figure 26B] 26A and 26B illustrate an example of using a totem to interact with a physical object.

[0050] [Figure 27] FIG. 27 illustrates an example of moving a virtual object using a six degree of freedom (6DOF) totem.

[0051] [Figure 28A] 28A and 28B illustrate an example that provides information of an object via the location and movement of a light pattern. [Figure 28B] 28A and 28B illustrate an example that provides information of an object via the location and movement of a light pattern.

[0052] [Figure 28C] FIG. 28C illustrates an example process for providing information associated with an object using light patterns.

[0053] [Figure 29A] FIG. 29A illustrates an example light location or movement pattern indicating receipt of a notification.

[0054] [Figure 29B] FIG. 29B illustrates an example process for providing notification using light patterns on a totem.

[0055] [Diagram 30] FIG. 30 illustrates example light patterns that may be used to inform people in the user's environment of the user's current interaction.

[0056] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] Overview of Illuminating User Input Devices A touch-sensitive user input device can support user inputs such as swipes, taps, clicks, presses, and the like. For example, when a user uses a touchpad to browse a website, the user can use one finger (e.g., thumb) to swipe left and right to move the webpage left and right, or tap up and down to move the webpage up and down. To achieve more user interface functionality, such as snapping content to a desired location, scrolling or resizing content, touch-sensitive user input devices often require multiple fingers. For example, a user may use two fingers to zoom in on a webpage and one finger to navigate within the webpage. However, when the touchpad is part of a handheld user input device for a wearable device (which may include an AR / VR / MR wearable display), the user may not have as many fingers available to interact with the touchpad. For example, a user may use the thumb to interact with the touchpad while using the other fingers to hold a totem. As a result, the type of user interface functionality that can be achieved by a touchpad on a conventional handheld device may be significantly reduced.

[0058] To remedy this problem, the touchpad of the totem embodiments described herein may be divided into multiple interactable regions, each of which may be mapped to one or more types of user interface interactions. For example, the touchpad may include a touch surface near the center of the touchpad and an outer region at least partially surrounding the touch surface. The outer region may include a light guide configured to output a light pattern (e.g., light location, illumination, color, and / or movement) that aids the user in interacting with the totem. The light pattern output by the light guide may sometimes be referred to herein as a "halo" because the light pattern may appear to surround the central touch-sensitive portion of the totem. The light guide may be on top of the touch sensor such that a user may interact with the light guide and provide touch-sensitive input to the totem via the light guide region. When a user activates the touch surface, the totem may simulate a cursor action (e.g., moving forward and backward on a browser, etc.). However, if the user desires to perform other types of user interface actions (e.g., scrolling a web page, etc.), the user may actuate the light guide. In some embodiments, the light guide may not be touch-sensitive. The user may actuate an area proximate the light guide (e.g., an area on the touchpad that may be surrounded by the light guide) to activate the totem.

[0059] As another example, a touch surface can be divided into two interactable regions, one region supporting a user's touch actions (e.g., simulating the functionality of a multi-degree-of-freedom (DOF) directional D-pad), while another region supports a user's swipe actions. A touch surface with two interactable regions may include concentric rings, with one inner ring (as a first interactable region) and one outer ring (as a second interactable region). In this example, optical waveguides surrounding the touch surface may be interactable or not, but may provide visual feedback related to the user's interaction or related to the wearable system. As will be further described herein, the type of user interaction supported by the interactable regions may be dynamically changed based on events in the virtual environment or objects the user interacts with. For example, the outer region may be used as a D-pad when the user is browsing the web, while the same region may support swipe interactions (e.g., circular swipes) when the user is playing a virtual game.

[0060] In some embodiments, the totem's light guide can provide cues for user interaction. For example, a halo can be used to inform the user of the type and location of available user interactions or to indicate a current user interaction. As an example, a light emitting diode (LED) below the light guide may illuminate to indicate to the user that the user can touch or tap a portion of the touch surface, where the LED will illuminate and select a virtual element on the wearable display. The LED may also be used in conjunction with haptic feedback (e.g., provided by a haptic actuator in the totem) or audio feedback (e.g., provided by a speaker in the wearable device) to provide an indication of or guide the user interaction.

[0061] When the totem is used in conjunction with a wearable device, the user interface experience can be extended to the 3D environment surrounding the user. However, the user's field of view (FOV) as perceived through the wearable display may be smaller than the natural FOV of the human eye, or smaller than the entire environment surrounding the user. Thus, there may be physical or virtual objects in the user's environment that are initially outside the FOV of the augmented reality display, but may subsequently move into the FOV of the wearable display (e.g., objects that may move relative to the user), or may subsequently become perceptible if the user's body, head, or eye posture changes (which would change the user's FOV). For example, in a gaming context, a user may attempt to find a robot avatar. If the robot is just outside the user's current FOV, the user may not receive a cue from the wearable display that the robot is nearby. If the user moves their head slightly, the robot may suddenly enter the user's FOV, which may surprise the user. Furthermore, if the user's FOV through the wearable display is relatively small, it may be difficult for the user to see the robot unless the user turns their head or looks directly at the robot.

[0062] To improve the user interface experience, a totem may provide information about objects outside the user's FOV. For example, a totem may provide a visual halo (e.g., emitted via a light guide) on an outer region of the touchpad for a corresponding object outside the user's current FOV. The light location or movement pattern of the halo may be used to indicate information associated with the object, e.g., a brighter or larger halo may indicate that the object is closer to the FOV, while a darker or smaller halo may indicate that the object is farther from the FOV. Similarly, the color of the halo may be used to indicate the type of object. For example, a competitor avatar (in a virtual game) may be associated with a red halo, while a companion avatar (in a virtual game) may be associated with a green halo. As another example, a flashing rainbow halo may indicate a system notification or warning. The light pattern of the halo may change as objects in the user's environment change or as the user changes posture.

[0063] Additionally or alternatively, the light pattern of the halo may be used to indicate the progress of the process. For example, while the totem is charging, the totem may display a halo that corresponds to the percentage of charge of the battery. For example, when the battery is only 25% charged, the totem may display 1 / 4 of the halo (e.g., a 90 degree arc). When the battery is charged to 100%, the totem may display the entire halo. The light location or movement pattern of the halo may also provide an indication of the user's interaction to persons in the user's environment. For example, when a user is recording a video using an augmented reality device, the LED halo may flash red to inform others in the vicinity that the display is in record mode so that they will not accidentally interrupt or interfere with the user's recording session.

[0064] The totem embodiments described herein may be programmable. For example, the location or movement of the light pattern may be customized, in various embodiments, by an application developer or by the user. The halo may be customized (e.g., via an application programming interface (API)) based on the type of application the user is interacting with. As an example, the light guide of the touchpad may be mapped to a four-way D-pad (corresponding to up, down, left, and right user interface actions) when the user is using a browser. As another example, the outer area of ​​the touch surface may be mapped to a three-way D-pad when the user is playing a racing game, where the three-way D-pad may correspond to turn left, turn right, and brake. The user may also customize the light location or movement pattern of the halo. For example, the user may turn off the halo or change the color of the halo associated with receiving an email notification. The user may customize the light pattern associated with the halo using the wearable display or by activating the totem. Detailed examples of the totem, halo, and user interface interactions using the totem and wearable device are described below.

[0065] Although the exemplary totem is described for use with a wearable system (or any type of AR, MR, or VR device), the exemplary totem and techniques described herein can also be used with other systems. For example, the totem may be used to interact with a projector or display (e.g., a television or computer display), a gaming system, an audio system, an Internet of Things (IoT) connectable device, or another computing device.

[0066] Example of a 3D display for a wearable system A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. A wearable system may comprise a wearable device that may present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device may include a wearable display device, such as, for example, a head mounted display (HMD). The wearable device may also include a belt pack, which may include a central processing unit that handles some of the data processing for the wearable device, a battery, etc. In some circumstances, a wearable device may be used synonymously with an augmented reality device (ARD).

[0067] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0068] In order for a 3D display to produce a true depth sensation, and more specifically a simulated sensation of surface depth, it is desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0069] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes and / or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0070] 2 illustrates an example of a wearable system 200. The wearable system 200 can include a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 can be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 can be positioned directly in front of the eyes of the user 210. A portion of the wearable system (such as the display 220) can be worn on the user's head.

[0071] In FIG. 2 , speaker 240 is coupled to frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). Wearable system 200 may also include an outwardly facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. Wearable system 100 may also include an inwardly facing imaging system 462 (shown in FIG. 4 ) that may track the user's eye movements. The inwardly facing imaging system may track either one eye movement or both eyes movement. The inwardly facing imaging system may be attached to frame 230 and may be in electrical communication with processing modules 260 and / or 270 that may process image information obtained by the inwardly facing imaging system and determine, for example, pupil diameter and / or orientation of the user's 210 eyes, eye movement, or eye posture.

[0072] As an example, the wearable system 200 can obtain images of the user's posture using the outwardly facing imaging system 464 and / or the inwardly facing imaging system 462. The posture may be used to determine the user's movement or to synthesize an image of the user. The images obtained by the outwardly facing imaging system 464 and / or the inwardly facing imaging system 462 may be communicated to a second user in a telepresence session to create a tangible sense of the user's presence within the second user's environment.

[0073] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-attached configuration).

[0074] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include data a) captured from sensors (e.g., which may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in the inwardly facing imaging system and / or the outwardly facing imaging system), microphones, inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, and / or gyroscopes, and / or b) obtained and / or processed using the remote processing module 270 and / or remote data repository 280, possibly for passing to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to a remote processing module 270 and / or a remote data repository 280 by communications links 262 and / or 264, such as via wired or wireless communication links, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other. The local processing and data module 260, the remote processing module 270, and the remote data repository 280 may each include a network interface to provide for communications via the communications links 262, 264.

[0075] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process the data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0076] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence movements of the two eyes relative to one another (i.e., pupil rotation such that the pupils move toward or away from one another, converging the gaze of the eyes to fixate on the object) are closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, a change in focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a matching change in vergence movements at the same distance, a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in vergence movements will induce a matching change in accommodation under normal conditions. Display systems that provide better matching between accommodation and vergence-divergence movements may produce more realistic and comfortable simulations of three-dimensional images.

[0077] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume a particular accommodated state, focusing objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length such that an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing a different presentation of an image for each eye 302 and 304, and by providing a different presentation of an image corresponding to each of the depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, while shown to be flat for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodative state. Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes.

[0078] Waveguide Stack Assembly FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2, of which FIG. 4 diagrammatically illustrates some portions in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of FIG. 2.

[0079] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers and / or structures to form air gaps).

[0080] The waveguides 432b, 434b, 436b, 438b, 440b and / or the multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits output surfaces of image launch devices 420, 422, 424, 426, 428 and is launched into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide to output an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0081] In some embodiments, the image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each of the corresponding waveguides 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of the image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0082] A controller 460 controls the operation of stacked waveguide assembly 480 and image input devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 460 may be part of processing modules 260 and / or 270 (illustrated in FIG. 2) in some embodiments.

[0083] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between their major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include a light extraction optical element 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light and propagating it within each individual waveguide and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective and / or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top and / or bottom major surface and / or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic pieces of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the material pieces.

[0084] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third upper waveguide 436b as originating from a second focal plane that is closer inward toward the person from optical infinity, which was the light from the next upper waveguide 434b.

[0085] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0086] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from its respective waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0087] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0088] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0089] In some embodiments, the number and / or distribution of depth planes or depths of field may be dynamically varied based on the size and / or orientation of the pupil of the viewer's eye. The depth of field may vary inversely with the size of the viewer's pupil. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the focal depth of the eye becomes distinguishable and may appear more in focus with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the accommodation of the eye from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus the user at different pupil sizes without changing accommodation.

[0090] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0091] In some embodiments, it may be desirable to have the exit beam meet the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0092] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV), and the imaging system 464 is sometimes referred to as the FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400. In some implementations of the wearable system 400, the FOR may include substantially all of the solid angle around the user of the display system 400, as the user can move their head or eyes and see objects around the user (in front of, behind, above, below, or to the sides of the user). Images obtained from the outward-facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) performed by the user, detect objects in the world 470 in front of the user, etc.

[0093] The wearable system 400 may also include an inwardly facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 466 may be used to capture images of the eye 410 and determine the size and / or orientation of the pupil of the eye 304. The inwardly facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size and / or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter and / or orientation of only one eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye posture and / or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU (e.g., accelerometer, gyroscope, etc.), accelerometer, gyroscope, etc.

[0094] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touch screen, joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem, a smartphone, a smartwatch, a tablet, or the like, combinations thereof, or the like. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A user may interact with the user input device 466 or objects (e.g., virtual or physical objects) in its environment, for example, by clicking a mouse, tapping a touchpad, swiping a touchscreen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way D-pad), pointing a joystick, wand, or totem toward an object, pressing a button on a remote control, or other interaction with the user input device. Actuation of the user input device 466 may cause the wearable system to perform user interface actions, such as, for example, displaying a virtual user interface menu associated with the object, animating the user's avatar in a game, etc. As described herein, the user input device 466 may be configured to emit light. The light patterns may represent information associated with an object in the user's environment, the user's interaction with the user input device 466 or the wearable device, etc.

[0095] In some cases, a user may use a finger (e.g., thumb) to press or swipe a touch-sensitive input device to provide input to the wearable system 400 (e.g., provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication. The user input device 466 may comprise an embodiment of a totem described herein. The totem may include a touch surface that may allow a user to actuate the totem, such as by swiping along a trajectory or tapping.

[0096] 5 shows an example of an output beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 480 may function similarly and that the waveguide assembly 480 includes multiple waveguides. Light 520 is launched into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light exits the waveguide as an output beam 510. Although the output beams 510 are shown as approximately parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., divergent output beam formation) depending on the depth plane associated with the waveguide 432b. It should be understood that a nearly collimated exit beam may refer to a waveguide with light extraction optics that outcouples light to form an image that appears to be set at a depth plane at a large distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0097] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, an image, or a bright field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal stereoscopic display, an image, or a bright field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersive planar waveguide 622b. The dispersive planar waveguide 622b may be similar or the same in at least some respects as the primary planar waveguide 632b, which has a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or the same in at least some respects as the DOE 632a. For example, the dispersive planar waveguide 622b and / or the DOE 622a may be made of the same material as the primary planar waveguide 632b and / or the DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0098] The relayed, exit pupil expanded light is optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b preferably relay the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is orthogonal to the first axis. It should be noted that the second axis can be a non-orthogonal axis to the first axis. The primary planar waveguides 632b expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b can relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 632b that relays and expands the light along the horizontal or X-axis.

[0099] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into a proximal end of a single mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 8 of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be, for example, plated on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) is also located in the core, center, inner periphery, or inner diameter of the tube 642.

[0100] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend the piezoelectric tube 642 independently in two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 644. By oscillating the piezoelectric tube 8 near the first mechanical resonant mode of the fiber cantilever 644, the fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0101] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned biaxially within an area filling a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 forms an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0102] The components of the optical coupler subsystem collimate the light emitted from the scanning fiber cantilever 644. The collimated light is reflected by the mirrored surface 648 into a narrow dispersive planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light propagates perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b by total internal reflection (TIR), and in so doing repeatedly intersects with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, and a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 622b via TIR.

[0103] At each point of intersection with DOE 622a, additional light is diffracted towards the entrance of primary planar waveguide 632b. By splitting the incoming light into multiple outcoupled sets, the exit pupil of the light is vertically expanded by DOE 4 in dispersive planar waveguide 622b. This vertically expanded light outcoupled from dispersive planar waveguide 622b enters the edge of primary planar waveguide 632b.

[0104] Light entering the primary waveguide 632b propagates horizontally (with respect to the view of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects with the DOE 632a at multiple points. The DOE 632a may be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern to generate both deflection and focusing of the light. The DOE 632a may have a low diffraction efficiency (e.g., 10%) such that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0105] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles that match the designed focal level.

[0106] Thus, these different paths can couple light out of the primary planar waveguide 632b by resulting in different fill patterns at the DOE 632a's multiplicity, focus level, and / or exit pupil at different angles. Different fill patterns at the exit pupil can be advantageously used to generate brightfield displays with multiple depth planes. Each layer or set of layers (e.g., three layers) in the waveguide assembly or stack may be employed to generate individual colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image brightfields with various focal depths. Other Components of a Wearable System

[0107] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a haptic sensation of pressure and / or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptic sensation may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the haptic device or component may be held by the user.

[0108] A wearable system may, for example, include one or more physical objects that are manipulable by a user to allow input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via the virtual keyboard and / or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0109] Examples of haptic devices and totems that can be used with the wearable devices, HMDs, ARDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety herein.

[0110] Exemplary Wearable Systems, Environments, and Interfaces The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0111] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs, etc. from user input devices 466) from various user systems 720a, 720b. The user systems 720a, 720b may comprise one or more user wearable systems (e.g., wearable system 200 and / or display system 220) and / or stationary room systems (e.g., room cameras, etc.). The wearable systems may use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward facing imaging systems, outward facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives and / or various cues. Image data acquired by a camera (such as an indoor camera or an exterior-facing imaging system camera) may be reduced to a set of mapping points.

[0112] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize and / or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.

[0113] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0114] Object recognition may be performed using a variety of computer vision techniques. For example, the wearable system may analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, image restoration, etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms, etc. (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.).

[0115] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine or Deep Neural Network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variables. In some embodiments, the wearable HMD can be configured to generate models for analysis of aggregated data using multiple techniques. Other techniques may include the use of predefined thresholds or data values.

[0116] The wearable system can also complement the recognized objects with semantic information, bringing the objects to life. For example, if the object recognizer recognizes that a set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that a set of points is a totem, the wearable system may attach the semantic information that the totem may be paired with the wearable system (e.g., via Bluetooth). Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene taking place in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, an object recognizer and other software components can map points collected from various images, recognize objects, etc. so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0117] 8 is a process flow diagram of an example embodiment of a method 800 for rendering virtual content in relation to recognized objects. Method 800 describes how a virtual scene may be presented to a user of an MR system (e.g., a wearable system). The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who resides in California.

[0118] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, and / or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a, 708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0119] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 comprises a map, which may include map data about the world. The map may reside in part locally on the wearable system, or in part in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may comprise images, data from sensors (such as inertial measurement devices, which generally include accelerometer and gyroscope components), and surface information about objects in the real or virtual environment.

[0120] The sparse representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find not only the location of various components in the world, but also what the world is composed of. Poses can be building blocks that accomplish many goals, including populating maps and using data from maps.

[0121] In one embodiment, the sparse point locations may not be completely correct by themselves and further information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process called stereoscopic vision 940, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of the depth map information may be fused together, some of which may be summarized using a surface representation. For example, mathematically definable surfaces are an efficient (e.g., for large point clouds) and easy to summarize input to other processing devices such as game engines. Thus, the output of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. Pose may also be an input to this fusion process 930, the output of which is an input to the map capture process 920. Sub-surfaces may interconnect to form larger surfaces, such as in topography mapping, where the map becomes a large-scale hybrid of points and surfaces.

[0122] Various inputs may be utilized to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be input to determine that a user of the system is playing a monster battle game with one or more monsters at various locations, that the monsters die or flee under various conditions (such as when the user shoots the monsters), walls or other objects at various locations, and the like. A world map may contain information about where such objects are relative to one another, which is another useful input to mixed reality. Posture relative to the world is an input as well, and plays an important role for nearly any interaction system.

[0123] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, and the like. To move around or play a game, for example, the user may need to command the wearable system 900 with respect to what they want to do. There are various forms of user control that can be utilized beyond just moving around in space. In one embodiment, an object such as a totem, another user input device, or a toy gun may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU that can help determine the situation that is occurring even when such activity is not within the field of view of any of the cameras).

[0124] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture left or right, stop, grab, hold, etc. for a button press. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or to "fist bump" with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as spreading the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0125] Eye tracking is another input (e.g., to track where the user is looking, control display technology, render at a specific depth or range). In one embodiment, eye vergence movement may be determined using triangulation, and then accommodation may be determined using a vergence movement / accommodation model developed for that particular person.

[0126] With respect to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a relative wide FOV or passive SLAM pair of cameras arranged on the sides of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and also capture hand gestures and totem / object tracking in front of the user's face. The cameras in the three pairs of cameras may be part of an outward facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include an eye tracking camera (which may be part of an inward facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0127] FIG. 10 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may designate one totem for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through the user's FOV camera or may be detected through sensors (e.g., haptic gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0128] Based at least in part on the detected gestures, eye poses, head poses, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame in block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. In block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user's interactions to the reference frame 1020, the system determines the user input in block 1040.

[0129] For example, a user may move a totem or physical object back and forth to indicate turning a virtual page and going to the next page, or moving from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, the real or virtual object may be selected as the user input. In some implementations, the vergence-divergence movement of the user's eyes can be tracked and an accommodation / vergence-divergence movement model can be used to determine the accommodation state of the user's eyes, which provides information about the depth plane on which the user is focused. In some implementations, the wearable system can use ray casting techniques to determine the real or virtual object along the direction of the user's head pose or eye pose. In various implementations, ray casting techniques can include casting a narrow beam of light with virtually no lateral width or casting a ray of light with substantial lateral width (e.g., a cone or frustum).

[0130] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2), or may be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a globe. Interactions with the user interface may be tracked using one or more cameras external to the system or part of the system (e.g., using inwardly facing imaging system 462 or outwardly facing imaging system 464, etc.).

[0131] 11 is a process flow diagram of an example of a method 1100 for interacting with a virtual user interface. The method 1100 may be implemented by a wearable system described herein.

[0132] In block 1110, the wearable system may identify a particular UI. The type of UI may be predetermined by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for the virtual UI. For example, data associated with the UI's borders, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical posture, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.

[0133] In block 1130, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a bright field display can project the virtual UI into one or both of the user's eyes. Once the virtual UI is created, the wearable system may simply wait for a command from the user to generate further virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.) and if it is recognized (block 1160), virtual content associated with the command may be displayed to the user (block 1170).

[0134] Additional examples of wearable systems, UI, and user experience (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety.

[0135] Example Totem Overview As described with reference to Figures 4 and 7-10, a user can use a user input device 466 (e.g., a totem, etc.) to perform various user interface operations on a display (e.g., display 220). Figure 12A illustrates an example embodiment of a totem 1200, illustrating a side view 1210 and a front (facing the user) 1250. The totem 1200 may be a user input device 466 (shown in Figure 4), alone or in combination with other user input devices. The totem 1200 can be sized and shaped to be handheld. Further examples of user input devices (e.g., totems, etc.) in one or more of the various embodiments disclosed herein are shown and described in U.S. patent application Ser. No. 15 / 683,677, filed Aug. 22, 2017, having an ornamental appearance identical or similar to the totem controller in U.S. Design Patent Application Ser. No. 29 / 575,031, filed Aug. 22, 2016 (both of the foregoing applications are incorporated herein by reference in their entireties).

[0136] The totem 1200 shown in FIG. 12A can include a body 1214 having a trigger 1212, a home button 1256, and a touchpad 1260, although more or less buttons, triggers, or features can be included in other example totems. The light guide 1264 can substantially surround the touchpad 1260. In the example shown in FIGS. 12A and 12B, the light guide 1264 is generally annular (e.g., circular ring) that substantially surrounds the circular touchpad 1260. In other embodiments, the touchpad 1260 and the light guide 1264 can be shaped differently. For example, the touchpad 1260 can be polygonal shaped (e.g., square, rectangular, hexagonal, etc.) or oval, and the light guide can have a shape (e.g., circular, oval, polygonal, etc.) that substantially surrounds the touchpad.

[0137] The trigger, home button, and touch pad can receive user input (e.g., by being pulled, pressed, or touched, respectively). The light guide 1264 can be illuminated and display light patterns. In some embodiments, the light guide 1264 is touch-sensitive and can receive user input. The totem 1200 can be removably attached to the base 1220 when not being held by a user. The base 1220 may include a power connection (e.g., to a wall socket) and can be used to charge the totem 1200 when the totem is attached to the base 1220.

[0138] Example of a trigger The trigger 1212 can be located on an upper portion of the totem body 1214 facing away from the user. The trigger 1212 can include a touch surface 1234 and a touch sensor (not shown in FIG. 12A ) that can receive user input. The touch sensor can sense a user's finger on (or near) the touch surface 1234 and the movement of the user's finger on the touch surface 1234. Additionally or alternatively, the trigger 1212 can include a button 1232 that the user can press. The button 1232 can include a pressure sensor that can detect when the user presses the button 1232. The button 1232 can be pressed 6-8 mm from its resting position. In some embodiments, the trigger 1212 can include multiple buttons, for example, select, back, options, etc.

[0139] The trigger 1212 may be implemented using, for example, an Alps force sensor. The trigger 1212 (or button 1232) can also be implemented using an analog or digital button. The trigger 1212 may be associated with a microprocessor. The microprocessor may reserve two general purpose input / output (GPIO) pins for various functions of the trigger 1212.

[0140] In some implementations, the trigger 1212 can be configured to provide haptic feedback. The trigger 1212 can include a haptic actuator, such as, for example, a linear resonant actuator (LRA), an eccentric rotating mass (ERM), a piezoelectric actuator, etc. For example, the trigger 1212 can employ an S-type LRA (e.g., an S-type Alps haptic on the analog button 1232, etc.) to generate vibrations.

[0141] A user can activate the trigger 1212 using various hand gestures. For example, a user can activate the trigger 1212 by touching, swiping, tapping, pressing, etc. The trigger 1212 can provide rich touch features, and various user interface interactions can be associated with different hand gestures used to activate the trigger 1212. For example, a user can switch between an AR interface and a VR interface by pressing the trigger 1212. As another example, a user can swipe a virtual object in and out of the user's FOV by swiping the touch surface 1234 of the trigger. User interface interactions can also be associated with the duration of the actuation. The totem 1200 can record the duration of the press and determine a user interface action based on the duration of the press. For example, pressing the trigger 1212 for a long duration (e.g., 3-5 seconds, etc.) may cause the wearable system to end a program (e.g., a movie, etc.), while a quick press of the trigger may cause the wearable system to select a virtual object in the user's line of sight.

[0142] The trigger may be actuated in conjunction with other components of the totem or the user's posture to perform user interface actions. For example, a user may press the trigger 1212 and swipe the touchpad 1260 to move a virtual object. As another example, a user may press the trigger 1212 with their right thumb and move their arm to the right to move a virtual object to the right.

[0143] The trigger 1212 can also be configured to provide an indication to the user that one or more user interface actions are available. For example, when the trigger 1212 is pressed for a long duration, the wearable system may allow the user to change a setting in the game. To indicate to the user that a setting can now be changed, the wearable system may provide haptic feedback (e.g., vibration, etc.) on the trigger 1212 (or the body of the totem). In addition to or as an alternative to providing an indication that a user interface action is available, haptic feedback can also be used to inform the user that they have activated the trigger 1212 using a certain gesture (e.g., a tap or press, etc.).

[0144] In other embodiments, some or all of the above-described functionality of the trigger 1212 can be implemented by other buttons or touch-sensitive surfaces of the totem 1200. For example, as described below with reference to FIGURE 17B, a button (also referred to as a bumper) can implement some or all of the functionality of the trigger 1212 (or the home button 1256).

[0145] Another embodiment of the totem 1200 is described with reference to Figures 17A and 17B.

[0146] Touchpad Example The totem may include a touchpad 1260 located on an upper front portion of the totem body 1214 facing towards the user. The touchpad 1260 may include a touch surface 1262 and a light guide 1264. In some embodiments, the structure and / or functionality of the touchpad 1260 may be substantially the same as or similar to that described in U.S. Patent Application Serial No. 15 / 683,677 (as noted above, which is incorporated by reference herein in its entirety). In the illustrated embodiment, the light guide 1264 substantially surrounds the touch surface 1262. The touchpad 1260 may include various interactable areas. For example, a user may activate the touch surface 1262, the light guide 1262, alone or in combination. The touch surface 1262 may further include multiple interactable areas, each area mapped to a type of user input (or user interface action).

[0147] The touch surface 1262 may be a circular surface with a diameter in the range of 27 mm to 40 mm. The touch surface 1262 may also be other shapes, for example, oval, rectangular, triangular, diamond, irregular, etc. In one implementation, the touchpad 1260 may have a traffic / pilot (T2P) latency of less than 50 ms.

[0148] The touch surface 1262 can be substantially surrounded by the light guide 1264. For example, in various embodiments, the light guide 1264 may be for an angular distance around the touch surface of more than 90 degrees, more than 180 degrees, more than 270 degrees, or up to 360 degrees. The light guide 1264 can be illuminated and display a halo with various locations and moving light patterns. The light guide 1264 can diffuse the light generated by the light sources 1330 (e.g., LEDs) of the touch pad such that the illumination from the individual discrete light sources is blended to display a halo. The light guide 1264 can comprise a diffusing optical element formed in a ring shape, for example, formed from a plastic or polymeric material, which can transmit (and diffuse) the light from the light sources 1330 to a viewer of the totem 1200 (see, for example, FIGS. 13A-14A). The light guide can be transparent or translucent. In various embodiments, the light guide 1264 may comprise a diffuser sheet, a diffuser film, an etched waveguide, a transmissive optical element with a layer of particles, an irregular surface, a holographic, a white surface, polished glass, polytetrafluoroethylene (PTFE or Teflon), milk glass, gray glass, colored gel, etc. Light guide embodiments comprised of optically diffusive materials may advantageously diffuse and scatter the light from the light sources 1330 such that the light guide appears to have an overall substantially continuous glow (when all of the light sources are illuminated) rather than appearing as discrete individual light sources (when the light guide is substantially transparent).

[0149] The touchpad may include several red, green, and blue (RGB) LEDs (e.g., 6-24 RGB LEDs, etc.). The light location or movement pattern of the halo may provide a visual indication of the user's interaction with the totem or other components of the wearable system, progress status associated with the wearable system, objects in the user's environment, etc. See, for example, LED 1383a in FIG. 13C or LED 1394a in FIG. 13E.

[0150] The light sources may (in addition or alternatively) include emitters that irradiate in non-visible portions of the electromagnetic spectrum, e.g., infrared or ultraviolet. In such embodiments, a camera on the wearable device may be sensitive to the corresponding non-visible spectrum and can image the non-visible light displayed by these emitters. Thus, the totem 1200 and the wearable device can exchange information via such non-visible spectrum modalities. For example, the totem 1200 and the HMD can exchange device pairing information using non-visible spectrum modalities.

[0151] The touchpad 1260 can include a haptic component under the touch surface 1262 and a portion of the light guide 1264. The haptic component can include a haptic actuator for providing haptic feedback to the user via the touch surface 1262 or the light guide 1264. The haptic feedback can be used alone or in combination with a visual halo to provide an indication of the user's interaction with the totem or an object in the user's environment.

[0152] The haptic component may also include a touch detector for detecting user actuation of the touchpad. The haptic component may be implemented using a tough type Alps LRA. The haptic component may include a strain gauge with an analog-to-digital converter (ADC).

[0153] The touchpad 1260 can employ hybrid control techniques, such as, for example, a trackpoint-style hybrid velocity control input technique, a hybrid force and velocity technique, etc. The touchpad 1260 can use an ADC to enable such input techniques. In some implementations, the sensitivity of the touchpad can be less than 5 Newtons. Additional structural examples of the touchpad 1260 are described with further reference to FIGS. 13A and 14E.

[0154] The touchpad 1260 can provide a rich set of touch features for a variety of user interface experiences. For example, a user can use the touchpad 1260 (alone or in conjunction with other components of a totem or wearable system) to move or orient virtual objects. The touchpad 1260 can also be used as a virtual keyboard to enter text.

[0155] Example of a totem body The body 1214 of the totem 1200 may be in the shape of a flashlight (see, e.g., FIG. 12A), with an elongated cylinder that may be angled outward toward the top of the totem to create a more frusto-conical shape (which may be easier or more comfortable to hold in a user's hand). The body 1214 may include recesses for the user's fingers, which may aid in gripping and holding the totem. FIG. 12B is a top view 1270 illustrating another example of the totem 1200. In this embodiment, the body 1214 is oval shaped so that it may fit better in a user's palm or be more stable when placed on a surface (e.g., a table). In some embodiments, the decorative appearance of the body 1214 of the totem 1200 may be substantially the same as or similar to that of the totem controller shown and described in U.S. Design Patent Application No. 29 / 575,031, filed August 22, 2016 (as noted above, which is incorporated herein by reference in its entirety). Other shapes for the body 1214 are also possible, such as an elongated body having a cross-section that is ovular, polygonal, etc. The cross-section of the body 1214 may be substantially symmetrical such that the totem 1200 may be easily held in a user's left hand or a user's right hand.

[0156] The body 1214 can include an upper portion 1242, which can include a trigger 1212 and a touchpad 1260. The body 1214 can also include a bottom portion 1244. The bottom portion 1244 can include an interface such that the totem 1200 can be removably attached to the base 1220. For example, the bottom portion 1244 can be connected to the base 1220 via a connection interface 1222 (e.g., a USB-C connection, etc.) when the totem 1200 is charged. As another example, the bottom portion 1244 can be connected to another computing device (such as a home computer) via the base 1220. Once connected to a computer, a user can configure the light location or movement pattern of the halo (on the touchpad 1260) as appropriate via a programming interface on the computing device (described below with reference to FIGS. 18A-18D).

[0157] The touchpad 1260 can be angled such that the touch surface 1264 and the light guide 1262 are easily viewable by a user when the totem 1200 is held in the user's hand. The angle of the touchpad 1260 with respect to the horizontal can be in the range of about 10-60 degrees. It follows that the body 1214 can assume a more frusto-conical shape with such an angle. As explained above, this shape can make the totem 1200 easier or more comfortable for a user to hold in their hand. In some embodiments, the angle of the touchpad 1260 with respect to the horizontal can be in the range of about 15-30 degrees. The trigger 1212 can be positioned opposite the touchpad 1260 such that it is easily depressible with the user's index finger when the totem 1200 is held in the user's hand.

[0158] In some embodiments, the bottom portion 1244 may include an illuminated power indicator. The illuminated power indicator may include one or more RGB LEDs. The light location or movement pattern of the LED power indicator may be used to provide information related to the remaining battery or battery charge status for the LED totem.

[0159] The body 1214 can be configured to provide haptic feedback to the user. The body 1214 can include a haptic actuator 1760 (shown in FIGS. 17A, 17B) for providing haptic feedback. The haptic actuator 1760 may be implemented with tough-type LRA or ERM haptics. Details of the components associated with the body 1214 are described with reference to FIGS. 17A, 17B.

[0160] Home button example The totem 1200 may include a home button 1256. The home button 1256 may be located on the body 1214 of the totem 1200. When a user activates the home button 1256, the wearable system may perform various user interface actions, such as returning to / opening a main menu of an application or display, exiting a program, selecting an application, etc. In some embodiments, when a user activates the home button 1256, the wearable system may enter or wake up from a sleep mode, turn virtual content on the display on or off, etc.

[0161] The wearable system may perform an action based on how the user actuates the home button 1256. For example, if the user actuates the home button 1256 for a long duration, the display may be turned off. As another example, if the user presses the home button 1256 twice, the display may present the main page of the display, while if the user presses the home button once, the display may present the main menu of the gaming application with which the user is currently interacting. Other functionality may also be implemented by the home button 1256, for example, a quick tap of the button may invoke an application launcher user interface, and within an application, a quick tap may invoke the application's menu, or a long press may pause the application and invoke the application launcher user interface.

[0162] The home button 1256 can include an RGB LED indicator that can be used to indicate the current status of the totem or wearable display (such as whether it is turned on / off), the battery status of the totem, the wearable display, or other components of the wearable system.

[0163] In various embodiments, the totem 1200 includes both the home button 1256 and the trigger 1212, or includes only one of these user input elements (or an additional button such as the bumper 1266 described with reference to FIG. 17B). The totem 1200 can be configured or shaped differently than shown. For example, the home button 1256 may be implemented as part of the trigger (rather than as part of the body 1214). As yet another example, the touchpad 1260 or the home button 1256 can provide a visual indication as to the progress of charging the totem 1200. Many variations are possible, and the accompanying figures illustrate example embodiments and are not intended to limit the scope of the disclosure.

[0164] Example of base The body 1214 of the totem 1200 may be removably attached to the base 1220 as described with reference to the totem's body 1214. The interface between the base and the totem may allow charging, access data within the totem, provide software updates, and the like. For example, the base 1220 may include a Universal Serial Bus (USB) plug, which may be plugged into a USB port on the bottom portion 1244 of the body 1214. The base 1220 may further be connected to an electrical outlet using a power cord to charge the totem. The base may also connect to other computing devices, such as personal computers, televisions, projectors, etc., using wired or wireless communication channels.

[0165] Example of activating a totem The totem 1200 can support a variety of input techniques, such as, for example, touch, swipe, scroll, dial rotate, click, press, D-pad emulation, etc. In some embodiments, the totem 1200 can support inertial swipes. For example, as a user swipes along a trajectory on the touchpad 1260, a virtual object may continue to move along the trajectory even when the user's finger stops swiping. The totem 1200 can also support edge swipes, such as, for example, a swipe on a horizontal or vertical edge of the touchpad 1260 or light guide 1264.

[0166] A user can actuate the totem 1200 using various hand gestures and body postures. For example, the user can swipe or touch the touchpad 1260 or the trigger 1212. In some embodiments (such as embodiments 1210 and 1250), the totem 1200 can support 6DOF of user interaction. For example, the user can rotate the totem 1200 and rotate the game avatar. As will be further described with reference to FIGS. 17A, 17B, the totem 1200 can detect user movement using environmental sensors 1710 (such as an IMU). In other embodiments (such as the example totem 1200 shown in FIG. 12B), the totem can support 3DOF (rather than 6DOF). In these embodiments, the totem can detect and translate the user's forward, backward, left, right, and right-height movements, but cannot detect and translate pitch, yaw, and roll movements, or vice versa.

[0167] Although the examples are described with reference to handheld devices, similar techniques may also be applied to user input devices designed for seated and tabletop control.

[0168] Exemplary Structures of Touchpads Touchpad cross section 13A illustrates a cross-sectional view of an example touchpad 1260 of the totem 1200. The example touchpad 1260 may be part of the totem 1200 described with reference to FIGS. 12A and 12B. The touchpad 1260 may include three layers: a top layer 1312, a middle layer 1314, and a bottom layer 1316.

[0169] The bottom layer 1316 can include a printed circuit board (PCB) 1340. The PCB 1340 can mechanically support and electronically connect to one or more components on the middle layer 1314, such as light sources 1330 (e.g., LEDs) and touch sensors 1350. The PCB 1340 can also support the armature 1320. Additionally or alternatively, the PCB 1340 can electronically connect to other components of the totem, such as, for example, haptic actuators (e.g., actuators for the touchpad 1260, trigger 1212, or body 1214, etc.), a microprocessor, etc.

[0170] The central layer 1314 of the touch pad 1260 may include a touch sensor 1350. The touch sensor 1350 may be an example of a haptic component described with reference to FIG. 12A. The touch sensor 1350 may use various touch screen technologies (e.g., resistive or capacitive sensor technologies, etc.) to determine a user's actuation of the touch pad 1260 or provide tactile feedback. The touch sensor 1350 may be configured to enable multi-touch technologies that may distinguish between different levels of force applied to the touch surface 1262 (and the optical waveguide 1264). The touch sensor 1350 may provide sensitivity in three orthogonal directions (e.g., XYZ). Exemplary touch screen technologies that may be employed by the touch sensor 1350 are described in FIG. 13B.

[0171] In FIG. 13A, the LEDs 1330 can be positioned between the touch sensor 1350 and the armature 1320. The center layer 1314 can include 6-12 LEDs, although other numbers (e.g., 3, 15, 18, 20, etc.) of LEDs are also possible. The LEDs can be positioned substantially uniformly around the circumference of the light guide (see, e.g., top views in FIGS. 13E and 14A), or some LEDs may be positioned closer to each other rather than uniformly spaced. The LEDs 1330 may comprise monochromatic LEDs (e.g., blue-violet LEDs, etc.), bi-color LEDs, RGB LEDs (or other multi-color LEDs), white LEDs, organic LEDs (OLEDs), quantum dot LEDs, infrared LEDs, ultraviolet LEDs, etc., alone or in combination. The LEDs 1330 may be controlled by a microprocessor (e.g., on a PCB board 1340). For example, by using a microprocessor to independently adjust each color (red, green, and blue) of the RGB LEDs, a wide range of colors (e.g., a large color gamut) can be generated by the RGB LEDs. The light patterns generated by the LEDs can be associated with a user interaction with the totem or wearable system and associated with objects in the user's environment. In some examples, the light patterns can also be referred to as halo light locations or movement patterns.

[0172] The armature 1320 may straddle the top layer 1312 and the middle layer 1314. The armature 1320 may hold the middle layer 1314 and the top layer 1312 together. A portion of the armature may touch a top portion of the LED 1330.

[0173] The top layer can include a portion of the armature 1320 as well as a light guide 1264 and a touch surface 1262. The touch surface 1262 can be surrounded by the light guide 1264 and can sit on top of the touch sensor 1350. A user can activate the touch surface 1262 using hand gestures. The touch surface 1262 can also provide tactile feedback to the user. A portion of the light guide 1264 can be overlaid on top of the touch sensor 1350 and the user can activate the light guide 1264 accordingly. In some embodiments, the light guide may not be touchable (e.g., because the touch sensor 1350 is not under the light guide 1264). However, the light guide can still provide visual or tactile feedback.

[0174] The light guide 1264 may be between the armature 1320 and the touch surface 1262 on the top layer 1312. The light guide 1264 may include a first portion 1362 and a second portion 1364. The first portion 1362 of the light guide 1264 may be overlaid on top of the LEDs 1330. The light guide 1264 may diffuse the light generated by the LEDs 1330 to spread the light over an area of ​​the light guide 1264 that is larger than an individual LED package, which may provide a more appealing visual appearance. The light patterns may provide visual feedback associated with a user's interaction with the wearable system or touch pad 1260. The light patterns may also provide information associated with objects in the user's environment, such as the relative position between the user and the object.

[0175] A second portion 1364 of the light guide 1264 may be positioned over the touch sensor 1350. For example, the light guide 1264 may be positioned with 0.75 mm to 2 mm recessed over the touch sensor 1350. A portion of the touch sensor 1350 may extend under the light guide 1264 so that the light guide 1264 can not only provide visual and tactile feedback, but can also be activated by a user. As shown by finger 1310, a user can activate the light guide 1264 by, for example, touching, tapping, swiping, or depressing the light guide 1264. Thus, when a user activates the light guide 1264, the totem can simulate a user interface experience, such as, for example, selectively tapping on an illuminated "button" or "arc" of light on the light guide.

[0176] The light guide 1264 does not need to be positioned over the touch sensor 1350. As a result, a user may not be able to activate the light guide 1264 and interact with the totem or wearable device. In some implementations, the second portion 1364 is positioned over the touch sensor 1350, but the processor 1770 or local processing and data module 260 may be configured not to recognize a user's input on the light guide 1264. As a result, the light guide 1264 may be used to provide visual or tactile feedback, but may not be responsive when a user activates the light guide 1264.

[0177] 13B illustrates examples of touchscreen technology. In these examples, touch sensors 1350 (e.g., 1350a, 1350b, and 1350) are sandwiched between the touch surface 1262 and the PCB 1340. Touchscreen technology embodiment 1362 illustrates an example of force under touch technology. In this example, touch sensor 1350a can comprise a pressure-sensitive sensor that can detect multiple levels of force applied on the touch surface 1262. In an embodiment, touch sensor 1350 can also be configured to relay tactile feedback to the touch surface 1262.

[0178] In example 1364, touch sensor 1350b may include a strain gauge. The strain gauge may be located under an edge of touch surface 1262. When a user activates touch surface 1262, touch sensor 1350b can determine the user's gesture (e.g., a press or tap gesture) by measuring the strain of touch surface 1262.

[0179] Example 1366 illustrates an Interpolated Force Sensitive Resistive touch sensor technology (IFSR). The touch sensor 1350c can use force sensitive resistors whose level of sensitivity can change depending on the level of pressure. Thus, the touch sensor 1350c can detect multiple levels of force applied on the touch surface 1262.

[0180] 13C and 13D illustrate additional cross-sectional views of an exemplary touchpad of the totem. Cross-sectional view 1380 shows top cover 1381a and top housing 1381b below top cover 1381a. Either top cover 1381a or top housing 1381b or a combination thereof may be part of the armature 1320 shown in FIG. 13A. The top housing may be attached to the light guide 1264 (e.g., light pipe 1381c) with an adhesive. The light pipe can be transparent or translucent. In various embodiments, the light pipe 1381c can comprise a diffuser sheet, diffuser film, etched waveguide, a transmissive optical element with a layer of particles, an irregular surface, a holographic, a white surface, ground glass, polytetrafluoroethylene (PTFE or Teflon), milk glass, gray glass, colored gel, etc. An embodiment of a light pipe made of an optically diffusing material may advantageously diffuse and scatter the light from LED 1383a such that the light output in direction 1384 appears to have an overall substantially continuous glow (when all LEDs are illuminated) rather than appearing as discrete individual LEDs (if the light pipe were substantially transparent).

[0181] The top housing 1381b can also be attached to white Mylar 1381c (or other reflective sheeting) using adhesive. The top housing can encapsulate one or more LEDs 1383a. For example, the top housing may include an RGB LED, which includes a package of red, green, and blue LEDs. Thus, if a touchpad includes 12 RGB LEDs, the touchpad may include 12 top housing structures, each including an RGB LED. As another example, the top housing may include all 12 RGB LEDs for the touchpad.

[0182] The top cover is adjacent to a touch cover 1383d. The touch cover may be an embodiment of the touch surface 1262 shown in FIG. 13A. The touch cover may be attached to a touch board 1383 using touch cover adhesive 1383c. The touch board includes an embodiment of a haptic component as described with reference to FIG. 12A. As further described with reference to FIG. 13D, an adhesive stack 1386 may be interposed between a portion of the touch board and a portion of the top housing. The adhesive stack 1386 may be used to securely attach the portion of the touch board and the top portion of the top housing.

[0183] The adhesive stack 1386 may include two layers: a shock absorbing foam pad (e.g., Poron available from Rogers Corp., Rogers, CT) and a flexible and adhesive layer. The top portion of the Poron pad 1387a may be attached to the bottom portion of the touchboard, and the bottom portion of the Poron pad may be attached to the flexible and adhesive layer 1387b. The flexible and adhesive layer may be attached to the upper housing. The Poron pad may consist of two 0.2 mm thick adhesives and one 1.00 mm thick adhesive. The one 1.00 mm thick adhesive may be positioned between the two 0.2 mm thick adhesives. Alternatively, one of the two 0.2 mm thick adhesives may be between the 1.00 mm thick adhesive and the other 0.2 mm thick adhesive.

[0184] Referring back to FIG. 13C, one or more LEDs 1383a may be attached to the touch board. The LEDs may emit light in direction 1382. The light may reach a light pipe, which may transmit the light through total internal reflection (TIR) ​​to light output direction 1384. The bottom portion of the light pipe may also be attached to a piece of white Mylar. The white Mylar (or other reflective sheeting) may reflect some of the light emitted by the LEDs, increasing the amount of light output in light output direction 1384. The light pipe may spread the light from the LEDs so that the light output appears as a more angularly continuous output rather than individual discrete sources of light.

[0185] Touchpad bottom view FIG. 13E illustrates a bottom view of an exemplary touchpad. The touchpad 1390 may be an embodiment of the touchpad 1260. The touchpad 1390 may include twelve LEDs 1394a, although other numbers of LEDs are also possible. The twelve LEDs 1394a are equally spaced circumferentially in this example, although in other examples (such as the examples described with reference to FIGS. 14B and 15), the spacing between the LEDs may not necessarily be equal. The twelve LEDs may be located on the bottom of the touchpad. The touchpad can selectively illuminate the LEDs to create a light pattern, which may appear as a halo around the periphery of the touchpad.

[0186] The LEDs can be attached to a housing 1392. The housing can have two alignment pin holes and three snap gaps 1394e. The two alignment pin holes can be on opposite ends of the diameter of the touchpad. The three snap gaps can be equally spaced. The alignment pin holes 1394d and the snap gaps 1394e can be used to orient and position the touchpad in relation to the totem body. The snap gaps 1394e can also be used to secure the touchpad 1390 to the totem body.

[0187] The touchpad 1390 can employ touch screen technology such as, for example, a force flex stack 1394b. A portion of the force flex stack 1394b can be overlaid on top of the touch board component area 1394c. The force flex stack 1394b can include the touch surface 1262, either alone or in combination with the touch sensor 1350. The force flex stack can detect user input on the touchpad 1390, such as, for example, presses, swipes, taps, touches, etc.

[0188] In one embodiment, the touchboard component area may include a PCB 1340. The touchboard component area may include haptic components (e.g., haptic actuators, etc.), a microprocessor, one or more communication units (e.g., ADCs, etc.), etc.

[0189] Touchpad top view Figure 14A illustrates a top view of an example touchpad of a totem. The touchpad 1260 in Figure 14A can include an armature 1320, a light guide 1264, and a touch surface 1262 (which can be further divided into interactable areas 1262a and 1262b). The light guide 1264 can be positioned on top of a light source (e.g., an LED) 1330 (shown in dashed lines in Figure 14A).

[0190] Advantageously, in this embodiment, the touch pad 1260 can be divided into three interactable areas: the light guide 1264, the first part of the touch surface 1262a, and the second part of the touch surface 1262b. As described with reference to Figures 12 and 13A, each interactable area may be mapped to a type of user interaction. For example, a user can touch the light guide 1264 and swipe the second part of the touch surface 1262b to move web content up / down while moving back / forth on a browser. A user can also swipe in a circular motion on the first part of the touch surface 1262a to move a virtual object closer to (or farther from) the user.

[0191] Although this example includes three interactable areas, the totem may include more or fewer interactable areas in other embodiments. For example, rather than dividing the touch surface into two interactable areas (corresponding to the first portion 1262a and the second portion 1262b), the touch surface may be divided into four interactable areas, each occupying a quadrant of the touch surface. The touch surface may also include only one interactable area. Additionally or alternatively, the light guide 1264 may not be interactable, for example, because the processor 1770 or another module of the wearable system may not be programmed to recognize user input from the light guide 1264 or because the touch sensor 1350 does not extend down the light guide 1264.

[0192] The type of interaction mapped to the touch area may also be customized based on the type of user interaction. For example, when a user is watching television, the touchpad may simulate a four-way D-pad (up, down, left, right) on the light guide 1264. However, if the user is browsing a web page, the light guide 1264 may support back / front user interface actions.

[0193] The interactable areas can also provide visual or tactile feedback to the user. For example, the touch surface 1262 can be configured to provide tactile feedback, while the light guide 1264 can provide both tactile and visual feedback (e.g., via the light location or movement pattern of an LED halo).

[0194] Light source layout FIG. 14B illustrates an overview of an example layout of light sources (e.g., LEDs) associated with a touchpad. FIG. 14B shows a touchpad 1450 having a visible circular surface 1452 with 12 LEDs (illustrated as LED0, LED2...LED11) on its periphery. These 12 LEDs can be used to represent specific information when they are illuminated in a specific sequence. For example, each LED may be assigned a bit, thus resulting in a 12-bit word. As described with reference to FIGS. 24B, 20A, and 20B, an outwardly facing imaging system 464 of the wearable device can capture the display sequence of the LEDs. The wearable device can then analyze the images and extract information from the display sequence. The extracted information may be used, for example, to pair a totem with the wearable device, calibrate the totem, etc. For example, when LEDs 0, 3, 6, 9 are illuminated, the wearable device may recognize that pairing between the wearable device and the totem is initiated. Thus, the wearable device can search for wireless (e.g., Bluetooth) devices in its environment and determine if there are any new devices (such as a totem associated with an LED). When the LEDs are multi-LEDs (e.g., RGB LEDs), the amount of information represented by the LED light pattern can increase (e.g., more than a 12-bit word). For example, a blue light on LEDs 6, 7, 8 can indicate that the totem is in pairing mode, while a yellow light on LEDs 6, 7, 8 can indicate that the totem is in charging mode.

[0195] In some embodiments, the LEDs may be split (e.g., multiplexed) into groups that are illuminated separately. Each group may be used to represent specific information. FIG. 14B shows four example groups (illustrated as MUX0 (1454a), MUX1 (1454b), MUX2 (1454c), and MUX3 (1454d)), each group containing three LEDs. For example, when the LEDs in MUX1 are illuminated (e.g., LEDs 3, 4, 5), the wearable device may know that the totem is currently indicating an upward movement of the totem, while when the LEDs in MUX0 are illuminated, the wearable device may know that the totem is being rotated to the left, and so on.

[0196] In other embodiments, a different number or layout of light sources or multiplexed groups can be utilized and different amounts of information (e.g., word lengths different from 12 bits) may be represented by illuminated groups or sequences of LEDs.

[0197] As previously mentioned, with reference to FIG. 12A, the light source can emit light in the non-visible spectrum (e.g., infrared or ultraviolet light). Such light can be captured by the wearable device, and light patterns in the non-visible spectrum can also be used to communicate information from the totem to the wearable device. The information may include device information of the totem (e.g., for pairing the totem with the wearable device), or other types of information, such as, for example, the status of the totem (e.g., whether low battery, whether a connection has been established, devices currently paired with the totem, etc.).

[0198] 15 illustrates an example LED layout or a pattern of light from an LED layout. In this example, the LEDs are essentially installed with reference to, for example, a North-East-South-West (NESW) coordinate system 1592, where when the totem is held in a user's hand, North may point away from the user, South may point towards the user, East may point towards the user's right, and West may point towards the user's left. The coordinate system 1592 may be rotated to some extent. For example, the layout in column 1550 is essentially installed with reference to a neutral position, where the coordinate system 1592 is not rotated. In contrast, the layouts in columns 1570 and 1580 are rotated 15 degrees clockwise from the base position shown in the NESW coordinate system 1592.

[0199] The LED layout may be user adjustable. For example, the touchpad (or light guide) on the totem may be mechanically rotatable. As an example, the LED layout of the totem may initially include a 15 degree clockwise rotation for a right-handed user. However, when a left-handed user uses the same totem, the left-handed user may rotate the touchpad 30 degrees counterclockwise to allow them to better interact with the touchpad using their thumb. An example layout in columns 1560 and 1580 may consist of 12 LEDs. These LEDs may be placed adjacent to each other. For example, the 12 LEDs may be placed next to each other with no spacing between them to form a circular layout, as illustrated in pattern I.

[0200] In some embodiments, two adjacent LEDs may be placed with a space between them (as illustrated by the patterns in columns 1550 and 1570). The space may be approximately the size of one LED. For example, pattern D shown in column 1550 and row 1540 may include six LEDs, which are placed approximately the same distance apart from each other. However, in some circumstances, to produce the patterns in columns 1550 and 1570, the totem may selectively turn off or not include one or more LEDs. For example, pattern D may include 12 LEDs (rather than 6 LEDs), and 6 of the 12 LEDs may not be illuminated in pattern D. In contrast, when a user changes a user interface action, more or less than the 6 LEDs as shown in pattern D may be illuminated. Advantageously, in some embodiments, the totem may conserve battery consumption because fewer LEDs are used (with spacing between them) or LEDs may be selectively turned off to achieve a desired lighting pattern (such as that shown in FIG. 15). Additionally, by spacing the illuminated LEDs apart from one another (either physically or by not illuminating some of the intermediate LEDs), a user may not need to move their thumb to a precise location to perform certain user interface actions, which can reduce user fatigue in some situations.

[0201] The LEDs may be divided into multiple groups (see also the example multiplexed groups described with reference to FIG. 14B). Rows 1510, 1520, 1530, and 1540 of FIG. 15 illustrate an example method of grouping the LEDs. For example, layouts A and G in row 1510 include two groups of LEDs, each group represented by a different color. Layouts B and H in row 1520 include three groups, layouts C, D, and F in row 1530 include four groups, and layouts D and I include six groups. The LEDs in each group may have similar characteristics, such as, for example, similar light patterns, similar responses to user interaction (e.g., all illuminate at the same time), etc. Additionally or alternatively, each group of LEDs may be used to represent specific information, as described with reference to FIG. 14B. For example, the wearable device may determine that the user may enter a video recording mode when its outward-facing imaging system captures that one or more LEDs associated with the top arc in pattern B are illuminated.

[0202] Advantageously, the LED layout may be aligned with the gripping orientation of the totem. Layout AI (highlighted in black) is such an example. In these layouts, when a user grips the totem, the user can reach the groups of LEDs without necessarily making significant adjustments to their hands.

[0203] In some implementations, only a portion of the LEDs are user-interactable. LEDs may be interactable if the user can activate them (or the light guides associated with them). For example, pattern E includes four LED groups, namely, an upper arc, a lower arc, a left arc, and a right arc, with each group having three LEDs. The user may be able to activate the upper and lower arcs (e.g., by pressing the light guides above these arcs), but not the left and right arcs. However, the left and right arcs can still provide visual or tactile feedback, either alone or in combination with the upper and lower arcs. Similar considerations applied to other layouts are shown in FIG. 15.

[0204] In addition to or as an alternative to the physical location of the LEDs, the patterns in FIG. 15 can be light patterns illuminated by the LEDs. For example, a totem can present pattern E by illuminating a ring of 12 LEDs, while pattern D can be presented by illuminating every other LED in the 12 LED ring. As another example, rather than a user rotating a touchpad or light guide, a totem can be programmed to adjust the location of the halo based on whether the user is left-handed or right-handed. The totem may display an arc at the 1 o'clock position for a left-handed user, but the same arc at the 11 o'clock position for a right-handed user. The arc may be associated with a user interface action, such as, for example, moving up a web page.

[0205] While many embodiments of user input devices are described in terms of the totem 1200, this is for purposes of illustration and not a limitation on the types of user input devices that may be used with the wearable system 200. For example, in other embodiments, a display screen (or a variation thereof) may be used to display the lighting shapes and patterns to the user. For example, in some of these embodiments, a smartphone may be utilized as the totem and may display various shapes and patterns on its screen. In some examples, the totem 1200 may include a display screen, which may be touch sensitive (similar to the surface of a smartphone). In some such embodiments that include a display screen, the use of the illuminated light guide 1264 may be optional, as the display screen may be used to display the lighting patterns and transition sequences described herein.

[0206] Example of location and movement of light patterns 16A and 16B illustrate example locations or movement patterns of light emission from a halo of a totem (or other type of light emitting user input device). As explained above, the halo may constitute an illuminated portion of the totem that surrounds the touchpad area. As illustrated in the embodiment shown in FIGS. 16A and 16B, the light guide substantially diffuses the light emitted by the individual light sources such that the halo appears as a ring or an arcuate portion of a ring, rather than individual discrete light sources.

[0207] The location and movement of the light pattern that may be projected by the halo may include one or more characteristics, such as, for example, shape, color, brightness, position, size, movement, animation, other visual effects (e.g., blinking or flashing, fade in, fade out), etc. For example, FIG. 16A illustrates four example patterns. In pattern 1612, the halo is shown as a light blue color with a light pink color at the two o'clock position. The halo in pattern 1614 has four colored arcs, namely, yellow, green, blue, and red, which correspond to the top, bottom, left, and right positions of the touchpad (respectively). Pattern 1616 shows a dark blue halo, and pattern 1618 shows a dark blue halo with a light blue arc at the top.

[0208] The light location or movement pattern of the halo can provide visual feedback to the user and persons in the user's environment. The light location or movement pattern may be determined based on context information, such as, for example, the user's environment, characteristics of the user, information associated with objects, processes, components of the wearable system, the user's interactions with the wearable system, etc.

[0209] As an example of generating a halo based on the user's environment, the environmental sensor 1710 (described below with reference to FIGS. 17A, 17B) in the totem 1200 can be a light sensor that can detect whether the user is in a light or dark environment. If the user is in a dark environment, the totem 1200 can display the halo in a lighter color (such as white) to help the user perceive the halo. On the other hand, if the user is in a light environment, the totem 1200 may display a darker color to help the user distinguish the halo from the ambient light in the user's environment. As another example, in a dark room, the intensity of the light in the halo may be reduced (e.g., "night mode") since it is easier to perceive a halo illuminated in a dark environment. Conversely, in a bright environment (e.g., outdoors in sunlight), the intensity of the light in the halo may be increased so that the halo pattern is visible in a bright environment.

[0210] The pattern in the halo can also be based on the user's characteristics, such as, for example, the user's physiological data, demographic information (e.g., age, location, occupation, preferences, etc.). For example, when a user is playing a racing game, the totem can display a halo that corresponds to the user's driving direction. The color of the halo may initially be red. However, if the wearable system (e.g., environmental sensors on the totem or HMD, etc.) detects that the user's heart rate or breathing rate exceeds a threshold condition, the wearable system may determine that the user is in an emotionally elevated state. Thus, to calm the user's emotional state, the wearable system may change the color of the halo from red to blue (e.g., pattern 1618, where a light blue arc illustrates the driving direction).

[0211] Information associated with the object may include, for example, notifications or alerts associated with the object, characteristics of the object (e.g., function, type, location, shape, orientation, etc.), etc. As an example, the object may be a virtual email application displayed to the user by the HMD. New emails received by the virtual email application may be indicated to the user by a halo having a rainbow light pattern on the totem. As another example, the halo may be an indication of whether the object is perceivable in the user's environment. For example, in a treasure hunt game, the halo may correspond to the location of the treasure. If the treasure is located in a front-right position of the user, the totem may display the halo at the 1 o'clock position of the light guide (see, for example, pattern 1612). As yet another example, if the object is not interactable by the user, the wearable system may be configured to not display the halo.

[0212] As described with reference to Figures 20A-21E, the halo can also be used to guide a process or indicate the status of a process. For example, the wearable system can apply computer vision algorithms to analyze the location of the halo to pair the totem with another device. The wearable system can also use computer vision algorithms to analyze the appearance of the light pattern and calibrate the totem in the user's environment. Once pairing or calibration status is complete, the totem may display pattern 1616 to indicate to the user that the process is completed.

[0213] The halo can also be used to indicate information associated with components of the wearable system. For example, the totem may be used to locate a battery pack used to provide power to the wearable device. In this example, the battery pack may include an electromagnetic emitter, while the totem may include an electromagnetic sensor. The electromagnetic sensor can detect the magnetic field generated by the electromagnetic emitter and calculate the relative position between the totem and the battery pack accordingly. The halo as displayed by the totem may correspond to the relative position between the totem and the battery pack (helping the user find the battery pack). For example, a halo with a large arc at the 3 o'clock position may indicate that the battery pack is near and to the right of the user. However, a halo with a small arc at the 3 o'clock position may indicate that the battery pack is farther away from the user, but still to the right of the user. As another example, when the wearable device is low on power, the totem may display a halo with a small arc to indicate that the battery is about to run out. Although described in terms of a battery pack, any component of the wearable system or other (e.g., the user's car keys) can contain an electromagnetic emitter that can be tracked by the totem, which illuminates a halo and assists the user in finding the component.

[0214] The halo may also provide an indication of the user's interaction with the wearable system. Advantageously, this indication may inform people in the user's environment of the user's current interaction. For example, when the user is recording a video, the halo may flicker red. Thus, a person next to the user would be able to see the flickering red light pattern and know that they should not interrupt the user's video recording experience. As another example, when the user completes a level in a game, the halo may light up in the same color as the game's level, indicating to the user's friends that the user has completed the level.

[0215] In addition to or as an alternative to visual feedback, the halo can guide user interaction. For example, the halo may show four arcs (up, down, left, and right) in different colors (as illustrated by effect 1614) to indicate that the user can use the totem's touchpad as a D-pad. As another example, while the user is watching a television program using the display 220, the totem may present a halo pattern 1662 in FIG. 16B. The pattern 1662 includes a red arc and a blue arc. The arc lengths for the red and blue arcs may be approximately 0.25π. However, when the user activates the totem (such as by tapping the totem's touch surface), the totem may present a fade-in effect of the arcs (as illustrated by process 1 in FIG. 16B). As a result of the fade-in effect, the length of the red and blue arcs may be increased from approximately 0.25π to 0.75π, as shown in pattern 1664. The brightness of the red and blue arcs may also increase due to the fade-in effect. This increased brightness and arc length may provide the user with a clearer indication that the user can swipe left or right on the touch surface. However, when the totem detects that the user has not interacted with the totem for a threshold duration, the totem may present a fade-out effect, as illustrated by process 2 in FIG. 16B. Thus, the halo is changed from pattern 1664 to 1662. As a result, the size of the arc and the brightness of the arc will decrease. Advantageously, by reducing the size of the arc and the brightness of the arc, the totem may reduce battery consumption when the user is not interacting with the totem.

[0216] In some embodiments, the totem 1200 can present a halo, provide information associated with contextual information, or guide user interaction, in conjunction with haptic, auditory, visual, or other effects.

[0217] Other Components of the Totem 17A and 17B illustrate example components of a totem 1200. The example totem 1200 can include a touch pad 1260 (which can include a touch surface 1262 and a light guide 1264), a trigger 1212, and a totem body 1214, as described with reference to FIGS. 12A and 12B. The light guide 1264 can comprise a user interactable area (e.g., touch sensitive) and can at least partially or completely surround the touch surface 1262. The totem 1200 can also include various components, at least some of which can be disposed inside the totem's body 1214. These components are described further below and can include an environmental sensor 1710, a battery 1720, a near field communication (NFC) interface 1732, a network interface 1734, a haptic actuator 1760, and a processor 1770. The connection interface 1222 is located on the bottom of the body 1214, for example, to allow the totem 1200 to be removably attached to a base. The connection interface 1222 can be used to provide power and charge the battery 1720, and to provide a communications link between the components of the totem and external devices (e.g., computing devices).

[0218] Bumper The totem 1200 may include a button 1266 (referred to as a bumper), which in the embodiment shown in FIG. 17B is located on the front edge of the totem, above the trigger 1212 and below the touchpad 1260. The bumper 1266 may provide an ergonomically comfortable place for a user to rest their index finger. The bumper 1266 may comprise a touch-sensitive surface, implemented, for example, as a depressible button, a capacitive touch sensor, a haptic element, or the like.

[0219] In the example shown in FIG. 17B, a user can primarily use three fingers to activate the totem 1200, e.g., activating the home button 1256 or touchpad 1260 with the thumb, the bumper 1266 with the index finger, and the trigger 1212 with the middle finger. Such a three-finger actuable totem can enable a user to provide user input quickly and efficiently without excessive and fatiguing use of only one finger (as may occur with a mouse setup for a desktop computer). The different buttons 1212, 1256, 1266 can provide different functionality depending, for example, on whether the user is working within an application, scrolling through an application launcher, selecting an object in the environment, etc. In some cases, a user may simply use the index finger to switch back and forth between depressing the bumper or pulling the trigger.

[0220] As examples of bumper functionality, tapping the bumper while the user is viewing an application can bring up an options menu for that application, while a long press of the bumper can activate a manipulation sequence for a virtual object in the environment. For example, a long hold of the bumper can grasp the object, and a long press of the bumper while pointing the totem 1200 towards the object can activate direct manipulation of the object (e.g., moving or reorienting the object). A tap of the bumper (or a pull of the trigger) in a manipulation sequence can end the sequence.

[0221] Environmental Sensors The environmental sensors 1710 can be configured to detect objects, stimuli, people, animals, places, or other aspects of the user's surrounding environment. The environmental sensors may include image capture devices (e.g., cameras), microphones, IMUs, accelerometers, compasses, global positioning system (GPS) units, wireless devices, gyroscopes, altimeters, barometers, chemical sensors, humidity sensors, temperature sensors, external microphones, light sensors (e.g., light meters), timing devices (e.g., clocks or calendars), or any combination or subcombination thereof. In some embodiments, the environmental sensors may also include various physiological sensors. These sensors may measure or estimate physiological parameters of the user, such as heart rate, respiration rate, galvanic skin response, blood pressure, brainwave state, etc. The environmental sensors may further include emitting devices configured to receive signals, such as lasers, visible light, invisible wavelengths of light, or sound (e.g., audible, ultrasonic, or other frequencies). In some embodiments, one or more environmental sensors (e.g., cameras or light sensors) may be configured to measure the ambient light (e.g., luminance) of the environment (e.g., capture the lighting conditions of the environment). Physical contact sensors, such as touch sensors, strain gauges, curb detectors, or the like, may also be included as environmental sensors.

[0222] Information obtained by the environmental sensor 1710 may be used to determine the light location or movement pattern of the halo displayed on the totem. For example, the environmental sensor may use a GPS sensor or an electromagnetic sensor (to detect electromagnetic signals associated with the physical object) to determine the relative location between the user and a physical object in the user's environment. The totem may present a halo, the location of which may correspond to the location of the physical object. For example, if the object is directly in front of the user, the totem may present the halo in the 12 o'clock direction on the light guide 1264.

[0223] Additionally or alternatively, information obtained by the environmental sensors 1710 can be used for one or more user interface operations. For example, the totem 1200 can use an IMU to detect 6DOF movement of the totem. For example, as a user rotates the totem 1200 while playing a game, an avatar (or other virtual object) controlled by the user (and displayed to the user via a wearable device) can rotate accordingly based on the movement data obtained by the IMU. In addition to or as an alternative to moving or rotating the totem 1200, the user can provide input on the touchpad 1260. For example, movement of the user's thumb (e.g., along the long axis of the totem) toward or away from the user can cause a virtual object to move toward or away from the user. Moving the user's thumb back and forth laterally on the touchpad 1260 can scale the size of a virtual object (e.g., from a larger size to a smaller size or vice versa), or rotating the user's thumb around the touchpad can rotate the virtual object.

[0224] In this example, the environmental sensors are located on the totem 1200, although in some embodiments the environmental sensors may be located on other components of the wearable systems described herein. For example, the environmental sensors (such as a camera or physiological sensor) may be part of the display 220 of the wearable system 200 (shown in FIG. 2).

[0225] Battery The battery 1720 stores power for the totem 1200. The totem can use the processor 1770 to determine the current status of the battery 1720. For example, the processor 1770 can measure and calculate the amount of power remaining in the battery 1720, whether the battery 1720 is currently being used, and the remaining life of the battery 1720 (such as when the battery 1720 will need to be replaced). As further described in FIGS. 21A-21E, the current status of the battery 1720 may be visualized via visual feedback presented on the light guide 1264, the home button 1256, an LED power indicator located on the bottom portion 1244 of the totem body 1214 (e.g., shown in FIG. 12A), or a display.

[0226] The totem 1200 can also use the processor 1770 to determine the power consumption level of the totem. For example, the totem can estimate power consumption based on an estimated current required to output a certain light signature. The light signature may include a haloed light location or movement pattern associated with an indication of a process, status, object, or user interaction. As an example of estimated current associated with a light signature, the totem 1200 may require 5.6 mA to perform a 3 or 6 DOF calibration, 48 mA to find or indicate a direction to an object in the user's environment, and 54 mA to perform wireless (e.g., Bluetooth) pairing. As another example of the current estimated for the light signature, 14.85 mA of current may be consumed to provide an indication that the battery is below 15%, 21.6 mA may be used to indicate that the totem is in sleep mode or idle, and 72.4 mA of current may be supplied to provide incoming notification (e.g., new messages, etc.).

[0227] These estimated currents are examples only. The specific amount of power required may depend on various factors such as the type of LEDs used, the color of the LEDs, the location and movement of the halo light pattern, etc. For example, the totem battery 1720 may be depleted as much as 2 hours faster if all RGB LEDs of the touchpad 1260 are set to display white at 100% brightness all the time instead of displaying green at 100% brightness. This is because when the RGB LEDs display green, only 1 / 3 of the LEDs (e.g., the green LEDs in the RGB LEDs) are utilized, but when the RGB LEDs display white, all of the LEDs in the RGB LEDs are utilized.

[0228] To reduce overall power consumption, the LEDs (particularly the interactable LEDs) may be spaced to reduce the total number of LEDs on the touchpad. For example, the totem may adopt the layout shown in columns 1550 and 1570 in FIG. 15. Advantageously, in some embodiments, by spacing out the LEDs, power consumption may be reduced by up to 50%. The specific amount of reduction in power consumption may depend on the layout of the LEDs. For example, pattern D in FIG. 15 may have lower power consumption than pattern A because pattern D has fewer LEDs illuminated. As another example, to reduce power consumption, the totem may activate the EDs at a lower brightness (e.g., <40% brightness, etc.).

[0229] The totem can manage power consumption based on the object the user interacts with. The totem can turn off the LEDs that do not have active user interface options. For example, the totem may simulate a D-pad using LED layout E shown in FIG. 15. However, during gaming, the down button may be disabled at some level. Thus, the totem may be configured not to display the down arc. Additionally or alternatively, the totem can manage power consumption based on the status of the totem, the display, or other components of the wearable system. For example, the totem can turn off the halo after detecting a certain amount of inactivity. The totem can make such detection by calculating the amount of totem movement data obtained by the IMU.

[0230] In some embodiments, the wearable system may include a battery pack, which may be attached to the user (e.g., on the user's waist, etc.). The battery pack may be connected to and provide power to the totem or the wearable device. The battery 1720 may be part of the battery pack or may be used in connection with the battery pack to provide electricity to the totem 1200.

[0231] Near Field Communication (NFC) Interfaces and Network Interfaces The NFC interface 1732 and network interface 1734 can be configured to enable the totem to pair or communicate with a target object, such as, for example, another totem (or other user input device), a wearable display, a personal computer, headphones, a key, a server, or another physical object. The NFC interface 1732 can be used for short-range wireless communication (such as when the totem is within 10 cm or less of the target object). In some embodiments, NFC employs electromagnetic induction between two loop antennas when NFC-enabled devices, such as a totem and an HMD, are positioned in close proximity to each other (within about 10 cm) and wirelessly exchange information. This exchange of information may be operated within the globally available unlicensed radio frequency ISM band of 13.56 MHz at rates ranging from 106 to 424 Kbits / sec via the ISO / IEC 18000-3 air interface. A variety of communication protocols and data exchange formats may be used for NFC. Some non-limiting examples include ISO / IEC 14443 and Felicia, ISO / ICE 18092, which are standards defined by the NFC Forum and the GSMA Group, among others.

[0232] The network interface 1734 can be configured to establish a connection and communicate with the target object over a network, which may be a LAN, a WAN, a peer-to-peer network, radio frequency, Bluetooth, Wi-Fi, a cloud-based network, or any other type of communication network.

[0233] For example, the totem and the wearable device may be paired using a wireless communication protocol such as, for example, Bluetooth® Low Energy (BLE). BLE may be advantageous in some embodiments because it can preserve bandwidth during BLE audio streaming. As will be further described with reference to FIGS. 20A-20C, various techniques may be used to pair the wearable device with the totem. For example, the wearable system may detect certain light patterns presented by the totem by analyzing images of the totem acquired by an outward-facing camera. Detection of the light pattern may trigger the wearable system to begin the pairing process. During the pairing process, a user may interact with the wearable display via dialog boxes or user interface (UI) wizards to define settings and parameters associated with the totem, the wearable display, the type of information to be shared between the totem and the wearable display, the type of communication channel, etc.

[0234] Although in this example, NFC interface 1732 and network interface 1734 are illustrated as separate components, in some embodiments, NFC interface 1732 and network interface 1734 may be part of the same communications device or system.

[0235] Haptic Actuator The totem 1200 can include a haptic actuator 1760. As described with reference to Figure 12A, the haptic actuator 1760 can provide haptic feedback to the user. One or more haptic actuators 1760 may be implemented within the totem 1200. The haptic actuator may be located in the trigger 1212, the touchpad 1260, or the totem body 1214.

[0236] Haptic feedback may be provided based on user interaction, contextual information associated with the object or user, the status of a totem or other component of the wearable device, and the like.

[0237] Processor The hardware processor 1770 can be configured to process data obtained by the environmental sensor 1710. The processor 1770 can also receive and transmit data from another device (such as a wearable display or another paired device) via the NFC interface 1732 or the network interface 1734. The processor 1770 can analyze these data to determine the location or movement pattern of the halo at a given time. In some embodiments, the processor 1770 can cooperate with and analyze another computing device (such as a wearable device, a remote server, or a personal computer). For example, the processor 1770 can detect the movement of a user using the environmental sensor 1710. The processor 1770 can pass the user's movement data to the local processing and data module 260 or the remote processing module 270 for further analysis. The processor 1770 can receive the results of the analysis from the local processing and data module 260 or the remote processing module 270. For example, the processor 1770 can receive information regarding the position and movement trajectory of a competitor in a game. Based on this information, the processor 1770 can command the LEDs to emit light to display a halo corresponding to the competitor's position and movement trajectory.

[0238] The hardware processor 1770 can process various user inputs from a user's actuation of the totem 1200. For example, the hardware processor 1770 can process user inputs on the touchpad 1260, trigger 1212, home button 1256, bumper 1266, etc., as described with reference to FIGS. 12A-17B. As an example, the hardware processor can detect a user's hand gestures on the touchpad 1260 by processing signals from a touch sensor. As described herein, the hardware processor 1770 can process the user's inputs and instruct other components of the totem 1200 (e.g., LEDs or actuators 1760, etc.) to provide an indication of the user's actuation of the totem 1200.

[0239] Connection Interface 12A, the connection interface 1222 may be used to establish a connection with another device. The connection may be a wired connection. For example, the connection interface 1222 may comprise a USB connector. The connection interface 1222 may also comprise a USB port, such as, for example, a USB-B type port, a USB-A type port, a micro USB, or a USB type C port. The connection interface 1222 may also include a power cord for connecting to a power source for charging.

[0240] The example components illustrated in Figures 17A and 17B are non-limiting examples and the totem 1200 may include fewer, more, or different components than those illustrated. For example, the totem 1200 may not have an NFC interface 1732. As another example, the totem 1200 may include an optical emitter configured to emit light (such as infrared light), an electromagnetic emitter configured to generate or sense a magnetic field (e.g., used for electromagnetic tracking), or the like. As yet another example, the NFC interface 1732 and the network interface 1734 may be part of the same communication system.

[0241] Example of configuring the location and movement of the halo light pattern The illuminated halo described herein can be customized by a user, developer, or another entity of the wearable system. For example, a user of the wearable system can set preferences associated with the location and movement of the light pattern of the halo. The preferences may be associated with context information as described with reference to FIGS. 16A and 16B. As an example, a user can set the color of the halo associated with system notifications (such as low battery status) to blue, while setting the color associated with game objects to red. The user can set its preferences via the wearable device, alone or in combination with a totem. Based on the user's preferences, the totem can automatically present the illuminated halo when the user interacts with the wearable system. In some circumstances, the user can turn off the illuminated halo by activating the totem (such as the trigger 1212 shown in FIG. 12A) or via a posture. For example, the outward-facing imaging system 464 can capture a user's hand gesture associated with turning off the halo. The wearable system can use computer vision algorithms to recognize hand gestures from images acquired by the outward-facing imaging system and, appropriately, command the totem to turn off the illuminated halo.

[0242] The location and movement of the halo light pattern can also be configured using an application programming interface (API). FIGS. 18A-18D illustrate an example programming interface for configuring the halo light location or movement pattern. The API can be interacted with by a user or developer to set the totem halo light pattern. FIG. 18A shows the API, including a totem visualization 1802. The totem visualization 1802 includes an image of a halo 1810 displayed on a visualization touchpad 1804. In this example, the length of the halo 1810 is set to 0.25, which can be approximately one-quarter of the top circumference of the visualization touchpad 1804. The end points of the halo 1810 are at the 12 o'clock and 3 o'clock positions.

[0243] The programming interface 1800 can include an application bar 1830. The application bar 1830 can include the type of application for which the halo light pattern is configured. For example, in FIG. 18A, a user of the programming interface 1800 is configuring a light pattern associated with a gaming application. The user can also configure light patterns associated with web applications, system applications, scenes (e.g., the location of objects in an environment, etc.), user characteristics (e.g., associating a halo pattern based on whether the user is right-handed or left-handed, etc.).

[0244] The programming interface 1800 may also include an arc configuration tab 1820. The arc configuration tab 1820 may include various options for configuring the shape (e.g., arc, ring, etc.), size (e.g., angular arc length), color, visual effect, or orientation (e.g., N, S, E, or W) of the halo. The visual effects may include fading in, fading out, blinking, flashing, clockwise or counterclockwise rotation, etc. For example, the programming interface 1800 may associate a fade-in or fade-out effect with the halo 1810 (or a portion of the halo). The programming interface 1800 may also vary the arc length of the halo, for example, from ¼ of a circumference to ¾ of a circumference. Also, the rotation of the halo 1800 (e.g., from clockwise to counterclockwise) may be changed, or the location of the halo 1800 may be changed (e.g., from the top right corner to the bottom right corner of the touch surface).

[0245] As an example of configuring the light location or movement pattern of the halo using the arc configuration tab 1820, FIG. 18B shows an example halo 1812 when the arc length is set to -0.25. In this example, the arc length of the halo 1812 is approximately 3 / 4 of the circumference of the touch surface. The endpoints of the halo 1812 are the 9 o'clock and 6 o'clock positions.

[0246] The programming interface 1800 may also include a color selection tool 1830, shown in FIG. 18C. The color selection tool 1830 may include a search bar 1832, where a user of the programming interface may enter a color for the halo. The color for the halo may be entered in the format of a HyperText Markup Language (HTML) color code. The color selection tool 1830 may also include a color adjustment panel 1834, where a user may adjust the saturation, brightness, and hue of the color. The color selection tool 1830 may also include an RGB tool 1836, which may select a color based on the relative ratio between RGB and black / white. The user may use a pointing device (e.g., a mouse) to select a color, saturation, brightness, hue, etc., from the panel of the color selection tool.

[0247] In the example shown in FIG. 18C , the user sets the color of the halo 1814 to blue. The user can also use the color selection tool 1830 to change the color of the halo 1814 or use the color adjustment panel 1834 to adjust the saturation, brightness, or hue of the selected color. In some embodiments, the color of the halo may be pre-set, for example, based on a theme or style sheet. The wearable system may pre-set the color based on a system default color or based on context information. A user of the programming interface 1800 can change or adjust the pre-set color using the color selection tool 1830.

[0248] FIG. 18D illustrates another portion of the programming interface 1800 , which includes a visualization tab 1842 , a source code tab 1844 , and a pattern adjustment tool 1850 .

[0249] The pattern adjustment tool 1850 can be used to configure the halo. For example, a user of the programming tool can adjust sliders associated with the arc length, rotation, and color to provide various visual effects for the halo. A user can also adjust the position and rotation of the halo by setting x, y, z coordinate values ​​for the halo (as shown in the transform tab 1852). A user can also adjust lighting and movement effects associated with the totem, such as fade in / fade out effects.

[0250] The visualization tab 1842 may be used to visualize one or more effects of the halo created by the pattern adjustment tool 1850, the arc configuration tab 1820, or the color selection tool 1830. As an example, the visualization tab 1842 may be associated with a totem visualization 1802. When a user of the programming tool 1800 selects the visualization tab 1842, the programming tool 1800 may show the totem visualization 1802 so that the user may directly perceive the effect of updates to the halo.

[0251] The source code tab 1844 contains source code associated with the location and movement of the totem's light pattern. For example, when a user of the programming tool 1800 configures the halo using the arc configuration tab 1820, color selection tool 1830, or pattern adjustment tool 1850, the source code may be automatically updated based on the user's selection. The source code (or executable file version) can be communicated to the totem using a wired connection (e.g., via the connection interface 1222) or a wireless connection (e.g., via the network interface 1734). Additionally or alternatively, the source code (or executable file) can be stored within the totem or a head-mounted device (e.g., local processing and data module 260, etc.), which can be used to generate analog signals and control the light pattern of the LEDs 1330.

[0252] In some embodiments, the halo may be composed of multiple arcs (see example halo pattern 1614 in FIG. 16A). The programming tool 1800 can allow a user to configure each arc using techniques described herein. The pattern adjustment tool 1850 shown in FIG. 18D can be used to configure the halo, alone or in combination with the arc configuration tab 1820 and color selection tool 1830.

[0253] Although the examples in FIGS. 18A-18D are described with reference to configuring the light location or movement patterns of a halo, similar techniques can also be used to configure tactile, auditory, visual, or other effects.

[0254] Example of totem calibration using halo User input devices (such as the totems described herein) may require calibration to minimize errors and improve the user interaction experience. For example, the IMU of the totem may need to be calibrated to accurately detect and respond to the user's movements of the totem. Calibration is typically performed at the time of manufacture (e.g., when the totem or IMU is manufactured). Post-manufacturing calibration may be difficult because the user may not have the control and precise calibration system that the totem manufacturer has. However, it may be beneficial for the user to perform post-manufacturing calibration because user degrees of movement may vary. For example, one user may move their arm to a greater extent than another user when the user bobs the totem. In addition, it may be advantageous to provide a customized user experience so that the totem may have variable degrees of response tailored to different users. Some users may prefer a more sensitive totem (that may respond to even slight movement), while other users may prefer a less sensitive totem (that may be configured to not perform a user interface action when slight movement is present).

[0255] The wearable system and totem described herein can advantageously support post-manufacturing calibration without requiring the user to use the manufacturer's calibration system. The totem can display a halo. As the user moves the totem 1200, the outwardly facing imaging system 464 of the wearable system can capture an image of the totem and the illuminated halo. The shape of the halo as captured by the outwardly facing imaging system 464 can vary based on the position and orientation of the totem. The wearable system can use the object organizer 708 to identify the shape of the halo and determine the position and orientation of the totem accordingly. To calibrate the totem's IMU, for example, the position and orientation of the totem as determined based on images obtained by the outwardly facing imaging system 464 can be compared to the position and orientation captured by the IMU. The wearable system can correct the totem's IMU based on the comparison.

[0256] FIG. 19A illustrates an example of totem calibration using an illuminated halo. The example in FIG. 19A is described with reference to calibrating a totem 1900, which is for illustration and not intended to be limiting. The totem 1900 may have three degrees of freedom in this example. The 3DOF totem 1900 may include a circular touch surface 1962 and a circular light guide 1964 surrounding the touch surface 1962. The light guide 1964 may display a circular halo. The 3DOF totem 1900 may be an example of the totem 1200 described herein. Thus, the touch surface 1962 may be an example of the touch surface 1262, and the light guide 1264 may be an example of the light guide 1264.

[0257] In this exemplary FIG. 19A, various poses of the totem are shown (e.g., poses 1910, 1920, 1930, and 1940). The poses of the totem can include the orientation and position of the totem. For example, pose 1920 indicates that the user is pointing the totem downwards. pose 1930 indicates that the totem is pointing to the right, and pose 1940 indicates that the totem is facing an upper right position. pose 1910 may be the origin position, which is associated with the natural resting position of the totem.

[0258] During a calibration process for the totem's IMU, the wearable system can capture an image of the user's environment using its outward facing imaging system 464. The image of the user's environment can include an image of the totem. The object recognizer 708 of the wearable system can apply computer vision algorithms to identify the totem in the captured image. The object recognizer 708 can also apply computer vision algorithms to identify the shape of the halo in the image. The wearable system can determine the position and orientation of the totem based on the shape of the halo, alone or in combination with an image of the totem body. For example, the wearable system can identify a shape associated with the origin position and calculate a change to the shape associated with the origin position. Based on the change to the shape associated with the origin position, the wearable system can determine a change to the position and orientation of the totem based on the origin position. For example, a halo having a circular shape can be changed to an elliptical shape if the normal to the circular shape is pointing in a direction that is not directly toward the imaging camera. The wearable system can thus determine the current position and orientation of the totem based on the changes to the base position.

[0259] As an example, the illuminated halo indicated by the light guide 1964 may have a circular shape when the totem 1900 is in the home position (as indicated by pose 1910). Additionally, the outward-facing imaging system 464 may not perceive the body of the totem 1900 when the totem is in the home position. With reference to pose 1940, the wearable system may identify that a portion of the totem's body is in a diagonal position based on the image obtained by the outward-facing imaging system 464. The wearable system may also identify that the illuminated halo in the totem's pose 1940 appears to have an elliptical shape, rather than a circular shape. Based on observation of the totem body and halo (among other possible information), the wearable system may determine that the totem is tilted to the right and facing up. As another example, the shape of the halo in poses 1920 and 1930 may both be elliptical. However, the long axis of the halo in pose 1930 is vertical (and the short axis in pose 1930 is horizontal), while the long axis of the halo in pose 1920 is horizontal (and the long axis in pose 1920 is vertical). Thus, the wearable system may determine that in pose 1930 the totem is positioned horizontally (e.g., facing left or right) and that in pose 1920 the totem is positioned vertically (e.g., facing up or down).

[0260] The origin position may be defined by the wearable system. In this example, the origin position is defined to be associated with the pose 1910. In other examples, the origin position may be associated with other poses of the totem (poses 1920, 1930, 1940, etc.). In this example, the halo appears to be circular at the origin position, but the totem can illuminate other light patterns for totem calibration, which the wearable system can detect and analyze. For example, the totem can illuminate a square shape, a triangle (e.g., with three arcs surrounding the touch surface of the totem), a pattern of overlapping line segments (e.g., having a cross-shaped arc or an "x" shaped arc), etc. In an embodiment, the light pattern may also include arcs with multiple colors (e.g., may be useful for calibrating totems with more degrees of freedom, such as a 6DOF totem). For example, if the light pattern includes a cross-shaped arc, the totem can illuminate the light pattern as illustrated in FIG. 25B. The outward facing imaging system 464 and object recognition device 702 can use similar techniques to determine the totem's position and orientation regardless of the halo light pattern.

[0261] The wearable system can also receive data related to the position and orientation of the totem from the IMU. For example, the wearable system can calculate the orientation of the totem based on data obtained from a gyroscope in the totem. The position and orientation data obtained by the IMU can be compared to that calculated based on an image of the totem. The wearable system can calibrate the IMU based on this comparison. For example, the wearable system can determine that the totem 1900 is in attitude 1940 based on data obtained by the IMU. However, the position of the totem 1900 as observed by the wearable system may actually be in attitude 1930. Thus, the wearable system can correlate the position and orientation observed by the IMU with attitude 1930 (as observed by the outward facing imaging system 464) and calibrate the IMU. In some embodiments, in addition to or as an alternative to calibrating the IMU, the wearable system may store a calibration transformation (e.g., a lookup table) to correct the IMU readings to represent calibrated (as opposed to raw) readings.

[0262] To facilitate the calibration process, the halo may be used to guide the user to move the totem to a certain pose (e.g., position or orientation). The wearable system can thus obtain the totem's position or orientation using an outward-facing imaging system and IMU. FIG. 19B illustrates an example of totem calibration, where light patterns are used as guides for user movement. For example, the totem may exhibit patterns 1972, 1974, 1976, 1978. These patterns may correspond to MUX2, 0, 1, 4 shown in FIG. 14B. For example, LED3, LED4, and LED5 in MUX1 may illuminate and exhibit pattern 1976.

[0263] During the calibration process, the totem presents pattern 1972, indicating to the user to move the totem to the right, and pattern 1974, indicating to the user to move the totem to the left. When the totem presents pattern 1976, the user may move the totem upwards, while when the totem presents pattern 1978, the user may move the totem downwards. Once the totem (or IMU) is calibrated, the totem may present pattern 1980, which shows a solid blue color and forms a circular shape around the perimeter of the totem's touch surface. In addition to or as an alternative to pattern 1980, the totem may also employ other indications to indicate that the totem (or IMU) has been calibrated. For example, the totem may provide a vibration (e.g., via actuator 1760) or a sound (e.g., via speaker 240). As another example, the totem may provide other visual indications, such as a fade in or fade out effect, a portion of a circular halo, or other visual scheme. In some embodiments, the totem (e.g., the processor 1770) may command the LEDs to stop illuminating once calibration is complete.

[0264] Although FIG. 19A is described with reference to calibrating an IMU in a 3DOF totem 1900, similar techniques can also be used to calibrate a 6DOF totem or other types of user input devices.

[0265] Process 1982 in FIG. 19C illustrates another example of a calibration process. Patterns 1984a, 1984b, 1984c, and 1984d indicate the current position / movement to which the totem is currently calibrating. For example, pattern 1984a (showing an arc on the top of the circular touchpad) may indicate that the totem is currently calibrating itself in association with upward movement, while pattern 1984b (showing an arc on the bottom of the circular touchpad) may indicate that the totem is currently calibrating itself in association with downward movement. As another example, pattern 1984c (showing an arc on the right of the circular touchpad) may indicate that the totem is currently calibrated for right movement, and pattern 1984d (showing an arc on the left of the circular touchpad) may indicate that the totem is currently calibrated for left movement.

[0266] In some embodiments, patterns 1984a, 1984b, 1984c, and 1984d may be triggered due to user interaction. For example, if a user moves the totem right, the totem may display halo 1984d and calibrate the totem accordingly in association with this right movement. Additionally or alternatively, patterns 1984a, 1984b, 1984c, and 1984d may automatically illuminate to guide the user to move the totem into a position, as described with reference to FIG. 19B.

[0267] Once the totem is calibrated in all four directions (left, right, up, and down), the totem can present a confirmation pulse. The confirmation pulse may include alternating between displaying patterns 1986a and 1986b. For example, the totem can change its halo from pattern 1986a to pattern 1986b (and vice versa) every 0.1 seconds.

[0268] 19A and 19B are described with reference to calibrating an IMU, similar techniques can also be used to calibrate other environmental sensors 1710.

[0269] In addition to, or as an alternative to, guiding a user through a totem calibration process, similar techniques can be used to calibrate a user's interactions with tangible objects. For example, when a user begins playing a game, the light location or movement pattern of the halo may be used to guide the user through a tutorial process. As an example, when the totem shows pattern 1984c, the user can move the totem to the left and the display can therefore show a left movement of the avatar. The user can also associate the movement of the totem with a particular user interface action. For example, the user can perceive that the avatar is in a running action after the avatar is created. The user can move the totem forward (away from the user) and set the forward movement of the totem to be associated with the running action.

[0270] In some embodiments, the light location or movement pattern of the halo may be different based on the type of calibration, for example, the halo may be blue when the IMU is being calibrated, while it may be green when the user configures the totem movement for a game.

[0271] 19D illustrates an example process 1990 of totem calibration using a light pattern associated with a halo. The example process 1990 can be performed by a wearable system (such as the local processing and data module 260 or remote processing module 270) alone or in combination with a totem.

[0272] In block 1992a of the example process 1990 in Figure 19D, the wearable system can receive an image of the user's environment. The image may be obtained by the outward-facing imaging system 464 shown in Figure 4.

[0273] In block 1992b, the wearable system may analyze the image and identify a totem (or a halo) in the image. The totem may include a body and a light guide. The light guide may be configured to display a halo. The wearable system may use an object recognizer 708 described with reference to FIG. 7 to identify the totem, the body of the totem, the halo, or the light guide. The object recognizer 708 may use computer vision algorithms to perform such identification.

[0274] In block 1992c, the wearable system can determine a first position of the totem. The first position can be a base position. For example, the base position, such as the natural resting position of the totem, can be predefined by the wearable system. The base position can also be identified or changed by the user. For example, the user can define the base position to be a position associated with the posture 1910. The wearable system can determine a first light pattern of the halo associated with the base position. The pattern associated with the base position can be prestored by the wearable system. In some embodiments, once the wearable system identifies the base position, the wearable system can prompt the user with a notification requesting that the user hold the totem at the base position, and the wearable system can optionally capture an image of the totem at the base position.

[0275] At block 1992d, the wearable system may identify a second pattern of the halo based at least in part on analysis of the image. The wearable system may analyze the image and identify the pattern using the object recognizer 708.

[0276] In block 1992e, the wearable system may determine a first change to the totem's position and orientation relative to the first position. For example, the wearable system may analyze the change to the pattern of the halo based on the pattern associated with the base position when the totem is in the updated position and the pattern acquired by the outward-facing imaging system. In addition to the image analysis, the wearable system may also identify a second pattern of the halo or calculate the first change to the totem's position and orientation based on other information, such as uncalibrated, user interaction, or data from one or more environmental sensors. For example, the user may provide an indication to indicate the change to the totem's position and orientation. The indication may indicate that the totem has been moved forward from its original position via posture or a user input device 466, such as pointing a finger forward.

[0277] In block 1994a, the wearable system may also receive movement data associated with the totem from the environmental sensors that need to be calibrated. The wearable system may, in block 1994b, optionally calculate a second change to the position and orientation of the totem based on the data received from the environmental sensors.

[0278] The wearable system may calculate the difference between the first change and the second change. In block 1992f, if the wearable system determines that the difference between the first change and the second change reaches a threshold condition, the wearable system may calibrate the environmental sensor in block 1992g. Otherwise, the wearable system may determine that the environmental sensor does not need to be calibrated and provide an indication, as shown in block 1992h. The indication may include, for example, a location and movement of a halo pattern or tactile feedback, such as a confirmation pulse shown in FIG. 19C. Additionally, the indication may be presented by the display 220, a speaker, or the like.

[0279] In some situations, the wearable system may collect data associated with multiple positions and orientations of the totem, and the wearable system may determine whether or how the environmental sensors will be calibrated based on the collected data. For example, the wearable system may determine that the IMU does not need to be calibrated when the totem moves left and right, but may need to be calibrated for forward / backward movement. However, to calibrate for forward / backward movement, the wearable system may also need to adjust the IMU to detect left / right movement. Thus, the wearable system may calculate the amount of adjustment needed for forward / backward and left / right movement, even though the IMU is already capable of detecting left / right movement with reasonable accuracy.

[0280] 19E illustrates another example process 1996 of totem calibration using a light pattern associated with a halo. The example process 1996 can be performed by a totem or wearable system described herein.

[0281] In block 1998a, the totem can provide a first indication of the totem's posture. For example, the totem's light guide may display a halo with an arc located on the left side of the light guide. This arc can provide an indication to the user to move the totem to the left. In some embodiments, the indication can also be provided by a wearable display or via haptic or auditory feedback.

[0282] In block 1998b, the totem can obtain movement data associated with the pose. For example, as a user moves the totem to the left, the totem's IMU can detect the user's movement and communicate the movement data to the processor.

[0283] In block 1998c, the totem can be calibrated based on the movement data. For example, the totem can correlate left movement with movement detected by the IMU.

[0284] In block 1998d, it may be determined whether the totem has been calibrated. For example, if the totem's IMU needs to be calibrated, the totem may determine whether the totem has been calibrated for 6 DOF. Once the totem has been calibrated, in block 1998f, the totem may provide a second indication to signal that the totem has been calibrated. For example, the totem may provide a flashing green halo on the light guide to indicate that the totem has been calibrated.

[0285] If the calibration process is unsuccessful, the process 1996 may return to block 1998a to repeat the calibration.

[0286] Example of device pairing using halo Two devices may be required to establish a wireless communication link before being paired (e.g., via Bluetooth). When devices pair, they may share information such as, for example, the address, name, and profile of the device's user or the device itself, or share information stored within their respective memories. Once two devices are paired, they may share a common secret key, which may allow them to be relinked for future information sharing.

[0287] Device pairing can be tedious and sometimes require a user to key in unique information about the device. For example, a user may manually enter the device's information after the device (or the user's other device) completes an initial search for nearby devices. Devices can use the NFC protocol to share information between the two devices for the pairing process, but NFC components are often relatively large and costly.

[0288] To reduce or minimize user interaction and simplify the pairing process between the totem and the wearable device, the totem can present a light pattern, which can be analyzed and used for pairing. The light pattern can include a message used for pairing between the totem and the wearable device. The light pattern can be presented by a halo illuminated on the totem's LEDs. The message can include an indication to initiate the pairing process. For example, the light pattern can encode a trigger message, which can cause the wearable device to search for nearby devices or broadcast its device information to nearby devices. As another example, the light pattern can encode the totem's device information. The device information can be encrypted. The device information can include a device identifier (ID), operating system information (e.g., operating system version), firmware information, the type of communication channel used for pairing (e.g., Bluetooth, wireless channel), a digital key shared between the totem and the wearable display, etc. In some implementations, the device ID can be provided or generated by a manufacturer, a distributor, a developer, or any suitable entity. Examples of device identifiers may include an Android Identifier (ID), an iPhone® Unique Identifier (UDID), an iPhone® Advertising Identifier (IFA or IDFA), a cookie ID, a login ID, an Internet Protocol (IP) address, a Media Access Control (MAC) address, a hash of any of the above, a combination of any of the above, or the like. In some cases, a device identifier may be derived based on one or more hardware and / or software parameters of the device identified by the device identifier. For example, a device identifier may be derived from the device's IP address, operating system version, and locale settings.In some embodiments, a device identifier may be used to identify the source or origin of a transaction, request, or network event. For example, a device identifier may include a user identifier, an account identifier, and the like. Additionally or alternatively, a device ID may be associated with an entity (e.g., a user) that uses the device.

[0289] As an example, the totem may include a ring of LEDs on its perimeter (e.g., 12 LEDs arranged circumferentially around the touchpad). These 12 LEDs, when illuminated in a specific temporal sequence, spatial sequence, or combination thereof, can be used to represent certain device information. Each LED may be assigned a bit, and the ring of LEDs can represent a 12-bit word in this example. One or more of the LEDs can be displayed in a sequence to indicate information necessary for pairing. For example, LED lights can be illuminated in a sequence to indicate the device identifier (ID) of the totem. Assuming the device ID is 2 and the binary representation of the number 2 is 10, the totem may illuminate LED0 (shown in FIG. 14B) to represent the binary number 2, while the other LEDs (e.g., LEDs 1-11) are not illuminated.

[0290] In some implementations, the color of one or more LED lights can also represent information used for pairing. For example, a red color may represent one manufacturer or model of the totem, while a blue color of the LED light may represent another manufacturer or model. A totem can also use the LED lights to indicate a combination of the totem's manufacturer and model. For example, a totem can indicate manufacturer A and model M by illuminating LED1 blue while illuminating LED3 pink.

[0291] Furthermore, if each LED is assigned a bit, the color of the LED can expand the amount of information the LED can represent. For example, instead of one LED being assigned one bit, an LED with a certain color may be assigned one unit of information. Thus, if an LED light can illuminate 9 colors, the LED can be used to indicate the numbers 0-9 based on the illuminated color. For example, if an LED is not illuminated, the wearable device can interpret it as 0, while purple can represent the number 1, green can represent the number 2, etc. As another example, the LED can also represent letters in the alphabet (e.g., red is A, white is B, etc.).

[0292] The wearable device may be associated with a display and a camera (e.g., an outwardly facing imaging system 464, etc.). The camera can be used to capture the light sequence, then demodulate it and extract usable information for automatic pairing. For example, the image captured by the camera can be analyzed using various computer vision algorithms described herein to extract the halo light pattern. The light pattern may be converted into numbers, alphabets, words, etc., which can represent the totem's device information. The wearable device can then use this device information for pairing with the totem.

[0293] Two totems may also be paired. For example, two friends may wish to pair their totems and share their game control while playing a video game using their individual wearable devices. In this example, the outward-facing imaging system of the first wearable device may capture the light sequence on the second friend's totem (and vice versa). The first wearable device may extract the device information of the second friend's totem and pass that information to the first friend's totem for pairing. In some implementations, the totem may also have a camera. Thus, a totem can capture an image of another totem's light pattern and achieve pairing of the two totems.

[0294] As another example, a wearable device (or HMD) may be paired with two totems. An outward-facing imaging system can obtain an image of the light pattern for one totem. Once the wearable device has extracted the device information of one totem from the light thereafter, the wearable device can pass the device information (of the wearable device or another totem) to the other totem for pairing.

[0295] In addition to or as an alternative to device pairing, similar techniques for communicating information by a totem can also be used to communicate other types of information. For example, the totem may communicate a user's profile information or the totem's status to the wearable device using a light pattern illuminated by the totem.

[0296] Advantageously, in some embodiments, the light location or movement pattern of the totem can be used to indicate the progress of wireless pairing between a totem and a wearable device or between two totems. Figures 20A and 20B illustrate an example showing a wireless (e.g., Bluetooth or WiFi) pairing process using a halo. The light patterns illustrated in these two figures can be used to indicate the progress of the device pairing process via wireless communication (e.g., a totem or device wirelessly broadcasting its device information to other devices) or through the light patterns described with reference to Figure 14B.

[0297] 20A shows a totem 2020, which may have a touchpad 2022 and a display area 2026. The totem 2020 may be an example of the totem 1200, and the touchpad 2022 may be an embodiment of the touchpad 1260. The display area 2026 may include a ring of LEDs and a light guide that diffuses the light emitted by the LEDs. In some embodiments, the display area 2026 may be activated by a user.

[0298] The display area 2026 may display a halo 2024. A portion of the halo may be dark blue. As shown in process 2012, the totem may play a clockwise spinning animation of the dark blue portion. Once the totem is paired, the totem may present the dark blue halo in a pulsing animation. An example of the animation is described with reference to FIG. 20B. The totem may begin the spinning animation by displaying pattern 2062. While wireless pairing is in progress, the totem may sequentially present patterns 2064, 2066, and 2068, as illustrated by arrows 1, 2, and 3. The totem may play pattern 2062 again after pattern 2068 if pairing is still ongoing. If pairing is successfully completed, the totem may present pattern 2070, which may provide a visual indication to the user.

[0299] Although these examples are described with reference to pairing a totem, similar techniques can also be used to pair other user input devices 466 or devices with a display or lighting component. For example, the techniques can be applied to pairing two or more of a smartwatch, a smartphone, a wearable device, a totem, a tablet, a computer, a television, a camera, a peripheral device, a home appliance, or other devices with image recognition and processing capabilities. As an example of pairing between a smartphone and a totem, the smartphone can use its camera to capture and analyze lighting patterns on the totem. As another example, a wearable device may be paired with a smartwatch, and the wearable device can capture and analyze patterns displayed by the smartwatch (e.g., on a display screen of the smartwatch). The techniques described herein further apply to Bluetooth pairing, WiFi pairing (e.g., Wi-Fi Direct), and other methods of pairing. TM), or other types of wireless pairing. As an example of Bluetooth pairing, when the totem is in discoverable mode, the totem can display a light pattern to indicate that it is in discoverable mode. The wearable device can capture this light pattern and complete the pairing when the wearable device is also in discoverable mode. When the light pattern encodes a trigger message, the wearable device can automatically enter discoverable mode in response to detection of the light pattern. As an example of wireless pairing, the totem can illuminate a light pattern to indicate that a connection can be made to the totem. In response to detection of the light pattern, the wearable device can receive an invitation and connect to the totem. In addition to these examples of wireless pairing, the pairing techniques can also be applied to types of pairing other than wireless pairing. For example, the techniques can also be used to share files between a group of user input devices in a home via an Ethernet connection.

[0300] Example Process for Device Pairing Using Halo 20C illustrates an example process 2080 of device pairing using a halo. Pairing may occur between a wearable device and another device (e.g., a totem, a smartphone, a television, a smartwatch, etc.). The example process 2080 can be performed by a wearable device having an HMD and an outwardly facing imaging system 464. The process 2080 can be used to assist in pairing a totem with another device, an HMD, etc.

[0301] In block 2082, the wearable device may receive a request from the totem to establish pairing. For example, the totem may broadcast a signal to nearby devices to establish communication. The wearable device may receive the broadcast signal and identify the totem that communicated the signal. As another example, the request may be a trigger message encoded in the totem's light pattern.

[0302] At block 2084, the wearable device may receive an image of the halo presented by the totem. The image may be captured by an outward-facing imaging system 464. As described with reference to Figures 20A and 20B, the totem may use the halo to communicate device information needed to establish pairing via the light location or movement pattern of the halo. For example, the halo may encode the device information into a binary form.

[0303] In block 2086, the wearable device can analyze the image of the halo and extract device information. For example, the wearable device can use the object recognizer 708n to identify the halo in the captured image. If the halo encodes the device information in binary form, the wearable device can convert the binary representation of the device information to a decimal or alphabetical representation.

[0304] In block 2088, based on the extracted device information, the wearable device may pair the wearable device with the totem. For example, the wearable device may add the totem's device ID to a whitelist of devices with which the wearable device may share its information.

[0305] Optionally, in block 2089, the wearable device may receive an indication from the totem that the pairing was successful. The indication may be an electromagnetic signal. Additionally or alternatively, the totem may display a halo with a location and movement to indicate that pairing is complete. For example, the totem may display a pulsating animation of a circular blue halo. The outward facing imaging system 464 may capture this halo in an image. Once the wearable device recognizes the light location or movement pattern of the halo, the wearable device may determine that pairing is complete.

[0306] In some embodiments, the totem also provides a visual indication that pairing is in progress. For example, the totem may present a halo with a spinning arc during pairing. The spinning arc may be presented after the totem provides device information to the wearable device.

[0307] Although examples are described herein with reference to a wearable device, in some embodiments, similar techniques can also be applied to another computing device. For example, the exemplary process 2080 can be used to pair a totem with a mobile phone configured to provide AR / VR / MR features.

[0308] 20D illustrates another example process 2090 of device pairing using a halo. Pairing may occur between a totem and a computing device. The computing device may be a wearable device having a wearable display and an outwardly facing imaging system 464, or other computing device (e.g., a television, a smartwatch, a smartphone, a tablet, another totem, etc.). The example process 2090 may be implemented by the totem 1200.

[0309] At block 2092, the totem may begin a pairing process with the computing device. For example, the totem may search for nearby devices that may be paired with the totem. The totem may also broadcast a signal and identify devices available for pairing based on responses to the signal received from other devices. A user may actuate the totem or the computing device, causing the totem to begin the pairing process. For example, a user may press the home button 1256 for an extended period of time, causing the totem to enter the pairing process. In one implementation, the totem may present a light pattern that indicates that the totem is now in pairing mode. The light pattern may be captured by the computing device, causing the computing device to enter the pairing process with the totem.

[0310] At block 2094, the totem may illuminate a first light pattern that encodes information for the pairing process 2094. The encoded information may include device information of the totem. The encoded information may also include a key for the computing device to connect with the totem. For example, the computing device may extract a key based on the light pattern extracted from the key image and transmit the key back to the totem over a wireless channel to accomplish pairing. In some embodiments, block 2092 may be optional. The computing device may automatically initiate the pairing process in response to detection of the first light pattern at block 2094.

[0311] At block 2096, the totem may receive a response from the computing device. The response may be device information of the computing device. In some circumstances, the response may include a request for more information from the totem. The computing device may also provide other information in the response, such as network communication information (e.g., network port, encryption standard, or communication protocol) that the totem may use to communicate with the computing device. In some circumstances, the response may also include an indication that the computing device has added the totem to a list of approved devices.

[0312] Optionally, at block 2098, once the pairing process is completed, the totem may illuminate a second light pattern to indicate that pairing was successful. The second light pattern may be pattern 2070 shown in FIG. 20B. In some embodiments, the totem may stop illuminating to indicate that pairing is complete. Although the exemplary process 2090 is described above as being performed by a totem, it should be understood that the exemplary process 2090 may also be performed by another device (e.g., a smartphone, a smartwatch, a computer, etc.).

[0313] An embodiment using halo light patterns to provide cues regarding the status of an object The location and movement of the halo light pattern can be used as a cue to indicate the status of the object. For example, a light location or movement pattern can be used to indicate whether a user has turned the totem on or off, whether the totem is in sleep mode, or the totem's battery charge status. Non-limiting illustrative examples are described below.

[0314] 21A illustrates an example showing the status of a totem. As shown in process 2110, when the power button is turned from off to on, the totem may present a portion of the halo and gradually fill the remainder of the portion until the totem is fully on. The totem may also rotate the halo clockwise for one cycle during the power on process.

[0315] When the power button is toggled from on to off, the totem may present the full halo and rotate the halo counterclockwise for one cycle (as shown in process 2120). The totem may also gradually conceal more portions of the halo until the totem is completely powered off (when the totem will no longer present the halo).

[0316] The halo color palette can be selected to be visually appealing to a user. Different color palettes can be used for different system functions, such as a startup color palette when the system is powered on (e.g., see process 2110 in FIG. 21A) and a shutdown color palette when the system is powered off (e.g., process 2120 in FIG. 21A). For example, the startup color palette can represent colors typically seen at sunrise, and the shutdown color palette can represent colors typically seen at sunset. Other color palettes can also be used for other system functions (e.g., charging or battery status or sleep idle mode shown in FIG. 21C, 21D). The color palette can include color, grayscale, lighting level, etc. The lighting pattern can include lighting information that is communicated to a light source controller (which can be a controller disposed on the touch board 1394c) that is used to control the brightness of the totem light sources (e.g., LEDs 1383a, 1394a). The lighting information may include a color palette, the timing (or rate of change), duration, and brightness level at which each of the LEDs (including the individual colors producible by the LEDs) is illuminated, etc. Some LEDs may include additional or different control functionality, for example RGBA, which includes RGB plus alpha channels that indicate the opacity of each pixel and allow the totem to perform alpha compositing (allowing for the appearance of partial or full transparency). The lighting information may include such additional or different control functionality depending on the choice of light source used in the totem.

[0317] There can be a vast variety of color palettes that can be displayed by RGB LEDs (e.g., typically, each color can be set to 256 values ​​from 0 to 255). The color palette can be selected for one or more of the halos described herein based on any of a variety of different factors, such as user characteristics, contextual information, device usage or status, system functionality, etc.

[0318] The halo lighting pattern can include gradients in color, shade, hue, etc. For example, due to a limited number of light sources in the totem (e.g., 12 RGB LEDs), variations in brightness of colored light sources can be used to provide the desired gradient (color or intensity). The gradient can provide the visual illusion of an intermediate end or fade-out of the lighting pattern. The gradient can be provided by dynamically adjusting the brightness of the light sources (time-varying lighting pattern). As an example, for a totem with 12 LEDs positioned at the hours of a clock, a lighting pattern starting at 4 o'clock and extending 45 degrees toward 6 o'clock, with the gradient ending at 6 o'clock in the arc, may have LED brightness values ​​set as follows: LED1-3, brightness 0.0; LED4, brightness 1.0; LED5 brightness 1.0; LED6 brightness 0.5; LED7-12, brightness 0.0.

[0319] Other shutdown processes may involve user interaction with a totem to confirm that the user desires to perform a shutdown operation (e.g., to shut down, suspend, or resume the wearable device 200). For example, the display 220 of the wearable system 200 may display a shutdown user interface to the user once shutdown is selected by the user. In one example, the shutdown user interface displays a circular ring (reminiscent of the circular light guide 1264 on the totem 1200) and a textual indication that the user should move their thumb over the totem's touchpad 1260 to confirm the shutdown operation. For example, the user interface may display "Complete the circle, power off" and the circular ring may include animation indicating that the user should move their thumb (for example) clockwise around the touchpad 1260. If the user presses the home button 1256, the shutdown operation may be aborted and the shutdown user interface may fade out.

[0320] When the user desires to complete the shutdown operation, the user may follow the displayed direction and move their thumb in a circular clockwise path around the touchpad 1260. As the user does so, the light guide 1264 may light up and create an arc that follows the thumb's motion (and optionally the display 220 may display an animated graphic showing progress around the circular ring). When the user's finger completes a full circle (confirming the shutdown operation), a haptic "bump" may be implemented by the totem 1200 to provide the user with haptic feedback that the shutdown operation has been initiated (optionally the system 200 may output an audio sound). The display 220 of the wearable system 200 may display a shutdown screen to the user and initiate the shutdown operation. The light guide 1264 of the totem 1200 may display a shutdown illumination pattern (e.g., as shown in FIG. 21B).

[0321] 21B illustrates an example of the light location or movement pattern of the halo during a power on and off process. During the power on process, the halo pattern can move from pattern 2122 to pattern 2124 and then to pattern 2126, as indicated by arrows 1 and 2. During this process, the arc of the halo (which may be illuminated in a color, e.g., yellow) can become gradually longer in length and brighter in color.

[0322] On the other hand, during the power off process, the halo pattern may move from pattern 2126 to 2124 and then to 2122, as shown by arrows 3 and 4. In this process, the shape of the halo may gradually reduce from a full ring to a small arc.

[0323] Additionally or alternatively, the halo can be configured to indicate the charge status of the totem's battery 1720, where the amount of battery may correspond to the size of the portion of the halo that is displayed. FIG. 21C illustrates an example of the location and movement of a light pattern that indicates the battery charge status. Process 2132 illustrates an animated sequence of the halo as the battery 1720 is charging. For example, the halo may be divided into four quadrants. Each quadrant may be associated with one or more LED lights. The totem can gradually fill all four quadrants by illuminating the LEDs in each quadrant. In one implementation, rather than filling one quadrant at a time, the totem can gradually illuminate the LEDs and track the percentage of battery remaining. For example, when the battery is at 30%, the totem can illuminate 30% of the LEDs.

[0324] When charging begins, if the battery is below 15% (say), the totem may present a halo with a first color (say, orange) in one of the four quadrants. Once the battery reaches 25% (say), the totem may change the color of the halo from the first color to a second color (say, orange to green). Once the battery reaches 50% (say), the totem may illuminate the portion of the halo associated with the next quadrant in the clockwise direction (say, with the second color). The totem may continue fading in the other quadrants as the totem charges. Once the battery is fully charged, the totem may present the halo in the second color and all four quadrants may be illuminated (as shown in process 2134).

[0325] While the battery is being depleted, the shape of the halo may initially be a full circle. The totem may gradually fade out the halo in a counterclockwise direction as the amount of power decreases, as illustrated in process 2136. In some embodiments, when the battery is only at 15% (for example), the totem may change the remaining portion of the halo to a third color (e.g., red) and play a pulsing animation (the totem may play a fade out animation of the halo) until no power remains.

[0326] The halo can also be used to indicate if the totem has entered a sleep mode. Figure 21D illustrates an example halo pattern when the totem has entered a sleep mode. As shown in process 2142, when the totem becomes idle because no user interaction has been detected within a threshold time period, the totem can present the halo in the form of a trail (e.g., a white trail). The totem can also repeat the trail in a clockwise cycle.

[0327] Example process of using a halo to indicate totem status 21E illustrates an example process for indicating a totem's status based on light location or movement patterns. The process 2150 in FIG. 21E can be performed by one or more components of a wearable system described herein. For example, the process 2150 can be performed by a totem or a wearable device.

[0328] In block 2152, the wearable system can determine the status of the totem. The totem can include a group of light sources (e.g., LEDs) configured to display a halo. The halo may be used to indicate the status of the totem. The status may include, for example, whether the totem is powered on / off, whether the totem is in a sleep mode, or the totem's battery status, such as the progress of the battery charge or consumption.

[0329] In block 2154, the wearable system can access a first light pattern for the halo. The first light pattern may define a location or movement pattern for the halo. The first light pattern can be associated with a current status of the totem. For example, when the battery is currently low, the light location or movement pattern for the halo may be a red arc around the 6 o'clock position.

[0330] In block 2156, the wearable system can instruct the group of LEDs to display a halo according to the first light pattern. Continuing with the previous example, the wearable system can generate and send instructions to illuminate a red light to LEDs located near the 6 o'clock position (such as LED9, LED10, and LED11 in MUX3 shown in FIG. 14B).

[0331] The wearable system can continually monitor the status of the totem. Optionally, in block 2158, the wearable system can determine if there is an update to the status. For example, the totem's battery may have previously been below 15%, but the user plugged the totem into a power source to charge. Thus, the amount of battery may gradually increase.

[0332] In optional block 2160, in response to the update, the wearable system can instruct the group of LEDs to display a halo according to a light pattern. A second light pattern is associated with the update. For example, once the totem enters charging mode, the color of the halo may be changed from red to green.

[0333] Although examples herein are described with reference to indicating the status of a totem, similar techniques can also be used to indicate the status of other components of a totem, such as a wearable display or a battery pack.

[0334] Example of using light patterns as cues for user interaction In addition to, or as an alternative to, providing information about device status, the halo can also be used as a cue to indicate or guide a current user interaction. FIGS. 22A and 22B illustrate example light location or movement patterns used as cues for user interaction. FIG. 22A displays halo patterns 2210 and 2220. The halo patterns may be displayed by a light guide around the totem's touch surface. A user can activate the touch surface using, for example, their thumb 2230. The halo pattern 2210 can include an arc 2222 (which may be illuminated in a pinkish purple color) on the left side of the light guide. The length and brightness of the arc 2222 may reflect the distance between the user's finger and the portion of the light guide that displays the arc 2222.

[0335] As an example showing a current user interaction with the totem, when the user's thumb is at position 2212, a halo pattern 2210 may be shown. The portion of the arc 2222 closest to the thumb 2230 may be brightest. The brightness of other portions of the arc may gradually decrease as they move further away from the thumb 2230, eventually transitioning to black (e.g., the corresponding LED is turned off). In this example, the user may activate the touch surface using a swipe gesture. For example, the user may swipe their thumb to the left. As the user moves their thumb 2230 left toward the arc 2222 (illustrated by arrow 2214), the length of the arc 2222 may increase from less than half a circle to more than half a circle. The brightness of the arc 2222 may also increase due to the reduced distance between the thumb 2230 and the arc 2222.

[0336] As another example, in Fig. 22B, the halo associated with pattern 2250 may include three arcs 2252, 2254, 2256. Arc 2252 is illustrated in a first pattern corresponding to a first color (which may be, for example, red), arc 2254 is illustrated in a second pattern corresponding to a second color (which may be, for example, blue), and arc 2256 is illustrated in a third pattern corresponding to a third color (which may be, for example, yellow). With reference to Fig. 14B, arc 2252 may be illuminated by LED1, LED2, and LED3, arc 2254 may be illuminated by LED5, LED6, and LED7, and arc 2256 may be illuminated by LED9, LED10, and LED11.

[0337] Pattern 2250 may be associated with a three-way D-pad, with each arc associated with a user interface action. For example, a user may tap an area near arc 2252 to move a virtual object left, and tap an area near arc 2254 to move a virtual object right. A user may also tap an area near arc 2256 to select or release a virtual object. When a user's thumb 2230 is near an arc, the length of the arc may decrease while the brightness of the arc may increase. For example, when a user's thumb 2230 is near arc 2254, the totem may turn off LED5 and LED7 (shown in FIG. 14B ) and reduce the size of arc 2254. This change to the light location or movement pattern of the halo may provide an indication that the user has begun (or is about to begin) moving a virtual object left. As the user's thumb moves away from arc 2254, the totem may re-illuminate LED5 and LED7 and cause arc 2254 to increase in size.

[0338] The totem can simulate the functionality of various user input devices, such as, for example, a D-pad, a touch device, a 6DOF controller, etc. The light location or movement pattern of the halo can be used to indicate the type of user input device the totem is currently being used with, or to indicate available user interface interactions. For example, while a user is browsing a web page using a 3D display, the totem can be used as a touch device, which supports left / right swipe gestures. To provide an indication to the user that swipe gestures are available, the totem can display the pattern shown in FIG. 22A.

[0339] In some implementations, the wearable system can automatically select a preferred user input device configuration based on context information. For example, a user may want to use a 3-way D-pad to move virtual objects in and out of the FOV and to select virtual objects. Thus, when the wearable device detects that the user is interacting with a virtual object in its environment, the wearable device can instruct the totem to enable the 3-way D-pad and present the halo pattern 2250 to inform the user that the 3-way D-pad is available.

[0340] FIG. 22C illustrates another example of using a light pattern associated with a halo to provide an indication of available user interface actions. The totem in FIG. 22C can be configured to act as a star-shaped D-pad. To inform the user that the totem is currently being used as a star-shaped D-pad, the totem may illuminate LED2, LED4, LED6, LED8, LED10, and LED0 in FIG. 14B. As an example, when a user selects a text box on a web page, the totem may present pattern 2260 to indicate that the user can use the star-shaped D-pad for text entry. When a user activates an area near an LED, the LED may display a brighter color to indicate that the user is activating that area. For example, when a user activates an area near LED0 (as shown in pattern 2270), the totem may command LED0 to emit a brighter light.

[0341] In some embodiments, the totem may be used to inform the user that the user is performing an incorrect user interaction or provide an indication of a correct / recommended user interaction. FIG. 23 illustrates an example using a light pattern as an alert to indicate an incorrect or inappropriate user action. A user's thumb 2230 may initially be at position 2312 on the touch surface. The touch surface may be surrounded by a light guide, which may display a halo 2222. When the user's thumb is at position 2312, the light guide may display a halo pattern 2310. The user may need to move the object to the left by swiping left (e.g., moving their thumb 2230 closer to the halo 2222). However, the user actually swipes right (as indicated by arrow 2232), which is the incorrect direction to perform the desired action of moving the object to the left. As a result, the user's thumb is now at position 2314. To remind the user that they need to swipe left and move the object appropriately, the light guide may show a halo pattern 2320, where the halo 2222 appears longer in length and brighter in color, which may attract the user's attention and ensure that they move their thumb 2230 in the correct direction.

[0342] The pattern of the halo may vary depending on the type of user interaction. Figures 24A and 24B illustrate example patterns for swipe and touch gestures, respectively. The touchpad in Figures 24A and 24B may include a light guide, which displays the halo. The touchpad may also include a touch surface, which is surrounded by the light guide. The touch surface may receive user input, such as a swipe gesture or a touch gesture.

[0343] The halo shown in Figures 24A and 24B may have three arcs: 2422 (illuminated with a first color, e.g., red), 2424 (illuminated with a second color, e.g., blue), and 2426 (illuminated with a third color, e.g., yellow). With reference to Figure 14B, arc 2422 may correspond to LED1, LED2, and LED3, arc 2424 may correspond to LED5, LED6, and LED7, and arc 2426 may correspond to LED9, LED10, and LED11.

[0344] The user's thumb 2230 may initially be at position 2412 on the touch surface. The light guide may thus display pattern 2402 for the halo. The user may swipe downward toward arc 2426, as indicated by downward arrow 2450. For example, the user may move their thumb 2230 from position 2412 to position 2414 (as indicated by arrow 1). In response to detecting the user's movement, the totem may dim the display of arcs 2422 and 2424, as indicated in pattern 2404. The user may further move their thumb downward, as indicated by arrow 2. In pattern 2406, when the user's thumb reaches position 2416, the totem may increase the brightness and length of arc 2426. For example, the brightness of LED9, LED10, and LED11 may be increased. The totem may also illuminate LED0 and LED8 (in a third color) and increase the length of the arc 2426. The totem may further illuminate LED7 in a third color (rather than the second color as shown in arc 2424) and LED1 in a third color (rather than the first color as shown in arc 2422). As the user moves their thumb 2230 further downward as shown by arrow 3, all LEDs (except LED4, opposite LED10) may illuminate light in a third color, indicating the expansion of the arc 2426 in the third color. On the other hand, as the user's thumb 2230 moves away from the arc 2426, the totem may display a pattern according to arrows 4, 5, and 6. Under this sequence, the size of the arc 2426 may gradually shrink and the other arcs 2422 and 2424 may gradually fade in.

[0345] A user can also activate the touchpad using a touch gesture in FIG. 24A. FIG. 24B illustrates the halo pattern when a user activates the touchpad using a touch gesture. The totem may initially display three arcs 2422, 2424, and 2426, as shown in pattern 2462. When a user touches an area proximate to the yellow arc 2426, the length of the arc 2426 may increase. When a user activates the totem using a touch gesture at location 2418, as shown in pattern 2464, LEDs 0 and 8 may be illuminated (in a third color) and show an increased length. However, arc 2426 will not expand to encompass other arcs (e.g., arcs 2422 and 2424) as compared to pattern 2408 (which may correspond to a swipe gesture), even if the user's thumb is in the same location 2418.

[0346] Exemplary Process for Providing Cues for User Interaction on a Totem 24C illustrates an example process for providing cues for user interaction on a totem. The example process 2480 may be performed by a component of a wearable system, such as a totem or a wearable device, for example.

[0347] In block 2482, the wearable system can identify available user interface actions based on the context information. For example, the wearable system can determine available types of user interactions and map these types of user interactions to interactable areas of the totem. As an example, a browser can allow a user to scroll up / down, expand / contract the size of a web page, select items on the browser, and move a cursor within the browser. The wearable system can map cursor movement and item selection to a first interactable area of ​​the totem, where the user can swipe the touch surface, move the cursor, click the touch surface, and select items. The wearable system can also map scrolling and resizing actions to a second interactable area. The second interactable area may surround the first interactable area. The second interactable area may simulate the functionality of a D-pad, where the user can tap the up / down keys to scroll the web page, tap the left / right keys, and resize the web page.

[0348] In block 2484, the wearable system can determine the location or movement pattern of the halo associated with the available interface actions. Continuing with the example above, the wearable system can display a halo with four arcs (as shown in FIG. 25B) to indicate that the totem has a second interactable area that the user can use as a D-pad.

[0349] In block 2486, the wearable system can generate and transmit instructions to the totem's LEDs (shown in FIG. 14B) to display the halo according to a light pattern. For example, if the halo has four arcs, the wearable system can instruct LED1 to illuminate blue, LED4 to illuminate yellow, LED7 to illuminate red, and LED10 to illuminate green.

[0350] In block 2488, the wearable system can receive user input on the totem. For example, the totem can detect a swipe or touch gesture in a first interactive area or detect a tap gesture in a second interactive area. In some embodiments, the user input can also include actuating other components of the totem, such as the trigger 1212 or home button 1256 (shown in FIG. 12), either alone or in combination with a user's posture (e.g., head posture or other body posture).

[0351] Optionally, in block 2492, the wearable system may determine whether the user input is a correct input. The wearable system may make the determination based on context information of the object the user interacts with. The wearable system may determine whether the user input belongs to a type of user input supported by the object. For example, a video recording application may allow user input via posture, while a web browser may support a combination of swipe and touch gestures. Thus, when a user is browsing a web page, the wearable system may not perform user interface actions based on changes in the posture of the user's feet. As another example, a web page may allow a user to view web content, but may not allow the user to enter comments on it. Thus, if a user tries to type a phrase, the wearable system may determine that the user's input is a correct input because it is not supported by the web page.

[0352] If the user input is not a correct input, optionally, at block 2494, the wearable system can provide an indication that the user's input is incorrect. The indication can be provided in the form of visual, auditory, tactile, or other feedback. For example, the totem can highlight the correct input by providing a brighter color proximate the interactable area associated with the correct input. As another example, the totem can provide vibrations via a haptic actuator in the body of the totem. As yet another example, the wearable system can provide an alert message via the HMD or via a speaker.

[0353] If the user input is a correct input, then in block 2496 the wearable system can update the location and movement of the light pattern to reflect the user input. For example, if the user taps the left side of the totem's touch surface, the totem may illuminate an LED in the vicinity of the user's tap. In some embodiments, once the totem receives the user's input, the totem can update the displacement and movement of the light pattern associated with the halo to reflect the user input, in addition to or as an alternative to blocks 2492 and 2494.

[0354] Exemplary interaction with virtual objects using totems A totem can include multiple interactable areas, each of which can be mapped to one or more types of user interactions or types of user input devices. As described with reference to Figs. 12A and 13A, in some embodiments, a totem has a first interactable area constituting a touch surface that can support touch events and gestures, such as swipe gestures, and a second interactable area constituting a light guide. The light guide can illuminate a halo at four points around the touch pad, or with four arcs, each with a different color. This illumination pattern can indicate that the light guide is used as a D-pad. For example, a user can tap or press on the illumination point to move a virtual object. The same user gesture can cause different user interface actions to be performed depending on whether the gesture is applied to the light guide or the touch surface (e.g., within the main center area of ​​the touch pad). For example, when a user taps on the light guide, the user can move a virtual object, but when the user taps on the touch surface, the user can select a virtual object.

[0355] FIG. 25A illustrates an example interactive use of a light guide. The totem 2502 in FIG. 25A may be an example of the totem 1200 described herein. The totem 2502 may include a touch pad 2506 and may include a touch surface and a light guide that may substantially surround the touch surface. The light guide may diffuse light (e.g., emitted by an LED below the light guide) and exhibit a halo, which may include one or more arcs of the halo. In the example in FIG. 25A, the halo may include four arcs 2504a, 2504b, 2504c, and 2504d that are illuminated in four areas around the touch surface. The light guide may be interactive (e.g., a user may press a portion of the light guide) to receive user input. For example, the light guide may simulate the functionality of a D-pad. The light guide can include an up key 2508a, a right key 2508c, a down key 2508d, and a left key 2508b. The totem can provide visual indications on the locations of these keys. For example, arc 2504a can correspond to up key 2508a, arc 2504b can correspond to left key 2508b, arc 2504c can correspond to right key 2508c, and arc 2504d can correspond to down key 2508d.

[0356] A user can activate regions adjacent individual keys to implement user interface functions associated with those keys. For example, a user can activate region 2518a, activate up key 2508a, activate region 2518b, activate left key 2508b, activate region 2518c, activate right key 2508c, activate region 2518d, and activate down key 2508d.

[0357] As an example of activating a light guide, a user can move a virtual object in and out of its FOV by activating the light guide. Scene 2500 in FIG. 25A diagrammatically illustrates an example of interacting with a virtual object in FOV 2550 and FOR 2502. As described with reference to FIG. 4, the FOR can include a portion of the user's surrounding environment that can be perceived by the user via the wearable system. FOR 2502 can include a group of virtual objects (e.g., 2510, 2520, 2530, 2542, and 2544), which can be perceived by the user via the display 220. Within FOR 2502, the portion of the world that the user perceives at a given time can be referred to as FOV 2550 (e.g., FOV 2550 can encompass the portion of the FOR that the user is currently looking at). In scene 2500, FOV 2550 is diagrammatically illustrated by dashed line 2552. A user of the display can perceive multiple objects within the FOV 2550, such as object 2542, object 2544, and a portion of object 2530. The FOV 2550 may correspond to a solid angle perceivable by a user when looking through a display, such as, for example, stacked waveguide assembly 480 (FIG. 4) or planar waveguide 600 (FIG. 6). As the user's posture (e.g., head posture or eye posture) changes, the FOV 2550 changes correspondingly and the objects within the FOV 2550 may also change. For example, virtual object 2510 is initially outside the user's FOV 2550. When the user is looking at virtual object 2510, virtual object 2510 may move into the user's FOV 2550 and virtual object 2530 may move out of the user's FOV 2550.

[0358] A user can actuate the light guide to move virtual objects in and out of the FOV 2550 without changing their body posture. For example, a user can actuate key 2508b to move the virtual object to the left within the FOV 2550. Thus, virtual object 2530 can be moved entirely within the FOV 2550. As another example, a user can actuate key 2508c to move the virtual object to the right, such that virtual object 2510 can be moved into the FOV 2550 and virtual object 2530 can be moved out of the FOV.

[0359] Additionally or alternatively, the wearable system may assign focus indicators to virtual objects within the FOV 2550 based on the user's gaze direction or the location of the virtual object. For example, the wearable system may assign focus indicators to virtual objects that intersect the user's gaze direction or that are closest to the center of the FOV. The focus indicators may include haloes (e.g., around the virtual object), changes in color, perceived size or depth (e.g., making the virtual object appear closer and / or larger when selected), or other visual effects that attract the user's attention. The focus indicators may also include audible or haptic effects, such as vibrations, rings, beeps, etc. For example, the wearable system may initially assign focus indicators to virtual object 2544. When the user taps on region 2518c, the user may activate key 2508c. Thus, the wearable system may move the virtual object to the right and shift the focus indicator from object 2544 to object 2542, and the wearable system may move object 2510 into the user's FOV 2550 while moving object 2530 out of the user's FOV.

[0360] Figure 25B illustrates an example interactive use of a totem with two interactable areas. The totem 2502 can include a touch surface 2506 and a light guide, as described with reference to Figure 25A. The light guide can simulate the functionality of a D-pad, as described with reference to Figure 25A, and can display a halo having four arcs 2514a, 2514b, 2514c, and 2514d, each arc having a different color.

[0361] A user can actuate the light guide or touch surface 2506 to activate a totem. For example, in a browser window, a user can actuate the touch surface 2506 to move a cursor. A left swipe on the touch surface 2506 can be mapped to a back function, while a forward swipe on the touch surface 2506 can be mapped to a forward function for the browser page. A tap (illustrated by position 2518) at the 6 o'clock position (illuminated by arc 2514d) can cause the browser to scroll down, and a tap at the 12 o'clock position (illuminated by arc 2514a) can cause the browser to scroll up. A hold at the 6 o'clock or 12 o'clock positions can rapidly scroll pages.

[0362] The totem can provide visual feedback when a user activates the totem. For example, when a user taps at the 6 o'clock position adjacent arc 2514d, the totem can increase the brightness of arc 2514d. Haptic feedback can be used to distinguish whether the user activated the touch surface 2506 or the light guide. For example, when a user taps the light guide, an actuator under the touchpad may be activated and provide a vibration.

[0363] 25C illustrates an example process for interacting with a totem. Process 2570 may be implemented by a wearable system (such as a totem or wearable device) described herein.

[0364] In block 2572, the wearable system can receive user input on the totem. User input may include hand gestures such as swipes, taps, touches, presses, etc. The totem can include multiple interactable areas. For example, the totem can include a touchpad, which can include a touch surface and an optical waveguide, both of which may be interactable.

[0365] In block 2574, the wearable system can determine an interactable area associated with a user input from the multiple interactable areas. For example, the touch sensor 1350 of the totem can determine whether the user activated the light guide or the touch surface. The touch sensor 1350 can also detect a user gesture used to activate the totem. For example, the touch sensor can detect whether the user activated the totem using a swipe gesture or a tap gesture.

[0366] In block 2576, the wearable system can access a light pattern associated with a user input detected within the interactable area. As described with reference to Figures 24A and 24B, the light pattern can be different based on the type of user input (e.g., touch gesture vs. tap gesture). The light pattern can also be different depending on the interactable area that is actuated. For example, if the user actuates the touch surface, the light pattern may be that illustrated in Figure 24A, but if the user actuates the light guide, the light pattern may include that shown in Figure 24B.

[0367] In block 2578, the wearable system can instruct the totem's light sources (e.g., LEDs) to present a halo with a light pattern. For example, the wearable system can provide instructions regarding which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, a movement pattern associated with the light emitted by the LEDs, etc.

[0368] Exemplary interactions with physical objects using totems 26A and 26B illustrate an example of using a totem to interact with physical objects. FIG. 26A illustrates an example of a physical environment 2600, which may be a living room in a user's home. The environment 2600 has physical objects such as a television (TV) 5110, a physical remote control 5120 (sometimes simply referred to as a remote), a TV stand 5130, and a window 5140. A user can perceive and interact with the physical objects. For example, a user may watch a TV 5110 and use the remote 5120 to control the TV. A user can control the remote 5120 to turn the TV 5110 on / off or change the channel or volume of the TV 5110.

[0369] A user can also interact with the TV 5110 using a totem 2602. The totem 2602 can be an embodiment of the totem 1200, which can include multiple interactable areas. The totem 2602 can be paired with the TV 5110 or remote 5120 and simulate the functionality of the remote 5120. The functionality of the remote 5120 can be mapped to one or more interactable areas associated with the totem's touchpad, trigger, or home button. The touchpad can include a touch surface 2634 and a light guide 2630, and the touch surface 2634 and the light guide 2630 can be mapped to different types of user interactions. For example, the touch surface 2634 can be used to switch channels via a swipe gesture. The light guide 2630 can be used as a D-pad to adjust volume or fast forward / rewind via a tap gesture.

[0370] In FIG. 26B, the light guide can show a light pattern 2622 for a halo having four arcs: arc 2632a (which may be a first color, e.g., blue), arc 2632b (which may be a second color, e.g., yellow), arc 2632c (which may be a third color, e.g., red), and arc 2632d (which may be a fourth color, e.g., green). The arcs are positioned to form an x-shape. When the user 2602 touches an area on the light guide near arc 2632c, as illustrated by arrow 1, the user can fast forward a program being played on the TV5110. The totem can show a pattern 2624 to indicate that the user tapped near arc 2632c. Arc 2632c appears brighter and longer while the other arcs 2632a, 2632b, and 2632d are hidden in the pattern 2624.

[0371] The user can also change to the next channel by swiping right (as illustrated by arrow 2). The totem can illuminate a light pattern 2626 to indicate that the user has activated the totem with a swipe gesture rather than a tap gesture. Even though the user's 2602 fingers are in approximately the same position in patterns 2626 and 2624, the light patterns illuminated in these two situations are different because of the different gestures used to activate the totem. For example, the brightness and length of arc 2632c has increased, but the other arcs 2632a, 2632b, and 2632d are not obscured in pattern 2626. Rather, the totem simply reduces the brightness of the other arcs.

[0372] 26A and 26B are described without a wearable device in some embodiments, but a user can use a totem to control physical objects while wearing the wearable device. For example, a user can perceive physical objects through the wearable device and perceive virtual objects projected by the wearable device. A user can use a totem to interact with physical objects and virtual objects.

[0373] An Exemplary 6DOF User Experience with Totem FIG. 27 illustrates an example of using a multi-DOF (e.g., 3DOF or 6DOF) totem to move a virtual object. A user can change the position of the totem by moving it forward / back (yaw), up / down (bob), or left / right (yaw). A user can also change the orientation of the totem by tilting it to the side (roll), tilting it forward / back (pitch), or steering it left and right (yaw). By changing the position and orientation of the totem, a user can perform various interface operations on the virtual object (e.g., by moving or rotating, etc.).

[0374] A user may use the totem 2602 to select a virtual object, hold and move the virtual object, and move the virtual object by physically moving the totem 2602. For example, the totem 2602 may initially be at a first location 5100a. The user 5300 may select a target virtual object 5400 located at a first location 5100b by activating the totem 504 (e.g., by activating a touch-sensitive pad on the totem). The target virtual object 5400 can be any type of virtual object that can be displayed and moved by the wearable system. For example, the virtual object may be an avatar, a user interface element (e.g., a virtual display), or any type of graphical element displayed by the wearable system. The user 5300 may move the target virtual object from the first location 5100b to the second location 5200b along a trajectory 5500b by moving the totem 2602 along a trajectory 5500a.

[0375] As another example, instead of moving the virtual object 5400 from 5100b to 5200b, the user 5300 may want to move the virtual object 5400 closer to himself / herself. Thus, the user can move the totem closer to himself / herself, bringing the virtual object 5400 closer.

[0376] As yet another example, totem 2602 may be at position 5200a. The user may rotate the totem 180 degrees clockwise, and virtual object 5400 may thus be rotated 180 degrees at position 5200b.

[0377] The totem can emit a light pattern to indicate the type of user interaction. For example, a ring of LEDs may illuminate blue when the user moves the totem, while illuminating green when the user rotates the totem. As another example, the totem's light movement pattern may correspond to the totem's rotation. For example, the light pattern may include an arc. As the user rotates the totem, the arc may also rotate in the direction of the user's rotation.

[0378] An example of decorative and informative visual feedback on a totem Some wearable devices may have a limited FOV. Advantageously, in some embodiments, the totem can provide information associated with objects outside the FOV using light patterns illuminated by the totem.

[0379] 28A and 28B illustrate examples that provide information of objects through the location and movement of light patterns. FIG. 28A illustrates a person's FOV 2820 and FOR 2810. The FOR 2810 can contain a group of objects (e.g., objects 2812, 2822, 6050b, 6050c) that can be perceived by a user wearing a wearable system. The FOV 2820 can contain multiple objects (e.g., objects 6050, 2822). The FOV can depend on the size or optical properties of the wearable system, for example, the effective aperture size of a transparent window or lens of a head mounted display through which light passes from the real world in front of the user to the user's eyes. In some embodiments, as the pose of the user 6100 (e.g., head pose, body pose, and / or eye pose) changes, the FOV 2820 can change correspondingly, and the objects within the FOV 2820 can also change. As described herein, the wearable system may include sensors, such as cameras, that monitor or image objects within the FOR 2810 and objects within the FOV 2820. In such embodiments, the wearable system may alert the user (e.g., via light patterns on the totem) to unnoticed objects or events occurring within the user's FOV 2820 or occurring outside the user's FOV but within the FOR 2810. In some embodiments, the wearable system can also distinguish between objects to which the user 6100 is or is not attending.

[0380] Objects in the FOV or FOR may be virtual or physical objects. Virtual objects may include, for example, operating system objects, such as a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. Virtual objects may also include objects within an application, such as, for example, avatars, virtual objects within a game, graphics or images. Some virtual objects can be both operating system objects and objects within an application. The wearable system may add virtual elements, viewed through the transparent optics of the head mounted display, to existing physical objects, thereby enabling user interaction with the physical objects. For example, the wearable system may add a virtual menu associated with a medical monitor in a room, which may give the user the option to turn on or adjust medical imaging equipment or medication controls using the wearable system. Thus, the wearable system may present additional virtual image content to the wearer in addition to objects in the user's environment.

[0381] The totem can provide information of objects outside the FOV through the location and movement of the light pattern. The location and movement of the light pattern can be calculated and determined by various components of the wearable system, such as, for example, the totem or the wearable device. The wearable system can use contextual information associated with the user or object to determine the light pattern. For example, the wearable system can employ factors such as, for example, the location of the object (including the proximity of the object to the user or the user's FOV), the urgency of the object, the type of object (e.g., whether the object is interactable, the type of interaction supported by the object, whether the object is physical or virtual, etc.), the nature of the object (e.g., a competitor avatar vs. a friend avatar, etc.), the amount of information (e.g., number of notifications, etc.), user preferences, etc. As shown in scene 2834 in FIG. 28B, the halo 2844 on the totem 2502 may appear smaller (e.g., having a shorter angular range) than the halo 2842 (which has a larger angular range) because the object 2840 is farther from the user 6100 than the scene 2832. As another example, a halo may have a larger and brighter appearance because the object associated with the halo is more urgent or closer to the user's FOV.

[0382] The wearable system may assign a color to the halo based on the characteristics of the associated object. For example, the wearable system may assign the color red to halo 6051b because object 6050b is associated with the color red. Similarly, the wearable system may assign the color blue to object 6051a because it is an operating system object and the wearable system assigns the color blue to all operating system objects.

[0383] The wearable system may assign colors to the halos based on the characteristics of the associated objects. For example, the wearable system may assign a green color to halos 2842 and 2844 because object 2840 is associated with the color green. Similarly, the wearable system may assign a red color to object 6050b because it is an operating system object and the wearable system assigns the color red to all operating system objects. As a result, halos associated with operating system objects (including object 6050b) may be red.

[0384] The appearance of the halo may change over time based on changes in the object associated with the halo. For example, the halo may grow darker as the object associated with the halo receives more messages. As another example, the halo 2844 may grow larger (or brighter) as the object 2840 moves closer to the user, or grow smaller (or dimmer) as the object 2840 moves away from the user. The position of the halo may also change as the object moves. For example, in FIG. 28A, the wearable system may initially show the halo associated with object 6050b to the left of the totem's light guide because object 6050b is to the left of user 6100. However, if object 6050b moves to the right of user 6100, the wearable system may update the position of the halo to the right of the totem's light guide.

[0385] The appearance of the visual halo may also change based on changes in the user's posture. For example, referring to FIG. 28A, as the user 6100 turns left, object 6050b may be closer to the user's FOV, while object 6050c may be further away from the user's FOV. As a result, the halo associated with object 6050b may become brighter (or larger).

[0386] In some situations, the FOR 2810 and FOV 2820 may include invisible objects that are not directly visible to the user. For example, a treasure hunt game may include a hidden treasure (e.g., objects 2812, 2822 shown in dashed lines) in the user's FOR 2810. The treasure may be embedded in a game object and therefore may not be directly visible to the user. However, the user may use a virtual tool in the game (e.g., a virtual shovel, etc.) to uncover the hidden object. As another example, the user's physical car key may be under a pile of papers and the user may not be able to perceive the physical car key in the user's FOV. A halo may be used to provide an indication of the location of the hidden car key. For example, the size and location of the halo arc may indicate the relative distance and orientation between the car key and the totem. The location of the car key may be determined based on a mapping of the user's environment (e.g., based on images previously obtained by an outward-facing imaging system) or, if the key is equipped with wireless or wireless communication capabilities, based on wireless communication between the Totem (or other component of the wearable system) and the car key.

[0387] As yet another example, an object (e.g., object 6050a) may be outside the user's visual FOR, but may nevertheless potentially be perceived by a sensor (e.g., an electromagnetic sensor, a radio frequency sensor, a sensor associated with detecting radio signals, or another environmental sensor) on the wearable device or totem. For example, object 6050a may be behind a wall in the user's environment such that object 6050a is not visually perceptible by the user. However, the wearable device or totem may include a sensor that may communicate with object 6050a.

[0388] The totem can provide information of invisible objects through light patterns. The location and movement of the light patterns can be based on contextual information associated with the invisible object or the user. For example, the location and movement of the light patterns on the light guide can act as a compass, directing the user to look or move in a certain direction. As another example, the light patterns may provide a map of invisible objects in the user's environment. For example, the totem can illuminate a 30 degree red arc at the 1 o'clock position to indicate that the direction of the car keys is to the right in front of the user, while illuminating a 90 degree green arc at the 6 o'clock position to indicate that the user's cell phone is behind the user. The red and green arcs can also be used to indicate the proximity of the car keys and cell phone. For example, the cell phone is closer to the user than the car keys, and as a result, the green arc is larger than the red arc. As the user moves, the location (or movement) of the light patterns can also change. For example, as the user moves forward, the size of the green arc can decrease while the size of the red arc can increase.

[0389] In some implementations, the wearable system may present a mapping of the user's physical or virtual environment, which may indicate the location of invisible objects. For example, the HMD may present a virtual map, which indicates in-game objects that the user may find. As another example, the HMD may provide a focus indicator or other type of visual indication of a hidden physical car key (or virtual object) as AR / MR content. As the user moves around in the environment, the appearance (e.g., size and relative direction from the user) of the focus indicator may change. In some situations, the HMD may present a route to an invisible object. The map presented by the HMD may be combined with a halo on a totem to guide the user to the invisible object.

[0390] In some embodiments, the totem can present a halo that includes information associated with multiple objects. For example, a user 6100 may have a red object 6050b to their left and an invisible object 2812 behind them. The totem can present a halo that has a red arc on the left side of the light guide and a silver arc on the bottom of the light guide, where the red arc corresponds to the red object 6050b and the silver arc corresponds to the invisible object 2812.

[0391] In addition to, or as an alternative to, providing information of objects using light patterns, the display can also be configured to provide information of objects that are outside the FOV or are not visible. For example, the display can display a visual halo for a corresponding object that is outside the user's current FOV. A portion of the visual halo can be located on an edge of the user's FOV of the display. The location of the halo can be based on contextual information of the object.

[0392] In addition to visual haloes, the wearable system can also alert the user to objects using haptic or auditory effects. For example, in a mixed reality game, the wearable system may notify the user of approaching competitors through vibrations on a totem. The wearable system may provide strong vibrations when a competitor is in close proximity to the user. In another example, the wearable system may provide location information of virtual objects using audible sounds. The wearable system may also use a high-pitched sound to alert the user to nearby virtual competitors. Haptic, auditory, or visual feedback can be used in combination with or as an alternative to alerting the user to surrounding objects.

[0393] 28C illustrates an example process for providing information associated with an object using light patterns. Process 2870 can be implemented by a wearable system (such as a totem or wearable device) described herein.

[0394] In block 2872, the wearable system can identify an object in the user's environment. The object may be a physical or virtual object. The object may be in the user's FOR but outside the user's FOV. The object may also be hidden from the user's view. For example, the object may be behind another object (e.g., a wall) or may only become visible with some user interface interaction (e.g., when the user completes a level in a game).

[0395] At block 2874, the wearable system can access contextual information associated with the object. The contextual information may be used to determine the movement and location of the light pattern associated with the object, as shown in block 2874. For example, the brightness of the halo with the light pattern may be determined based on the proximity or urgency of the object. The color of the halo may also match the color of the object.

[0396] In block 2878, the wearable system can instruct the totem's LEDs to illuminate according to a light pattern. For example, the instructions may include which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, a movement pattern associated with the light emitted by the LEDs, etc.

[0397] FIG. 29A illustrates an example light location or movement pattern indicating receipt of a notification. Patterns 2912 and 2914 include rainbow patterns. The totem can display a rainbow trail in a clockwise cycle (as shown by process 2920) to provide a notification. The notification may be associated with a virtual object, such as, for example, an email application, a game, a video application, etc. The notification may also be associated with an operating system of the wearable device. One example notification may be an error message from the operating system of the wearable device. Additionally or alternatively, the notification may be associated with a physical object. For example, rainbow patterns 2912 and 2914 may indicate that a coffee machine in the user's kitchen has finished brewing.

[0398] The totem may repeat the rainbow trajectory until a threshold condition is reached, which may be based on duration, user interaction (such as activation of a trigger 1212), or other contextual information.

[0399] 29B illustrates an example process for providing notification using a light pattern on a totem. The example process 2970 can be implemented by a wearable system (such as a totem or wearable device) described herein.

[0400] In block 2972, the wearable system may determine a status of the virtual object. For example, the status may include whether the virtual object has received new information (e.g., a new message), whether the virtual object is idle, whether the virtual object has encountered a problem, etc.

[0401] Optionally, at block 2974, the totem can generate an alert based at least in part on the status. For example, the virtual object can be a messenger application. When the virtual object receives a new message from a friend of the user, the wearable system can generate an alert to indicate that a new message has arrived.

[0402] In block 2976, the wearable system can access a light pattern associated with the status of the virtual object (or optionally, the alert). The location or movement pattern may be determined based on context information associated with the virtual object (or alert), such as, for example, the importance of the alert, the urgency of the alert, the location of the virtual object, the type of virtual object, etc.

[0403] In block 2978, the wearable system can instruct the totem's LEDs to illuminate according to a light pattern. For example, the wearable system can provide instructions regarding which LEDs to illuminate, the brightness of the LEDs, the color of the LEDs, the movement pattern associated with the light emitted by the LEDs, etc.

[0404] In addition to, or as an alternative to, providing information about objects outside the FOV, light patterns can also be used to provide information about objects that are not necessarily outside the FOV. For example, when a user completes a level in a game, a totem may illuminate with a halo, which has the same color as the game. As another example, referring to FIG. 29A, a virtual application associated with notifications may be within the FOV. A user may receive a notification that a new email has arrived while interacting with an email application, for example, by composing a message.

[0405] The totem can provide visual feedback of the user's current interaction to the user or others in the user's environment. For example, when a user is recording video with their HMD, the totem can illuminate a blinking red light pattern to inform the user and indicate to others in the vicinity that the HMD is in recording mode.

[0406] FIG. 30 illustrates an example light pattern that may be used to inform people in the user's environment of the user's current interaction. For example, when a user is in a telepresence session, the totem may illuminate a light pattern 3000 (e.g., a green halo with a clockwise movement) via a light guide. The light pattern 3000 may inform people in the user's environment that the user is in telepresence. This may help to keep other people from getting too close to the user, thereby preventing other people from disrupting the user's telepresence session. In addition to or as an alternative to light patterns, other types of feedback, such as, for example, haptic, auditory, video feedback, etc., may also be used to indicate the user's current interaction with the wearable system.

[0407] Additional aspects In a first aspect, a system includes a light emitting assembly of a user input device, the light emitting assembly configured to illuminate a plurality of light patterns to provide information of an object in an environment; and a hardware processor communicatively coupled to the light emitting assembly and programmed to: identify an object in a user's environment, access contextual information associated with the object, determine a characteristic of the light pattern illuminated by the light emitting assembly based, at least in part, on the contextual information, and instruct the light emitting assembly to illuminate according to the light patterns.

[0408] In a second aspect, the system of aspect 1, wherein the object includes at least one of a physical object or a virtual object.

[0409] In a third aspect, the system of aspect 2, wherein the status of the virtual object includes at least one of a current interaction with the virtual object by a user, whether the virtual object has received new information, whether the virtual object is idle, or whether the virtual object is in an error state.

[0410] In a fourth aspect, the system of any one of aspects 1-3, wherein the characteristics of the light pattern include at least one of brightness, position, shape, size, or color.

[0411] In a fifth aspect, the system of any one of aspects 1-4, wherein the contextual information associated with the object includes at least one of a location of the object, an urgency of the object, a type of the object, a nature of the object, an amount of information associated with the object, or a user preference.

[0412] In a sixth aspect, the system further comprises a wearable display device, wherein the object is invisible from the user's view or outside the view of the wearable display device, and the hardware processor is programmed to determine at least one of a size, shape, or color of the light pattern and provide a cue to the user for locating the object.

[0413] In a seventh aspect, the system of any one of aspects 1-6, wherein the object is a component of a wearable system for presenting virtual content to a user, and the light pattern indicates a status of the component of the wearable system.

[0414] In an eighth aspect, the system of aspect 7, wherein the component includes at least one of a user input device, a wearable display device, ...

Claims

1. A light emitting user input device for providing user input to a display system including a wearable display device, the light emitting user input device comprising: a plurality of interactable areas including a central touch surface configured to receive user input; a light emitting assembly including a light guide exterior to and at least partially surrounding the central touch surface; a hardware processor communicatively coupled to the plurality of interactable areas and the light emitting assembly; Equipped with The hardware processor includes: Detecting actuation of the light emitting user input device by a user of the wearable display device; determining an interactable area among the plurality of interactable areas corresponding to the actuation; and translating the action into a user input for performing a user interface action for interacting with a virtual environment presented through the wearable display device, the virtual environment including virtual content generated by an application executing on the display system, the user interface action being determined based at least on a type of the action and the interactable area corresponding to the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input, the light pattern being based at least in part on the application executing on the display system; a light emitting user input device programmed to:

2. The plurality of interaction regions include a first interaction region and a second interaction region, In response to determining that the user has activated the first interactable area, the hardware processor is programmed to cause the light emitting assembly to illuminate a first light pattern associated with the user input within the first interactable area; 2. The light-emitting user input device of claim 1, wherein in response to determining that the user has activated the second interactable area, the hardware processor is programmed to cause the light-emitting assembly to illuminate a second light pattern associated with the user input within the second interactable area.

3. The light-emitting user input device of claim 1, wherein the actuation includes at least one of a swipe, a tap, a press, or a touch gesture.

4. The light-emitting user input device of claim 1, wherein the light pattern includes an arc-shaped region having one or more of color, arc length, or visual effect.

5. The light-emitting user input device of claim 1, wherein the hardware processor is further programmed to communicate with the display system based on the user input and cause the display system to perform the user interface operation.

6. The light-emitting user input device of claim 1, wherein the optical waveguide is substantially circular.

7. The light emitting user input device of claim 1, wherein the optical waveguide is touch sensitive to receive user input.

8. The luminous user input device of claim 1, wherein the central touch surface is substantially circular.

9. The light emitting user input device of claim 1, wherein the optical waveguide completely surrounds the central touch surface.

10. The light-emitting user input device of claim 1, wherein the wearable display device is separate from the light-emitting user input device.

11. A method comprising: Under the control of a hardware processor, Detecting actuation of a light-emitting user input device by a user of a wearable display device, the light-emitting user input device configured to provide user input to a display system including the wearable display device, the light-emitting user input device comprising: a plurality of interactable areas including a central touch surface configured to receive input; and a light-emitting assembly comprising an optical waveguide exterior to and at least partially surrounding the central touch surface; determining an interactable area among the plurality of interactable areas corresponding to the actuation; and translating the action into a user input for performing a user interface action for interacting with a virtual environment presented through the wearable display device, the virtual environment including virtual content generated by an application executing on the display system, the user interface action being determined based at least on a type of the action and the interactable area corresponding to the action; instructing the light emitting assembly to illuminate a light pattern in response to the user input, the light pattern being based at least in part on the application running on the display system; The method of claim 1, further comprising:

12. The plurality of interaction regions comprises a first interaction region and a second interaction region, The method comprises: in response to determining that the user has activated the first area of ​​interaction, causing the light emitting assembly to illuminate a first light pattern associated with the user input within the first area of ​​interaction; in response to determining that the user has activated the second area of ​​interactivity, causing the light emitting assembly to illuminate a second light pattern associated with the user input within the second area of ​​interactivity; The method of claim 11 further comprising:

13. The method of claim 11, wherein the actuation includes at least one of a swipe, a tap, a press, or a touch gesture.

14. The method of claim 11, wherein the light pattern includes an arc-shaped region having one or more of color, arc length, or visual effect.

15. The method of claim 11, further comprising causing the display system to perform the user interface action based on the user input.

16. The method of claim 11, wherein the optical waveguide is substantially circular.

17. The method of claim 11, wherein the optical waveguide is touch sensitive to receive user input.

18. The method of claim 11, wherein the central touch surface is substantially circular.

19. The method of claim 11, wherein the optical waveguide completely surrounds the central touch surface.

20. The method of claim 11, wherein the wearable display device is separate from the light-emitting user input device.

Images