Multi-mode execution and text editing for wearable systems

Abstract

To provide multimode execution and text editing for a preferred wearable system.SOLUTION: In embodiments of a wearable system and a method, multiple inputs (e.g., gestures, head pose, eye gaze, voice, and / or environmental factors (e.g., location)) are used, so that a command to be executed and an object to be worked in a three-dimensional (3D) environment can be determined. The multiple inputs can also be used by the wearable system to allow the user to interact with the text, e.g., compose, select, or edit the text.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 62 / 487,364, filed April 19, 2017, and entitled “MULTI-MODAL CONTEXTUAL TASK EXECUTION FOR AUGMENTED REALITY,” and U.S. Provisional Application No. 62 / 609,647, filed December 22, 2017, and entitled “MULTI-MODAL TEXT COMPOSITION AND EDITING FOR AUGMENTED REALITY,” the disclosures of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to interacting with virtual objects or text within a three-dimensional (3D) environment using a combination of user inputs. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or may be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or "MR" refers to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to produce VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] Examples of wearable systems and methods described herein can use multiple inputs (e.g., gestures, head pose, eye gaze, voice, or environmental factors (e.g., location)) to determine commands to be performed and objects in a three-dimensional (3D) environment to be acted upon. Multiple inputs can also be used by the wearable device to allow a user to interact with text, such as to compose, select, or edit text.

[0005] For example, a wearable display device can be configured to analyze multi-modal inputs for task execution. The wearable device can use a combination of multiple inputs, such as head pose, eye gaze, hand gestures, voice commands, environmental factors (e.g., the user's location or objects around the user), to determine virtual objects in the user's environment that the wearable device will interact with, the types of actions the wearable device may perform on the virtual objects, and how the wearable device will perform the actions.

[0006] As another example, the wearable device can be configured to analyze multimodal input to interact with text. The wearable device can compose (e.g., input) and edit text using a combination of multiple inputs, such as voice input, eye gaze, hand gestures, and totem input. The wearable device may enable a user to dictate text to the system using a first input mode (e.g., voice input), select a portion of the text for editing using a second, different input mode (e.g., eye gaze input or body gestures), and edit the selected text using the first mode, the second mode, yet another mode, or a combination of the modes. The present invention provides, for example, the following: (Item 1) 1. A system comprising: a first sensor of the wearable system configured to obtain first user input data in a first input mode; a second sensor of the wearable system configured to obtain second user input data in a second input mode, the second input mode being different from the first input mode; and a hardware processor in communication with the first and second sensors, the hardware processor comprising: receiving a multi-modal input comprising first user input data in the first input mode and second user input data in the second input mode; identifying a first set of candidate objects for interaction based on the first user input data; identifying a second set of candidate objects for interaction based on the second user input data; determining a target virtual object from the first set of candidate objects and the second set of candidate objects based on a combination of the first user input data and the second user input data; determining a user interface action on the target virtual object based on at least one of the first user input data or the second user input data; generating a multi-modal input command that causes the user interface action to be performed on the target virtual object; a hardware processor programmed to A system comprising: (Item 2) Item 1, wherein the multi-modal input comprises at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice. (Item 3) 2. The system of claim 1, wherein the user interface action comprises at least one of selecting, moving, or resizing the target virtual object. (Item 4) 2. The system of claim 1, wherein the hardware processor is further configured to determine at least one of a target location, orientation, or movement for the target virtual object in the user interface action. (Item 5) 5. The system of claim 4, wherein to determine a target location for the target virtual object, the hardware processor is programmed to identify a workable surface within a physical environment for placing the target virtual object. (Item 6) The workable surface is calculating a distance function for a point of interest (POI) on a physical object within the physical environment; eliminating one or more of the POIs that are out of plane tolerance; and Delineating the workable surface based on the remaining POIs; and Item 6. The system according to item 5, characterized by the following: (Item 7) Item 6. The system of item 5, wherein the hardware processor is programmed to automatically orient the target virtual object to match an orientation of the target location. (Item 8) 2. The system of claim 1, wherein the action is determined based on first user input data in the first input mode, and at least one of the object or the parameters is determined based on a combination of the first input mode and the second input mode. (Item 9) 2. The system of claim 1, wherein the first input mode comprises an indirect input mode based on location information of a user of the wearable system. (Item 10) 10. The system of claim 9, wherein the hardware processor is programmed to identify a virtual object as the target virtual object from the first set of objects and the second set of objects in response to determining that an object is within a threshold range of the user. (Item 11) 2. The system of claim 1, wherein the user interface action is associated with a virtual application, the virtual application being programmed to be more responsive to one of the first sensor or the second sensor. (Item 12) 2. The system of claim 1, wherein the hardware processor is programmed to perform a tree-based analysis on the first set of candidate objects and the second set of candidate objects based on the first user input data and the second user input data to determine the target virtual object from the first set of candidate objects and the second set of candidate objects. (Item 13) To determine the target virtual object from the first set of candidate objects and the second set of candidate objects, the hardware processor: calculating a first confidence score for a candidate object in the first set of candidate objects based on the first user input data; calculating a second confidence score for the candidate object based on the second user input data; calculating an aggregate score for the candidate object from at least the first confidence score and the second confidence score; in response to determining that the aggregated score satisfies a threshold condition, setting the candidate object as the target virtual object; Item 13. The system according to item 12, programmed to: (Item 14) To determine the target virtual object, the hardware processor: the uniformity of the space around the virtual object within the field of view; a proportional area of ​​a first portion of the virtual object that is within the user's field of view relative to a second portion of the virtual object that is outside the user's field of view; or A historical analysis of user interactions with said virtual object. Item 14. The system of item 1, wherein the system is programmed to calculate a confidence score for the virtual object by calculating at least one of: (Item 15) 2. The system of claim 1, wherein the hardware processor is further programmed to detect a start condition for the interaction event that triggers the hardware processor to determine the target virtual object and the user interface action based on the multi-modal input. (Item 16) Item 16. The system of item 15, wherein the initiation condition comprises a trigger phrase. (Item 17) 2. The system of claim 1, wherein the first input mode is a primary input mode and the second input mode is a secondary input mode, and the hardware processor is programmed to resolve ambiguities in at least one of the target virtual object and the user interface action based on the second user input data. (Item 18) 2. The system of claim 1, wherein the first user input data comprises a deictic or anaphoric reference to a virtual object, and a hardware processor is programmed to identify the virtual object as the target virtual object as the subject based on the second user input data. (Item 19) 2. The system of claim 1, wherein the hardware processor is further programmed to automatically enable, disable, or adjust the sensitivity of the first input mode, the second input mode, or both based, at least in part, on a user setting or the user's environment. (Item 20) 2. The system of claim 1, wherein the hardware processor is programmed to identify a target virtual object outside of the user's field of view based, at least in part, on the multi-modal input, and automatically move the virtual object into the field of view for user interaction. (Item 21) 1. A method comprising: Under control of a hardware processor of the wearable system in communication with a plurality of sensors configured to obtain user input data; receiving the user input data from the plurality of sensors relating to a user-environment interaction event; analyzing the user input data and identifying a multi-modal input for interacting with the environment, the multi-modal input comprising a first input on a first input channel and a second input on a second input channel; determining a multi-modal input command to describe the interaction event with the environment based on the first input and the second input, the multi-modal input command comprising one or more of an object, a parameter, or an action, wherein at least one of the object, the parameter, or the action is identified based on a combination of the first input and the second input; causing the wearable device to execute the multimodal input command for the interaction event; A method comprising: (Item 22) 22. The method of claim 21, wherein the action comprises at least one of selecting, moving, or resizing the object, the object comprises a target virtual object with which the user intends to interact, or the parameter comprises at least a target location, orientation, or movement of the object. (Item 23) The target location is Calculating a distance function for a point of interest (POI) on a physical object; Rejecting one or more of the POIs that are outside a given planarity tolerance; and Delineating a workable surface on the physical object based on the remaining POIs; and 23. The method according to claim 22, wherein the (Item 24) 23. The method of claim 22, wherein the method includes automatically orienting the virtual object to match that of the target location. (Item 25) 22. The method of claim 21, wherein the multi-modal input comprises at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice. (Item 26) 26. The method of claim 25, wherein the first input channel is speech and the second input channel is head pose or hand gesture. (Item 27) 22. The method of claim 21, wherein at least one of the actions, objects, or parameters is further identified based on environmental or location information of the user. (Item 28) 28. The method of claim 27, wherein the target is selected from a group of objects within a threshold range of the user. (Item 29) 22. The method of claim 21, wherein the interaction event occurs within a virtual application, the virtual application being programmed to be more responsive to one of the first input channel or the second input channel. (Item 30) 22. The method of claim 21, wherein the target is identified by analyzing the first input and the second input using a lattice tree analysis to identify a virtual object to set as the target. (Item 31) The object or the parameter in the multi-mode input command is: calculating, for a candidate object or parameter, a first confidence score that correlates with the first input and a second confidence score that correlates with the second input; calculating an aggregate score for the candidate object or parameter from at least the first confidence score and the second confidence score; setting the candidate object or parameter as the target or the parameter for the interaction event, respectively, based at least in part on the aggregated score; 22. The method according to claim 21, wherein the (Item 32) Candidate virtual objects are identified as the subject based on confidence scores of virtual objects in the user's field of view, the confidence codes being: the uniformity of the space around the candidate virtual object within the field of view; a proportional area of ​​a first portion of the candidate virtual object within the user's field of view relative to a second portion of the candidate virtual object outside the user's field of view; or A historical analysis of user interactions with the candidate virtual objects. 22. The method according to claim 21, wherein the calculation is based on at least one of the following: (Item 33) 22. The method of claim 21, further comprising detecting a start condition for the interaction event based on data received from one or more of the plurality of sensors, the start condition triggering the receiving, analyzing, determining, and generating steps. (Item 34) The first input from the first input channel is a primary input and the second input from the second input channel is a secondary input, the method comprising: analyzing the first input to identify the object, the parameter, and the action; resolving ambiguities in at least one of the object, the parameter, or the action based on the second input and generating the multi-modal input command; 22. The method according to claim 21, comprising: (Item 35) 22. The method of claim 21, further comprising automatically enabling, disabling, or adjusting sensitivity of input channels based, at least in part, on user settings or the user's environment. (Item 36) 22. The method of claim 21, further comprising identifying a virtual object outside the user's field of view as the subject based at least in part on the user interaction, and automatically moving the virtual object into the field of view for user interaction. (Item 37) 22. The method of claim 21, wherein the first input comprises a deictic or anaphoric reference as the target, and the method further comprises selecting a target object as the target based on the second input. (Item 38) 1. A system comprising: a head-mounted display (HMD) of a wearable system configured to present three-dimensional (3D) virtual content to a user; two or more user input components configured to receive user input of respective modalities, one of the user input components comprising an audio sensing device configured to capture sound; a hardware processor communicatively coupled to the display and the two or more user input components, the hardware processor comprising: receiving speech data from the audio sensing device encoding an utterance of one or more words spoken by a user; obtaining a transcript of the one or more words spoken by a user based on at least the received speech data; and controlling the display to present to a user a string of text characters representing the captured transcript; receiving user input data indicative of a user input of another different modality from another of the two or more user input components; determining that the user input data received from the other user input component represents a command to select a particular subset of the text characters for editing; in response to determining that the user input data received from the other user input component represents a command to select a particular subset of the text characters for editing; determining whether subsequently received data from any of the two or more user input components represents a command to modify a particular subset of the text characters in a particular manner; a hardware processor programmed to A system comprising: (Item 39) Item 39. The system of item 38, wherein the other user input component comprises an eye gaze tracking device configured to obtain data indicative of the user's eye gaze direction. (Item 40) The hardware processor further comprises: determining, based on at least the data received from the eye tracking device, that the user is fixating on a particular subset of the text characters for a period of time longer than a threshold; determining that the user input data received from the other user input component represents a command to select the particular subset of the text characters for editing in response to determining that the user has fixated the particular subset of the text characters for a period of time longer than a threshold; 40. The system of item 39, programmed to: (Item 41) The hardware processor further comprises: receiving additional speech data from the audio sensing device encoding an utterance of a phrase spoken by a user; determining, based on at least the data received from the eye tracking device and additional speech data received from the audio sensing device, that the user uttered one or more predefined hot words while fixating on the particular subset of text characters; determining, in response to determining that the user uttered one or more predefined hot words while fixating on the particular subset of text characters, that the data received from the eye tracking device and the additional speech data received from the audio sensing device represent a command to select the particular subset of text characters for editing; 40. The system of item 39, programmed to: (Item 42) the two or more user input components further comprise a gesture tracking device configured to obtain data indicative of a hand gesture of the user; The hardware processor further comprises: receiving data indicative of an eye gaze direction of the user from the eye gaze tracking device; receiving data indicative of a hand gesture of the user from the gesture tracking device; determining that the user performed one or more predefined hand gestures while fixating on the particular subset of text characters based on at least the data received from the eye tracking device and the data received from the gesture tracking device; determining that the data received from the eye tracking device and the gesture tracking device represents a command to select the particular subset of the text characters for editing in response to determining that the user has made one or more predetermined hand gestures while fixating on the particular subset of the text characters; 40. The system of item 39, programmed to: (Item 43) the two or more user input components further comprise a touch-sensitive device configured to obtain data indicative of a physical interaction of the user therewith; The hardware processor further comprises: receiving data indicative of an eye gaze direction of the user from the eye gaze tracking device; receiving data from the touch-sensitive device indicative of the user's physical interaction with the touch-sensitive device; determining whether the user provided one or more predefined touch inputs while fixating on a particular subset of the text characters based on at least the data received from the eye tracking device and the data received from the touch sensitive device; determining that, in response to determining that the user has provided one or more predefined touch inputs while fixating on the particular subset of text characters, data received from the eye tracking device and the touch sensitive device represents a command to select the particular subset of text characters for editing; 40. The system of item 39, programmed to: (Item 44) Item 39. The system of item 38, wherein the hardware processor implements an automatic speech recognition (ASR) engine and is programmed to obtain the transcript. (Item 45) 45. The system of claim 44, wherein the ASR engine is configured to produce a score associated with one or more words in the string of text that indicates the likelihood that the ASR engine has correctly transcribed such words. (Item 46) Item 46. The system of item 45, wherein the hardware processor is further programmed to cause the HMD to highlight the one or more words if the likelihood of a correct transcription is below a threshold level. (Item 47) 1. A system comprising: a display of the wearable system configured to present virtual content to a user; an audio sensing device configured to capture words spoken by a user and generate speech data; an eye gaze tracking device of the wearable system configured to track the gaze of the user; a hardware processor communicatively coupled to the display, the audio sensing device, and the eye tracking device, the hardware processor comprising: obtaining a text transcription of one or more words spoken by a user based at least in part on the speech data from the audio sensing device; and controlling said display to present said text to a user; determining, based on at least the data received from the eye tracking device, that the user has provided a command to select a portion of the presented text for editing; and performing an editing action on the portion of the presented text; and a hardware processor programmed to A system comprising: (Item 48) Item 48. The system of item 47, wherein the hardware processor is further programmed to determine that the user has given a command to select the given word for editing based on data from the eye tracking device indicating that the user's gaze remained on a portion of the presented text presented by the display for at least a threshold time period. (Item 49) Item 48. The system of item 47, further comprising a user input device, wherein the hardware processor is further programmed to determine that the user has given a command to select a portion of the presented text for editing based on data from the user input device and data from the eye gaze tracking device indicating that the user input device received user input while the user's gaze was focused on a portion of the presented text presented by the display. (Item 50) Item 48. The system of item 47, wherein the hardware processor is programmed to determine that the user has given a command to select a portion of the presented text for editing based on data from the audio sensing device and data from the eye gaze tracking device indicating that the audio sensing device received a voice command while the user's gaze was focused on a portion of the presented text presented by the display. (Item 51) Item 48. The system of item 47, further comprising an imaging system that images at least one hand of the user, and the processor is configured to determine that the user has given a command to select a portion of the presented text for editing based on data from the imaging system and data from the eye tracking device indicating that the user made a command gesture with his or her hand while the user's gaze was focused on a portion of the presented text presented by the display. (Item 52) Item 48. The system of item 47, wherein the hardware processor is further programmed to control the display to present an alternative transcript of a portion of the presented text in response to a command to select the given word for editing. (Item 53) The hardware processor further comprises: determining, based on additional data received from the eye tracking device, that the user has provided a command to replace a portion of the presented text with a selected alternative transcript; and revising the text and replacing a portion of the presented text with the selected alternative transcript; controlling the display to present the revised text to a user; Item 48. The system of item 47, programmed to: (Item 54) 48. The system of claim 47, wherein the hardware processor is further programmed to produce a score associated with one or more words in the text indicating the likelihood that such words were correctly transcribed. (Item 55) 55. The system of claim 54, wherein the hardware processor is further programmed to cause the display to highlight the one or more words if the likelihood of a correct transcript is below a threshold level. (Item 56) 1. A method comprising: Under the control of a hardware processor, receiving spoken input from a user via a microphone; converting the spoken input into text comprising a plurality of words; causing a wearable display to present the text to a user; receiving a selection of a portion of the presented text within the displayed text based on at least data from an eye-tracking system; providing the user with an opportunity to edit a portion of the presented text; A method comprising: (Item 57) The step of receiving a selection of a portion of the presented text includes: determining that the user's gaze is focused on the given word for at least a predetermined threshold period of time; determining that the user's gaze is focused on a portion of the presented text while receiving a predetermined spoken command from the user requesting an edit using the microphone; determining, while receiving data for actuation of a user input device, that the user's gaze is focused on a portion of the presented text; or determining that the user's gaze is focused on a portion of the presented text while receiving data from a gesture tracking system indicating that the user has made a predefined command gesture substantially requesting editing; 57. The method of claim 56, comprising one or more of the following: (Item 58) receiving a selection of additional words within the displayed text based on at least data from the eye tracking system; providing the user with an opportunity to edit a portion of the presented text or a phrase formed from an additional portion of the text; 57. The method of claim 56, further comprising: (Item 59) 57. The method of claim 56, wherein at least a portion of the text is highlighted on the display, the portion being associated with a low confidence that a conversion of the spoken input to a corresponding portion of the text is correct. (Item 60) 1. A method comprising: receiving a multi-mode input, the multi-mode input comprising: a first user input from a hardware component of the wearable device, the first user input being associated with a first mode of user input; and a second user input, the second user input being associated with a second mode of user input; and identifying a multi-mode input command based on the first user input and the second user input, the multi-mode input command comprising at least: an object comprising at least a portion of a text; a command action comprising an interaction with the text; and executing the multi-modal input command and displaying to a user the results of the user multi-modal input; A method comprising: (Item 61) Item 61. The method of item 60, wherein the user input for the first mode comprises speech input received from an audio sensor of the wearable device, the method further comprising transcribing the speech input and identifying at least one of the text, the object, or the command action. (Item 62) Item 61. The method of item 60, wherein the user input of the second mode comprises input from at least one of a user input device, a gesture, or an eye gaze. (Item 63) Item 61. The method of item 60, wherein interacting with the text comprises at least one of selecting, editing, or composing the text. (Item 64) 61. The method of claim 60, wherein the object comprises one or more of a word, a phrase, or a sentence. (Item 65) The object comprises a sentence, and the command action comprises selecting the sentence for editing, the method further comprising: performing a first user interface action to cause the sentence to be extracted from the body of text; displaying the sentence as a sequence of tokens as a primary result, the primary result comprising the sentence as transcribed from an utterance of the user; further displaying secondary results, said secondary results comprising alternative words for tokens in said sequence; receiving another multi-modal input having a third input mode and a fourth input mode for editing the sequence word by word; responsive to an indication that editing of the sequence is complete, inserting the edited sequence back into the body of the text; Item 61. The method according to item 60, comprising:

[0007] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purport to define or limit the scope of the inventive subject matter. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0009] [Diagram 2] FIG. 2 illustrates diagrammatically an embodiment of a wearable system.

[0010] [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0011] [Figure 4] FIG. 4 diagrammatically illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0012] [Diagram 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0013] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal volumetric display, image, or light field.

[0014] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0015] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0016] [Figure 9] FIG. 9 is a block diagram of another embodiment of a wearable system.

[0017] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0018] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0019] [Figure 12A] FIG. 12A diagrammatically illustrates examples of a field of view (FOR), a world camera field of view (FOV), a user's field of view, and a user's fixation field.

[0020] [Figure 12B] FIG. 12B diagrammatically illustrates an example of a virtual object in a user's field of view and in the kinetic field of view.

[0021] [Figure 13] FIG. 13 illustrates an example of interacting with a virtual object using one mode of user input.

[0022] [Figure 14] FIG. 14 illustrates an example of selecting a virtual object using a combination of user input modes.

[0023] [Figure 15]FIG. 15 illustrates an example of using a combination of direct user inputs to interact with a virtual object.

[0024] [Figure 16] FIG. 16 illustrates an exemplary computing environment for aggregating input modes.

[0025] [Figure 17A] FIG. 17A illustrates an example of identifying a target virtual object using lattice tree analysis.

[0026] [Figure 17B] FIG. 17B illustrates an example of determining a target user interface action based on multi-modal input.

[0027] [Figure 17C] FIG. 17C illustrates an example of aggregating confidence scores associated with input modes for a virtual object.

[0028] [Figure 18] 18A and 18B illustrate an example of calculating confidence scores for objects within a user's FOV.

[0029] [Figure 19A] 19A and 19B illustrate an example of using multi-modal input to interact with the physical environment. [Figure 19B] 19A and 19B illustrate an example of using multi-modal input to interact with the physical environment.

[0030] [Figure 20] FIG. 20 illustrates an example of automatically resizing a virtual object based on multi-modal input.

[0031] [Figure 21]FIG. 21 illustrates an example of identifying a target virtual object based on the object's location.

[0032] [Figure 22] 22A and 22B illustrate another embodiment of interacting with a user's environment based on a combination of direct and indirect inputs.

[0033] [Diagram 23] FIG. 23 illustrates an example process for interacting with a virtual object using multi-modal input.

[0034] [Figure 24] FIG. 24 illustrates an example of setting a direct input mode associated with user interaction.

[0035] [Diagram 25] FIG. 25 illustrates an example of a user experience using multi-modal input.

[0036] [Figure 26] FIG. 26 illustrates an example user interface with various bookmarked applications.

[0037] [Figure 27] FIG. 27 illustrates an exemplary user interface when a search command is issued.

[0038] [Figure 28A] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input. [Figure 28B] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input. [Figure 28C] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input. [Figure 28D] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input. [Figure 28E] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input. [Figure 28F] 28A-28F illustrate an example user experience for composing and editing text based on a combination of voice and eye gaze input.

[0039] [Figure 29] FIG. 29 illustrates an example of selecting words based on input from a user input device and eye gaze.

[0040] [Diagram 30] FIG. 30 illustrates an example of selecting words for editing based on a combination of voice and eye gaze input.

[0041] [Diagram 31] FIG. 31 illustrates an example of selecting words for editing based on a combination of gaze and gesture input.

[0042] [Diagram 32] FIG. 32 illustrates an example of replacing words based on a combination of eye gaze and voice input.

[0043] [Diagram 33] FIG. 33 illustrates an example of modifying words based on a combination of voice and eye gaze input.

[0044] [Diagram 34] FIG. 34 illustrates an example of editing a selected word using a virtual keyboard.

[0045] [Diagram 35]FIG. 35 illustrates an exemplary user interface displaying possible actions to apply to a selected word.

[0046] [Diagram 36] FIG. 36 illustrates an example of interacting with a phrase using multimodal input.

[0047] [Figure 37A] 37A and 37B illustrate additional examples of interacting with text using multimodal input. [Figure 37B] 37A and 37B illustrate additional examples of interacting with text using multimodal input.

[0048] [Figure 38] FIG. 38 is a process flow diagram of an exemplary method for interacting with text using multiple modes of user input. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. overview

[0050] Modern computing systems can possess a variety of user interactions. Wearable devices can present interactive VR / AR / MR environments, which can comprise data elements that can be interacted with by a user through various inputs. Modern computing systems are typically engineered to generate a given output based on a single direct input. For example, a keyboard will relay text input as received from a user's keystrokes. A voice recognition application can create an executable data string based on the user's voice as direct input. A computer mouse can guide a cursor in response to a user's direct manipulation (e.g., a user's hand movements or gestures). The various ways in which a user may interact with a system are sometimes referred to herein as modes of user input. For example, user input via a mouse or keyboard is a hand gesture-based mode of interaction (because the fingers of a hand press keys on a keyboard or the hand moves a mouse).

[0051] However, in data-rich and dynamic interactive environments (e.g., AR / VR / MR environments), traditional input techniques such as keyboards, user input devices, gestures, etc., may require a high degree of specificity to accomplish a desired task. Otherwise, in the absence of precise input, a computing system may suffer from high error rates and cause incorrect computer actions to be performed. For example, when a user intends to move an object in 3D space using a touchpad, the computing system may be unable to correctly interpret the movement command if the user does not specify a destination or does not specify an object using the touchpad. As another example, entering a string of text using a virtual keyboard as the only input mode (e.g., as operated with a user input device or by gestures) may be slow and physically tiring, requiring long periods of fine motor control to type keys depicted in the air or on a physical surface (e.g., a desk) on which the virtual keyboard is rendered.

[0052] To reduce the degree of specificity required in input commands and to reduce error rates associated with imprecise commands, a wearable system as described herein can be programmed to apply multimodal input for the performance of an interaction event, such as a task to select, move, resize, or target a virtual object. The interaction event can include causing an application associated with the virtual object to execute (e.g., if the target object is a media player, the interaction event can include causing the media player to play a song or video). Selecting a target virtual object can include causing an application associated with the target virtual object to execute. Multimodal input utilizes two or more types of input (or input from multiple input channels) to generate a command for the performance of a task. As will be further described herein, input modes can include, but are not limited to, hand gestures, head poses, eye gaze, voice commands, environmental input (e.g., the user's position or objects in the user's environment), and the like. For example, when a user utters "move it there," the wearable system can use head pose, eye gaze, hand gestures, as well as other environmental factors (e.g., the user's location or the location of objects around the user), in combination with the voice command, to determine the object to be moved (e.g., the object that corresponds to "it") and the intended destination (e.g., "there") in response to these multi-modal inputs.

[0053] As will be further described herein, techniques for multimodal input are not simply an aggregation of multiple user input modes. Rather, wearable systems employing such techniques can advantageously support an additional depth dimension in 3D (compared to traditional 2D interactions) provided to the wearable system. The additional dimension not only enables additional types of user interaction (e.g., rotation, i.e., movement along additional axes in a Cartesian coordinate system), but also requires highly accurate user input to provide correct results. However, user input for interacting with virtual objects is not always accurate due to the user's limitations on motor control. While traditional input techniques can calibrate and adjust for inaccuracies of a user's motor control in 2D space, such inaccuracies are magnified in 3D space due to the additional dimension. However, traditional input methods, such as keyboard input, are less well suited to adjust for such inaccuracies in 3D space. Thus, one advantage (among other advantages) provided by multimodal input techniques is that they adapt traditional input methods for fluid and more accurate interaction with objects in 3D space.

[0054] Additionally, advantageously, in some embodiments, the techniques described herein can reduce hardware requirements and costs of wearable systems. For example, rather than employing a high-resolution eye-tracking camera alone (which can be expensive and complex to use) in conjunction with a low-resolution eye-tracking camera, the wearable device may use voice commands to perform the task and determine the task based on a multi-modal combination of the user's eye gaze direction and voice commands. In this example, the use of the user's voice commands can compensate for the lower resolution at which eye tracking is performed. Thus, the multi-modal combination of multiple user input modes can provide lower cost, lower complexity than the use of a single input mode, and more robust user interaction with the AR / VR / MR device. Additional advantages and examples of techniques related to multi-modal input for interacting with virtual objects are described with further reference to FIGS. 13-27.

[0055] Wearable systems can also advantageously support interaction with text using multimodal input control. As mentioned above, traditional input techniques employed individually are problematic in AR / VR / MR environments. For example, input using a user input device, gestures, or eye gaze (e.g., via interaction with a virtual keyboard) requires fine motor control and can therefore be slow and fatiguing. A virtual keyboard with character insertion operated by gaze has a relatively low limit in terms of the speed of text input (e.g., estimated at about 10-35 words / min). Spoken input is very fast (e.g., estimated at about 100-150 words / min), but can be prone to errors due to misrecognition or stuttering artifacts (e.g., for various reasons such as poor pronunciation, environmental noise, use of homonyms, use of idiosyncratic expressions, or simply uncommon vocabulary). Using unimodal input, text editing (eg, correcting errors, revising for other reasons) can be particularly difficult because, in the absence of a very precise set of commands, selections and substitutions are difficult.

[0056] Advantageously, in some embodiments, the wearable system described herein can facilitate text entry and editing in various systems, including mixed reality systems, by combining available input methods and enabling users to utilize a combination of user inputs to accomplish efficient interactions with text (e.g., composing, selecting, and editing text, etc.). As an example, speech may be used as the primary method of inputting text into the system. Speech can be combined with eye gaze as a method of selecting text for editing and revision, as well as the manipulation of graphical user interface elements in general. The wearable system can also enable the performance of any given task (e.g., replacing a particular text string with a different string) using a combination of interaction modalities (e.g., selection using gaze and replacement using speech).

[0057] Thus, as will be further described with reference to Figures 28A-38, the wearable system provides the user with the ability to compose text, messages using speech and edit such messages using gaze or another type of input (e.g., body gestures) distinct from speech. The wearable system may be configured, for example, to receive speech input (e.g., vocalization) or audio input, such as sounds from the environment, from the user, generate a transcript of the speech input (e.g., using automatic speech recognition (ASR)), present the user with a text representation of the generated transcript (e.g., text characters displayed in mixed reality), receive another type of input from the user (e.g., gaze input, gesture input, etc.), and select and edit portions of the transcript based on the other type of input received from the user.

[0058] By combining user input modalities in this way, the text composition and editing process can be faster and more intuitive, as speech input can be more effective than gaze input for composition (e.g., gaze typing can result in fatigue) and gaze input (or gesture input) can be more effective than speech input for editing (e.g., using speech to select or modify a transcribed message can be prone to error). Example of a 3D display for a wearable system

[0059] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. A wearable system can include a wearable device that can present VR, AR, or MR content, alone or in combination, within an environment for user interaction. The wearable device can be a head-mounted device (HMD) that can include a head-mounted display. In some circumstances, a wearable device is referred to synonymously as an AR device (ARD).

[0060] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0061] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0062] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple rendering planes are provided to a viewer. The rendering planes can correspond to a depth plane or multiple depth planes. The images may be different for each rendering plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different rendering planes, or based on observing different image features on different rendering planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0063] FIG. 2 illustrates an example of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can comprise a head-mounted display (HMD) worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display 220 can include an audio sensor 232 (e.g., a microphone) to detect audio streams from the environment to perform sound recognition.

[0064] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information obtained by the inwardly facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movements, or eye posture.

[0065] As an example, the wearable system 200 can obtain an image of the user's posture using the outwardly facing imaging system 464 or the inwardly facing imaging system 462. The image may be a still image, a frame of video, a video, a combination thereof, or the like.

[0066] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0067] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include (a) data captured from environmental sensors (e.g., which may be operatively coupled to the frame 230 or otherwise attached to the user 210), audio sensors 232 (e.g., microphones), or (b) data obtained or processed using the remote processing module 270 or remote data repository 280, possibly for passing to the display 220 after processing or reading. The local processing and data module 260 may be operatively coupled to the remote processing module 270 or remote data repository 280 over a communication link 262 or 264, such as via a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.

[0068] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process the data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0069] 2, the wearable system 200 may include environmental sensors to detect objects, stimuli, people, animals, places, or other aspects of the world around the user. The environmental sensors may include image capture devices (e.g., cameras, inwardly facing imaging systems, outwardly facing imaging systems, etc.), microphones, inertial measurement units (IMUs), accelerometers, magnetometers (compasses), global positioning system (GPS) units, wireless devices, gyroscopes, altimeters, barometers, chemical sensors, humidity sensors, temperature sensors, external microphones, light sensors (e.g., light meter), timing devices (e.g., clocks or calendars), or any combination or subcombination thereof. In an embodiment, the IMU may be a 9-axis IMU, which may include a triple-axis gyroscope, a triple-axis accelerometer, and a triple-axis magnetometer.

[0070] The environmental sensors may also include various physiological sensors. These sensors may measure or estimate the user's physiological parameters, such as heart rate, respiration rate, galvanic skin response, blood pressure, brainwave state, etc. The environmental sensors may further include an emitting device configured to receive signals, such as laser, visible light, light of invisible wavelengths, or sound (e.g., audible sound, ultrasound, or other frequencies). In some embodiments, one or more environmental sensors (e.g., cameras or light sensors) may be configured to measure the ambient light (e.g., luminance) of the environment (e.g., to capture the lighting conditions of the environment). Physical contact sensors, such as strain gauges, curb detectors, or the like, may also be included as environmental sensors.

[0071] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence shifts and accommodation. Vergence shifts of the two eyes relative to one another (e.g., pupil rotation such that the pupils move toward or away from one another, converging the gaze of the eyes to fixate on an object) are closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, a change in focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a matching change in vergence at the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. Display systems that provide better matching between accommodation and convergence-divergence movements may produce more realistic and comfortable simulations of three-dimensional images.

[0072] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple rendering planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume particular accommodated states that focus objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the rendering planes 306 having an associated focal length such that an object or part of an object in a particular rendering plane is in focus when the eye is in an accommodated state relative to that rendering plane. In some embodiments, a three-dimensional image may be simulated by providing a different presentation of an image for each eye 302 and 304, and by providing a different presentation of an image corresponding to each of the rendering planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, while shown to be flat for ease of illustration, it should be understood that the contours of the rendering plane may be curved in physical space such that all features within the rendering plane are in focus with the eye in a particular accommodative state. Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of rendering planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of rendering planes. Waveguide Stack Assembly

[0073] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2, of which FIG. 4 illustrates several portions in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of FIG. 2.

[0074] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0075] The waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular rendering plane and configured to output image information corresponding to that rendering plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide, outputting an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) corresponding to the rendering plane associated with the particular waveguide.

[0076] In some embodiments, the image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each of the corresponding waveguides 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of the image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0077] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may be part of the processing module 260 or 270 (illustrated in FIG. 2) in some embodiments.

[0078] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between their major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include a light extraction optical element 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide to where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of the drawings, in some embodiments the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surface, or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on a surface of and / or within that piece of material.

[0079] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular rendering plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 436b as originating from a second focal plane that is even closer inward toward the person from optical infinity, which was the light from the next upper waveguide 434b.

[0080] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0081] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to both redirect light from its respective waveguide and output this light with the appropriate amount of divergence or collimation for the particular rendering plane associated with the waveguide. As a result, waveguides with different associated rendering planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated rendering plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0082] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, i.e., "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency, such that only a portion of the light of the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, which can result in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0083] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0084] In some embodiments, the number and distribution of rendering planes or depth of field may be dynamically varied based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases such that one plane that is indistinguishable because its location exceeds the focal depth of the eye may become distinguishable and appear more in focus with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced apart rendering planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive details of both the first and second rendering planes at one pupil size without adjusting the accommodation of the eye from one rendering plane to the other. However, these two rendering planes may be sufficient to simultaneously focus the user at another pupil size without changing accommodation.

[0085] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of a pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two rendering planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between an on and off state, the DOE may be switched to the off state when the waveguide receives image information.

[0086] In some embodiments, it may be desirable to have the exit beam meet the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0087] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400, since the wearer can move their body, head, or eyes to perceive virtually any direction in space. In other contexts, the wearer's movements may be more constrained, and accordingly, the wearer's FOR may subtend a smaller solid angle. Images obtained from the outwardly facing imaging system 464 may be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0088] The wearable system 400 may also include an inwardly facing imaging system 462 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 462 may be used to capture images of the eye 410 and determine the size and / or orientation of the pupil of the eye 304. The inwardly facing imaging system 462 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 462 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0089] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., thumb) to press or swipe on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication.

[0090] 5 shows an example of an exit beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 480 may function similarly and that the waveguide assembly 480 includes multiple waveguides. Light 520 is launched into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light exits the waveguide as an exit beam 510. Although the exit beams 510 are shown as approximately parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., divergent exit beam formation) depending on the rendering plane associated with the waveguide 432b. It should be understood that a nearly collimated exit beam may refer to a waveguide with light extraction optical elements that outcouples light to form an image that appears to be set at a rendering plane at a large distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optical elements may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0091] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal volumetric display, image, or light field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal volumetric display, image, or light field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersive planar waveguide 622b. The dispersive planar waveguide 622b may be similar or the same in at least some respects as the primary planar waveguide 632b, which has a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or the same in at least some respects as the DOE 632a. For example, the dispersive planar waveguide 622b or the DOE 622a may be made of the same material as the primary planar waveguide 632b or the DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0092] The relayed and exit pupil expanded light may be optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b may relay the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is preferably orthogonal to the first axis. It should be noted that the second axis may be a non-orthogonal axis to the first axis. The primary planar waveguides 632b may expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b may relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 632b, which may relay and expand the light along the horizontal or X-axis.

[0093] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into a proximal end of a single mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be, for example, plated on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0094] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend the piezoelectric tube 642 independently in two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 644. By oscillating the piezoelectric tube 642 near the first mechanical resonant mode of the fiber cantilever 644, the fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0095] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned in two axial directions within an area filling a two-dimensional (2D) scan. By modulating the intensity of the light source 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0096] The optical coupler subsystem components can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into a narrow dispersive planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light can propagate perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b by TIR, and in so doing repeatedly intersect with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, and allows a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 622b via TIR.

[0097] At each point of intersection with DOE 622a, additional light can be diffracted towards the entrance of primary planar waveguide 632b. By splitting the incoming light into multiple outcoupled sets, the exit pupil of the light can be vertically expanded by DOE 4 in dispersive planar waveguide 622b. This vertically expanded light outcoupled from dispersive planar waveguide 622b can enter the edge of primary planar waveguide 632b.

[0098] Light entering the primary waveguide 632b can propagate horizontally (with respect to the view of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects with the DOE 632a at multiple points. The DOE 632a can be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, producing both deflection and focusing of the light. The DOE 632a can advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the light of the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0099] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles that match the designed focal level.

[0100] Thus, these different paths can couple light out of the primary planar waveguide 632b by resulting in different fill patterns at the exit pupil, focus levels, and / or multiplicities of the DOE 632a at different angles. Different fill patterns at the exit pupil can be advantageously used to generate light field displays with multiple rendering planes. Each layer or set of layers in a stack (e.g., three layers) in a waveguide assembly may be employed to generate individual colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image light fields with various focal depths. Other Components of the Wearable System

[0101] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile device or component may be operable to provide a sense of touch to the user. For example, the tactile device or component may provide a sense of pressure or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptics may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0102] A wearable system may include one or more physical objects that can be manipulated by, for example, a user to allow input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system or other users.

[0103] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. Exemplary Wearable Systems, Environments, and Interfaces

[0104] The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0105] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs, etc. from user input devices 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by the cameras (e.g., room cameras and / or cameras of the outward-facing imaging systems) may be reduced to a set of mapping points.

[0106] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.

[0107] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0108] Object recognition may be performed using a variety of computer vision techniques. For example, the wearable system may analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, image restoration, etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.

[0109] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to generate models for analysis of the aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0110] Based on this information and the set of points in the map database, the object recognizer 708a-708n may recognize objects, complement the objects with semantic information, and bring them to life. For example, if the object recognizer recognizes that the set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that the set of points is a mirror, the system may attach the semantic information that a mirror has a reflective surface that can reflect an image of an object in a room. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene generating in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, an object recognizer and other software components can map points collected from various images, recognize objects, etc. so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0111] 8 is a process flow diagram of an example embodiment of a method 800 for rendering virtual content in relation to recognized objects. Method 800 illustrates how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who is in California.

[0112] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a, 708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0113] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 comprises a map, which may include map data about the world. The map may reside in part locally on the wearable system, or in part in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. A pose process 910 (e.g., head or eye pose) may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The pose data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may comprise images, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information about objects in the real or virtual environment.

[0114] The sparse representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find not only the location of various components in the world, but also what the world is composed of. Poses can be building blocks that accomplish many goals, including capturing in and using data from maps.

[0115] In one embodiment, the sparse point locations may not be completely correct by themselves and further information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process called stereoscopic vision 940, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of the depth map information may be fused together, some of which may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., for large point clouds) and easy to summarize input to other processing devices such as a game engine. Thus, the output of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. Pose may be an input to this fusion process 930 as well, the output of which is input to the map capture process 920. Sub-surfaces may interconnect to form larger surfaces, such as in topographical mapping, where the map becomes a large-scale hybrid of points and surfaces.

[0116] Various inputs may be utilized to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs to determine that a user of the system is playing a monster battle game with one or more monsters at various locations, whether a monster is dead or fleeing under various conditions (such as when the user shoots the monster), walls or other objects at various locations, and the like. A world map may contain information regarding the location of such objects relative to one another, which is another useful input for mixed reality. Orientation relative to the world is an input as well, and plays an important role for nearly any interactive system.

[0117] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, and the like. To move around or play a game, for example, the user may need to command the wearable system 900 with respect to a desired object. There are various forms of user control that can be utilized beyond just moving around in space. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU that can help determine the situation occurring even when such activity is not within the field of view of any of the cameras).

[0118] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture left or right, stop, grab, hold, etc. for a button press. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or to "fist bump" with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as spreading the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0119] Eye tracking is another input (e.g., to track where the user is looking, control display technology, render at a specific depth or range). In one embodiment, eye vergence may be determined using triangulation, and then accommodation may be determined using a vergence / accommodation model developed for that particular person.

[0120] Voice recognition may be another input that may be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). The system 900 may include an audio sensor 232 (e.g., a microphone) that receives an audio stream from the environment. The received audio stream may be processed (e.g., by the processing modules 260, 270 or the central server 1650) to recognize the user's voice (from other voices or background audio) and extract commands, targets, parameters, etc. from the audio stream. For example, the system 900 may identify from the audio stream that the phrase "move it there" was uttered, identify that the phrase was uttered by the wearer of the system 900 (and not another person in the user's environment), and extract from the phrase an executable command ("move it") and that there is an object ("it") that is to be moved to a location ("there"). The object to be acted upon by the command may be referred to as the target of the command, and other information is provided to the command as parameters. In this example, the location where the object should be moved is a parameter for the "move" command. Parameters can include, for example, location, time, other objects to be interacted with (e.g., "move it next to the red chair" or "give Linda the magic wand"), how the command should be executed (e.g., "play a song using the speakers on the second floor"), etc.

[0121] As another example, the system 900 can process an audio stream using speech recognition techniques to input a string of text or modify the content of the text. The system 900 can incorporate speaker recognition techniques to determine who is speaking as well as speech recognition techniques to determine what is being said. Speech recognition techniques can include hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker diarization, decision trees, and dynamic time warping (DTW) techniques, either alone or in combination. Speech recognition techniques can also include anti-speaker techniques such as cohort models and world models. Spectral features may be used in representing speaker characteristics.

[0122] With respect to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras, namely, a relative wide FOV or passive SLAM pair of cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and also capture hand gestures and totem / object trajectories in front of the user's face. The FOV cameras or pair of cameras for the stereoscopic process 940 may also be referred to as cameras 16. The FOV cameras and pair of cameras for the stereoscopic process 940 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include an eye tracking camera (which may be part of the inward facing imaging system 462 shown in FIG. 4) oriented toward the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0123] FIG. 10 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through an outward facing imaging system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0124] Based at least in part on the detected gestures, eye poses, head poses, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame at block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. At block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the reference frame 1020, the system determines the user input at block 1040.

[0125] For example, a user may move a totem or physical object back and forth, turn a virtual page, indicate moving to the next page, or move from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as the user input. In some implementations, the vergence-divergence movement of the user's eyes can be tracked and an accommodation / vergence-divergence movement model can be used to determine the accommodation state of the user's eyes, which provides information about the rendering plane the user is focused on. In some implementations, the wearable system can use cone casting techniques to determine real or virtual objects that are in line with the direction of the user's head pose or eye pose. The cone casting technique, as generally described, projects an invisible cone in the direction the user is looking and can identify any objects that interact with the cone. Cone casting can include casting a thin bundle of light rays with virtually no lateral width or a light ray with substantial lateral width (e.g., a cone or a truncated cone) from an AR display (of a wearable system) toward a physical or virtual object. Cone casting with a single light ray may also be referred to as ray casting. Detailed examples of cone casting techniques are described in U.S. Application No. 15 / 473,444, entitled "Interactions with 3D Virtual Objects Using Poses and Multiple-DOF Controllers," filed March 29, 2017, the disclosure of which is incorporated herein by reference in its entirety.

[0126] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2), or may be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras outside or part of the system (e.g., using inward-facing imaging system 462 or outward-facing imaging system 464).

[0127] 11 is a process flow diagram of an example of a method 1100 for interacting with a virtual user interface. The method 1100 may be performed by a wearable system described herein.

[0128] In block 1110, the wearable system may identify a particular UI. The type of UI may be provided by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for the virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI may be displayed around the user, or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.

[0129] In block 1130, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for commands from the user and generate more virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.) and if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). As an example, the wearable system may wait for a user's hand gesture before mixing multiple stream tracks.

[0130] Additional examples of wearable systems, UI, and user experience (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. Example of objects in the field of view (FOR) and field of view (FOV)

[0131] FIG. 12A diagrammatically illustrates an example of a kinetic field of view (FOR) 1200, a world camera field of view (FOV) 1270, a user's field of view 1250, and a user's fixation field of view 1290. As described with reference to FIG. 4, the FOR 1200 comprises a portion of the user's surrounding environment that can be perceived by the user via the wearable system. The FOR may encompass a solid angle of 4π steradians surrounding the wearable system, since the wearer can move their body, head, or eyes and perceive virtually any direction in space. In other contexts, the wearer's movement may be more constrained, and thus the wearer's FOR may cover a smaller solid angle.

[0132] The field of view of the world camera 1270 may include the portion of the user's FOR currently being observed by the outwardly facing imaging system 464. With reference to FIG. 4, the field of view of the world camera 1270 may include the world 470 as observed by the wearable system 400 at a given time. The size of the FOV of the world camera 1270 may depend on the optical properties of the outwardly facing imaging system 464. For example, the outwardly facing imaging system 464 may include a wide-angle camera that may image a 190 degree space around the user. In some implementations, the FOV of the world camera 1270 may be greater than or equal to the natural FOV of the user's eyes.

[0133] The user's FOV 1250 may comprise the portion of the FOR 1200 that the user perceives at a given time. The FOV may depend on the size or optical properties of the wearable device's display. For example, the AR / MR display may include optics that provide AR / MR functionality when the user looks through a particular portion of the display. The FOV 1250 may correspond to a solid angle perceivable by the user when looking through an AR / MR display, such as, for example, stacked waveguide assembly 480 (FIG. 4) or planar waveguide 600 (FIG. 6). In an embodiment, the user's FOV 1250 may be smaller than the natural FOV of the user's eyes.

[0134] The wearable system can also determine the user's fixation field 1290. The fixation field 1290 can include a portion of the FOV 1250 on which the user's eye may fixate (e.g., maintain visual line of sight to that portion). The fixation field 1290 may correspond to the foveal region of the eye where light strikes. The fixation field 1290 can be smaller than the user's FOV 1250, e.g., the fixation field can be a few degrees to about 5 degrees across it. As a result, the user can perceive some virtual objects within the FOV 1250 that are not within the fixation field 1290 but are within the user's peripheral vision.

[0135] FIG. 12B diagrammatically illustrates an example of a virtual object in a user's field of view (FOV) and a virtual object in a kinetic field of view (FOR). In FIG. 12B, the FOR 1200 can contain a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244), which can be perceived by the user via the wearable system. The objects in the user's FOR 1200 can be virtual and / or physical objects. For example, the user's FOR 1200 can include physical objects such as a chair, a sofa, a wall, etc. The virtual objects can include operating system objects such as, for example, a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, text, text editing applications, messaging applications, etc. The virtual objects can also include objects within an application, such as, for example, an avatar, a virtual object within a game, a graphic, or an image. Some virtual objects can be both operating system objects and objects within an application. In some embodiments, the wearable system can add virtual elements to existing physical objects, for example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may give the user options to turn on the television or change the channel using the wearable system.

[0136] A virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, a virtual object may be a 3D coffee mug (which may represent a virtual control for a physical coffee maker). A virtual object may also be a 2D graphical representation of a clock (displaying the current time to the user). In some implementations, one or more virtual objects may be displayed within (or associated with) another virtual object. A virtual coffee mug may be shown inside the user's interface plane, but the virtual coffee mug appears to be 3D within this 2D planar virtual space.

[0137] The objects in the user's FOR can be part of a world map, as described with reference to FIG. 9. Data associated with the objects (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as, for example, arrays, lists, trees, hashes, graphs, etc. An index for each stored object may be determined, for example, by the location of the object, if applicable. For example, the data structure may index the objects by a single coordinate, such as the distance of the object from a base position (e.g., distance to the left or right of the base position, distance from above or below the base position, or distance per depth from the base position). The base position may be determined based on the user's position (such as the position of the user's head). The base position may also be determined based on the position of a virtual or physical object (such as a target object) in the user's environment. Thus, the 3D space in the user's environment may be represented in a 2D user interface, with the virtual objects arranged according to the object's distance from the base position.

[0138] In FIG. 12B, FOV 1250 is diagrammatically illustrated by dashed line 1252. A user of the wearable system can perceive multiple objects within FOV 1250, such as object 1242, object 1244, and a portion of object 1230. As the user's posture (e.g., head posture or eye posture) changes, FOV 1250 changes correspondingly, and objects within FOV 1250 may also change. For example, map 1210 is initially outside the user's FOV in FIG. 12B. When the user turns his or her eyes to map 1210, map 1210 may move into the user's FOV 1250 and object 1230 (for example) may move out of the user's FOV 1250.

[0139] The wearable system may track objects in the FOR 1200 as well as objects in the FOV 1250. For example, the local processing and data module 260 may communicate with the remote processing module 270 and the remote data repository 280 to retrieve virtual objects in the user's FOR. The local processing and data module 260 may store the virtual objects, for example, in a buffer or temporary storage device. The local processing and data module 260 may use techniques described herein to determine the user's FOV and render a subset of the virtual objects in the user's FOV. As the user's pose changes, the local processing and data module 260 may update the user's FOV and render a different set of virtual objects corresponding to the user's current FOV as appropriate. Overview of the different user input modes

[0140] A wearable system can be programmed to receive various input modes to perform operations. For example, a wearable system can receive two or more of the following types of input modes: voice commands, head pose, body pose (e.g., may be measured by an IMU in a beltpack or a sensor external to the HMD), eye gaze (also referred to herein as eye pose), hand gestures (or gestures by other body parts), signals from a user input device (e.g., a totem), environmental sensors, etc. Computing devices are typically engineered to generate a given output based on a single input from a user. For example, a user can enter a text message by typing on a keyboard or use a mouse to guide the movement of a virtual object, which is an example of a hand gesture input mode. As another example, a computing device can receive a stream of audio data from a user's voice and use speech recognition techniques to convert the audio data into executable commands.

[0141] User input modes may be non-exclusively classified as direct user input or indirect user input in some cases. Direct user input may be user interaction directly provided by the user via, for example, volitional movements of the user's body (e.g., turning head or eyes, gazing at an object or location, uttering a phrase, moving fingers or hands). As an example of direct user input, the user may interact with a virtual object using postures such as head posture, eye posture (also referred to as eye gaze), hand gestures, or another body posture. For example, the user may look at a virtual object (with head and / or eyes). Another example of direct user input is the user's voice. For example, the user may utter "launch browser" to cause the HMD to open a browser application. As yet another example of direct user input, the user may actuate a user input device, for example, via touch gestures (such as touching a touch-sensitive portion of a totem) or body movements (such as rotating a totem that functions as a multi-degree-of-freedom controller).

[0142] In addition to or as an alternative to direct user input, a user can also interact with virtual objects based on indirect user input. Indirect user input may be determined from various contextual factors, such as, for example, the geographic location of the user or virtual object, the user's environment, etc. For example, the user's geographic location may be in the user's office (rather than the user's home), and different tasks (e.g., work-related tasks) can be performed based on the geographic location (e.g., derived from a GPS sensor).

[0143] Contextual factors may also include affordances of virtual objects. Affordances of virtual objects may comprise a relationship between a virtual object and the object's environment that provides an opportunity for an action or use associated with the object. Affordances may be determined based on, for example, the function, orientation, type, location, shape, and / or size of the object. Affordances may also be based on the environment in which the virtual object is located. As an example, the affordance of a horizontal table is that the object can be set on the table, and the affordance of a vertical wall is that the object can be hung from or projected onto the wall. As an example, "put it there" may be issued and a virtual office calendar is set to appear horizontally on the user's desk in the user's office.

[0144] A single mode of direct user input may create various limitations, and the number or types of available user interface actions may be limited due to the type of user input. For example, a user may not be able to zoom in or out using head pose because the head pose may not be able to provide precise user interaction. As another example, a user may need to move their thumb back and forth across the touchpad (or move their thumb over a long distance) to move a virtual object from the floor to the wall, which may cause fatigue to the user over time.

[0145] However, some direct input modes may be more convenient and intuitive for users to provide. For example, a user may speak to the wearable system and issue voice commands without having to type sentences using gesture-based keyboard input. As another example, a user may use hand gestures to point to a target virtual object rather than moving a cursor and identifying the target virtual object. Although they may not be as convenient or intuitive, other direct input modes may increase the accuracy of user interaction. For example, a user may move a cursor to a virtual object and indicate that the virtual object is the target object. However, as described above, if a user wishes to select the same virtual object using direct user input (e.g., head pose, or other input that dictates the outcome of the user's action), the user may need to control precise movements of the head, which may cause muscle fatigue. 3D environments (e.g., VR / AR / MR environments) may add additional challenges to user interaction, since user input would also need to be defined with respect to depth (as opposed to a planar surface). This additional depth dimension can create more opportunities for error than a 2D environment. For example, in a 2D environment, user input can be transformed relative to the horizontal and vertical axes in a coordinate system, whereas in a 3D environment, user input may need to be transformed relative to three axes (horizontal, vertical, and depth). Thus, imprecise translation of user input may cause errors in three axes (rather than just two axes in a 2D environment).

[0146] To improve accuracy and reduce user fatigue in interacting with objects in 3D space while leveraging existing advantages of direct user input, multiple modes of direct input may be used to perform user interface operations. Multimodal input may further enhance existing computing devices (particularly wearable devices) for interacting with virtual objects in data-rich and dynamic environments, such as AR, VR, or MR environments.

[0147] In a multimodal user input technique, one or more of the direct inputs may be used to identify a target virtual object (also referred to as a target) with which the user will interact and to determine a user interface action to be performed on the target virtual object. For example, the user interface action may include a command action such as select, move, zoom, pause, play, etc., and parameters of the command action (e.g., how to perform the action, where or when the action will occur, the object with which the target object will interact, etc.). As an example of identifying a target virtual object and determining an interaction to be performed on the target virtual object, a user may look at a virtual sticky note (head or eye posture input mode), point to a table (gesture input mode), and utter "move it there" (speech input mode). The wearable system may identify that the target virtual object in the phrase "move it there" is a virtual sticky note ("it") and may determine that the user interface action involves moving the virtual sticky note to the table ("there") (an executable command). In this example, the command action may be to "move" the virtual object, while the parameters of the command action may include the destination object, which is the table the user is pointing to. Advantageously, in an embodiment, the wearable system may increase the overall accuracy of the user interface action or increase the convenience of the user's interaction by implementing the user interface action based on multiple modes of direct user input (e.g., in the above example, three modes: head / eye posture, gesture, and voice). For example, instead of uttering "move the leftmost browser 2.5 feet to the right," the user may utter "move it there" (without pointing to the object being moved in the speech input) while using a head or hand gesture to indicate that the object is the leftmost browser, and using a head or hand movement to indicate the distance of the move. Exemplary interactions in a virtual environment using different input modes

[0148] FIG. 13 illustrates an example of interacting with virtual objects using one mode of user input. In FIG. 13, a user 1310 is wearing an HMD and interacting with virtual content in three scenes 1300a, 1300b, and 1300c. The user's head position (and corresponding eye gaze direction) is represented by a geometric cone 1312a. In this example, the user can perceive the virtual content through the display 220 of the HMD. While interacting with the HMD, the user can input a text message through the user input device 466. In the scene 1300a, the user's head is in its natural resting position 1312a and the user's hands are also in their natural resting position 1316a. However, although the user may be more comfortable typing text on the user input device 466, the user cannot see the interface on the user input device 466 to ensure that the characters are typed correctly.

[0149] To see the text being typed on the user input device, the user can move their hands to position 1316b, as shown in scene 1300b. Thus, the hands will be within the FOV of the user's head when the head is in its natural resting position 1312a. However, position 1316b is not a natural resting position for the hands and may result in fatigue for the user. Alternatively, as shown in scene 1300c, the user can move their head to position 1312c to keep the hands in the natural resting position 1316a. However, the muscles around the user's neck may fatigue due to the unnatural position of the head and the user's FOV is directed towards the ground or floor rather than towards the outside world (which may be unsafe if the user is walking in a crowded area). In either scene 1300b or scene 1300c, the user's natural ergonomics are sacrificed to meet the desired user interface operation when the user performs the user interface operation using the single input mode.

[0150] The wearable systems described herein can at least partially mitigate the ergonomic limitations depicted in scenes 1300b and 1300c. For example, a virtual interface can be projected into the user's field of view in scene 1300a. The virtual interface can allow the user to observe input being typed from a natural position.

[0151] The wearable system can also display and support interaction with virtual content without device constraints. For example, the wearable system can present multiple types of virtual content to the user, and the user can use the touchpad to interact with one type of content while using the keyboard to interact with another type of content. Advantageously, in some embodiments, the wearable system can determine the virtual content (that the user is intended to act on) that is the target virtual object by calculating a confidence score (where a higher confidence score indicates a higher confidence (or likelihood) that the system has identified the correct target virtual object). A detailed example of identifying the target virtual object is described with reference to FIGS. 15-18B.

[0152] 14 illustrates an example of selecting a virtual object using a combination of user input modes. In scene 1400a, the wearable system can present a user 1410 with multiple virtual objects, represented by a square 1422, a circle 1424, and a triangle 1426.

[0153] A user 1410 can interact with a virtual object using a head pose, as illustrated in scene 1400b. This is an example of a head pose input mode. The head pose input mode may involve cone casting to target or select a virtual object. For example, the wearable system can cast a cone 1430 from the user's head toward the virtual object. The wearable system can detect if one or more of the virtual objects fall within the volume of the cone and identify the object that the user intends to select. In this example, the cone 1430 intersects with the circle 1424 and the triangle 1426. Thus, the wearable system can determine that the user intends to select either the circle 1424 or the triangle 1426. However, because the cone 1430 intersects with both the circle 1424 and the triangle 1426, the wearable system may not be able to ascertain whether the target virtual object is the circle 1424 or the triangle 1426 based on the head pose input alone.

[0154] In scene 1400c, a user 1410 can interact with a virtual object by manually orienting a user input device 466, such as a totem (e.g., a handheld remote control device). This is an example of a gesture input mode. In this scene, the wearable system can determine that either the circle 1424 or the square 1422 is the intended target because these two objects are in the direction that the user input device 466 is pointing. In this example, the wearable system can determine the direction of the user input device 466 by detecting the position or orientation of the user input device 466 (e.g., via an IMU in the user input device 466) or by performing cone casting originating from the user input device 466. Because both the circle 1424 and the square 1422 are candidates for the target virtual object, the wearable system cannot be sure, based on the gesture input mode alone, which of them is the object that the user actually wants to select.

[0155] In scene 1400d, the wearable system can determine a target virtual object using multi-modal user input. For example, the wearable system can identify a target virtual object using both results obtained from cone casting (head pose input mode) and the orientation of a user input device (gesture input mode). In this example, a circle 1424 is an identified candidate in both the results from cone casting and the results obtained from the user input device. Thus, the wearable system can determine with high confidence that the target virtual object is the circle 1424 using these two input modes. As further illustrated in scene 1400d, the user can give a voice command 1442 (illustrated as "move it"), which is an example of a third input mode (i.e., voice), to interact with the target virtual object. The wearable system can associate the word "it" with the target virtual object and the word "move" with the command to be performed, thus moving the circle 1424. However, the voice command 1442 by itself (without any indication from the user input device 466 or cone casting 143) may cause confusion to the wearable system because the wearable system may not know the object associated with the word "it."

[0156] Advantageously, in some embodiments, by receiving multiple input modes and identifying and interacting with a virtual object, the amount of precision required per input mode may be reduced. For example, cone casting may not be able to accurately indicate objects in a rendering plane that are far away because the diameter of the cone increases as the cone moves farther away from the user. As another example, the user may need to hold the input device in a particular orientation, point toward the target object, and speak with a particular phrase or pace to ensure correct voice input. However, by combining results from voice input and cone casting (either from head pose or gestures with the input device), the wearable system can still identify the target virtual object without requiring either of the inputs (e.g., cone casting or voice input) to be precise. For example, even if cone casting selects multiple objects (e.g., as described with reference to scenes 1400b, 1400c), voice input may help narrow down the selection (e.g., increase the confidence score for the selection). For example, cone casting may capture three objects, of which a first object is to the user's right, a second object is to the user's left, and a third object is in the center of the user's FOV. The user can narrow the selection by uttering "select the rightmost object." As another example, two identically shaped objects may be present within the user's FOV. In order for the user to select the correct object, the user may need to provide further description of the object via voice command. For example, instead of uttering "select the square object," the user may need to utter "select the red square object." However, with cone casting, the voice command may not need to be precise. For example, the user may look at one of the square objects and utter "select the square object" or even "select the object."The wearable system can automatically select square objects that are aligned with the user's gaze direction, and will not select square objects that are not in the user's gaze direction.

[0157] In some embodiments, the system may have a hierarchy of preferences for input mode combinations. For example, a user tends to look in the direction their head is facing. Thus, eye gaze and head pose may provide similar information to each other. Combinations of head pose and eye gaze may be less preferred because the combination does not provide much redundant information compared to using eye gaze alone or head pose alone. Thus, the system may use a hierarchy of modal input preferences for selecting modal inputs that generally provide contrast information rather than redundant information. In some embodiments, the hierarchy is one in which head pose and speech are used as primary modal inputs, followed by eye gaze and gestures.

[0158] Thus, as described further herein, based on the multi-modal input, the system can calculate confidence scores for various objects in the user's environment, with each such object being a target object, and the system can select the particular object in the environment having the highest confidence score as the target object.

[0159] FIG. 15 illustrates an example of interacting with a virtual object using a combination of direct user input. As depicted in FIG. 15, a user 1510 is wearing an HMD 1502 configured to display virtual content. The HMD 1502 may be part of the wearable system 200 described herein and may include a belt-mounted power and processing pack 1503. The HMD 1502 may be configured to receive user input from a totem 1516. The user 1510 of the HMD 1502 may have a first FOV 1514. The user may observe a virtual object 1512 within the first FOV 1514.

[0160] A user 1510 can interact with a virtual object 1512 based on a combination of direct inputs. For example, a user 1510 can select a virtual object 1512 through a cone casting technique based on the user's head or eye pose, or by a totem 1516, by a voice command, or by a combination of these (or other) input modes (e.g., as described with reference to FIG. 14 ).

[0161] The user 1510 may shift his / her head pose to move the selected virtual object 1512. For example, the user may turn his / her head to the left, updating the FOV from the first FOV 1514 to the second FOV 1524 (as shown in scene 1500a to scene 1500b). The user's head movement may be combined with other direct inputs to move the virtual object from the first FOV 1514 to the second FOV 1524. For example, the head pose change may be aggregated with other inputs, such as, for example, a voice command ("move it there"), guidance from the totem 1516, or eye gaze direction (e.g., as recorded by the inward-facing imaging system 462 shown in FIG. 4). In this example, the HMD 1502 may use the updated FOV 1524 as a general area into which the virtual object 1512 should be moved. The HMD 1502 can further determine the destination of the movement of the virtual object 1512 based on the user's gaze direction. As another example, the HMD may capture a voice command "move it there." The HMD can identify the virtual object 1512 as the object with which the user will interact (because the user has pre-selected the virtual object 1512). The HMD can further determine that the user intends to move the object from the FOV 1514 to the FOV 1524 by detecting a change in the user's head pose. In this example, the virtual object 1512 may initially be in a central portion of the user's first FOV 1514. Based on the voice command and the user's head pose, the HMD may move the virtual object to the center of the user's second FOV 1524. Example embodiments using multi-modal user input to identify a target virtual object or user interface action

[0162] 14, in some circumstances, the wearable system may not be able to identify (with sufficient confidence) the target virtual object with which the user intends to interact using a single input mode. Furthermore, even when multiple modes of user input are used, user input in one mode may indicate one virtual object, while user input in another mode may indicate a different virtual object.

[0163] To provide an improved wearable system that resolves ambiguities and supports multi-modal user input, the wearable system can aggregate the modal user inputs and calculate a confidence score to identify the desired virtual object or user interface action. As discussed above, a higher confidence score indicates a higher probability or likelihood that the system has identified the desired target object.

[0164] FIG. 16 illustrates an exemplary computing environment for aggregating input modes. The exemplary environment 1600 includes three virtual objects associated with applications A 1672, B 1674, and C 1676, for example. As described with reference to FIG. 9, the wearable system can include various sensors, receive various user inputs from these sensors, analyze the user inputs, and interact with the mixed reality 960. In the exemplary environment 1600, a central runtime server 1650 can aggregate direct inputs 1610 and indirect user inputs 1630 and produce multimodal interactions for the applications. Examples of direct inputs 1610 may include gestures 1612, head poses 1614, voice inputs 1618, totems 1622, eye gaze direction (e.g., eye gaze tracking 1624), other types of direct inputs 1626, and the like. Examples of indirect inputs 1630 may include environmental information (e.g., environmental tracking 1632), and geographic location 1634. The central runtime server 1650 may include a remote processing module 270. In some implementations, the local processing and data module 260 may perform one or more functions of the central runtime server 1650. The local processing and data module 260 may also communicate with the remote processing module 270 and aggregate input modes.

[0165] The wearable system can track gestures 1612 using an outwardly facing imaging system 464. The wearable system can track hand gestures using various techniques described in FIG. 9. For example, the outwardly facing imaging system 464 can obtain images of the user's hands and map the images to corresponding hand gestures. The outwardly facing imaging system 464 may image the user's hand gestures using a FOV camera or a depth camera (configured for depth detection). The central runtime server 1650 can identify the user's head gestures using the object recognizer 708. The gestures 1612 can also be tracked by the user input device 466. For example, the user input device 466 may include a touch-sensitive surface, which can track the user's hand movements, such as, for example, swipe gestures or tap gestures.

[0166] The HMD can use the IMU to recognize head pose 1614. The head 1410 may have multiple degrees of freedom, including three types of rotation (e.g., yaw, pitch, and roll) and three types of translation (e.g., pitch, sway, and heave). The IMU can be configured to measure, for example, 3-DOF or 6-DOF movement of the head. Measurements obtained from the IMU may be communicated to a central runtime server 1650 for processing (e.g., to identify head pose).

[0167] The wearable system can implement eye gaze tracking 1624 using an inwardly facing imaging system 462. For example, the inwardly facing imaging system 462 can include an eye camera configured to capture images of the user's eye region. The central runtime server 1650 can analyze the images (e.g., via the object recognizer 708) and infer the user's gaze direction or track the user's eye movements.

[0168] The wearable system can also receive input from a totem 1622. As described herein, the totem 1622 can be an embodiment of a user input device 466. Additionally or alternatively, the wearable system can receive a voice input 1618 from the user. The input from the totem 1622 and the voice input 1618 can be communicated to a central runtime server 1650. The central runtime server 1650 can use natural language processing in real-time or near real-time to analyze the user's audio data (e.g., obtained from the microphone 232). The central runtime server 1650 can identify the content of the speech by applying various speech recognition algorithms such as, for example, hidden Markov models, dynamic time warping (DTW) based speech recognition, deep learning algorithms such as neural networks, deep feedforward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms (described with reference to FIGS. 7 and 9), other algorithms using semantic analysis, acoustic modeling or language modeling, etc. The central runtime server 1650 can also apply speech recognition algorithms, which can identify the identity of the speaker, such as whether the speaker is the user of the wearable device or a person within the user's background.

[0169] The central runtime server 1650 can also receive indirect input when the user interacts with the HMD. The HMD can include various environmental sensors described with reference to FIG. 2. Using data obtained by the environmental sensors (either alone or in combination with data related to the direct input 1610), the central runtime server 1650 can reconstruct or update the user's environment (e.g., map 920, etc.). For example, the central runtime server 1650 can determine the user's ambient light conditions based on the user's environment. The ambient light conditions can be used to determine a virtual object with which the user can interact. For example, when the user is in a bright environment, the central runtime server 1650 may identify that the target virtual object is a virtual object that supports gestures 1612 as an input mode because the camera can observe the user's gestures 1612. However, if the environment is dark, the central runtime server 1650 may determine that the virtual object can be an object that supports voice input 1618 rather than gestures 1612.

[0170] The central runtime server 1650 can perform environment tracking 1632, aggregate direct input modes, and produce multi-modal interaction for multiple applications. As an example, when a user moves from a quiet environment into a noisy environment, the central runtime server 1650 may disable voice input 1618. Additional examples of selecting an input mode based on the environment are described with further reference to FIG.

[0171] The central runtime server 1650 may also identify target virtual objects based on the user's geographic location information. The geographic location information 1634 may also be obtained from environmental sensors (e.g., GPS sensors, etc.). The central runtime server 1650 may identify virtual objects for potential user interaction where the distance between the virtual object and the user is within a threshold distance. Advantageously, in some embodiments, the cone in the cone casting may have a length that is adjustable by the system (e.g., based on the number or density of objects in the environment). By selecting objects within a certain radius of the user, the number of potential objects that may be target objects may be significantly reduced. Additional examples of using indirect input as an input mode are described with reference to FIG. 21. Example of identifying a target object

[0172] The central runtime server 1650 can determine the target object using various techniques. FIG. 17A illustrates an example of identifying a target object using lattice tree analysis. The central runtime server 1650 can derive given values ​​from input sources and produce a lattice of possible values ​​for candidate virtual objects with which a user may potentially interact. In some embodiments, the value can be a confidence score. The confidence score can include a ranking, rating, evaluation, quantitative or qualitative value (e.g., a number in the range of 1-10, a percentage or percentile, or qualitative values ​​of "A", "B", "C", etc.), etc. Each candidate object can be associated with a confidence score, and in some cases, the candidate object with the highest confidence score (e.g., higher than the confidence scores of other objects or higher than a threshold score) is selected by the system as the target object. In other cases, objects with confidence scores below a threshold confidence score are eliminated by the system from consideration as target objects, which can improve computational efficiency.

[0173] Many examples herein refer to the selection of a target virtual object or a selection from a group of virtual objects. This is intended to illustrate example implementations and is not intended to be limiting. The described techniques can be applied to virtual objects or physical objects in the user's environment. For example, the voice command "move it there" may refer to moving a virtual object (e.g., a virtual calendar) onto a physical object (e.g., a horizontal surface of a user's desk). Or the voice command "move it there" may refer to moving a virtual object (e.g., a virtual word processing application) to another location within another virtual object (e.g., another location within the user's virtual desktop).

[0174] The context of the command may also provide information about whether the system should attempt to identify a virtual object, a physical object, or both. For example, in the command "move it there," the system may recognize that "it" is a virtual object because AR / VR / MR systems cannot actually move physical objects. Thus, the system may eliminate physical objects as candidates for "it." As described in the above example, the target location "there" may be a virtual object (e.g., the user's virtual desktop) or a physical object (e.g., the user's desk).

[0175] The system may also assign confidence scores to objects in the user's environment, which may be in the FOR, FOV, or field of fixation (e.g., see FIG. 12A), depending on the context and the system's goal at the time. For example, a user may want to move a virtual calendar to a location on the user's desk, both of which are in the user's FOV. The system may analyze objects in the user's FOV rather than all objects in the user's FOR, because the context of the situation suggests that a command to move the virtual calendar would be a target destination in the user's FOV, which may improve processing speed or efficiency. In another case, a user may be browsing a menu of movie selections in a virtual movie application and may fixate on some movie choices. The system may analyze (and, for example, provide confidence scores for) only the movie selections in the user's field of fixation (e.g., based on the user's eye line of sight), rather than the entire FOV (or FOR), which may also increase processing efficiency or speed.

[0176] Referring to the example shown in FIG. 17A, a user can interact with the virtual environment using two input modes: head pose 1614 and eye gaze 1624. Based on the head pose 1614, the central runtime server 1650 can identify two candidate virtual objects associated with application A 1672 and application B 1674. The central runtime server 1650 can distribute the 100% confidence score evenly between application A 1672 and application B 1674. As a result, application A 1672 and application B 1674 may each be assigned a confidence score of 50%. The central runtime server 1650 can also identify two candidate virtual objects (application A 1672 and application C 1676) based on the direction of eye gaze 1624. The central runtime server 1650 can also split the 100% confidence between application A 1672 and application C 1676.

[0177] The central runtime server 1650 may implement a lattice compression logic function 1712 to reduce or eliminate mismatched confidence values ​​or those confidence values ​​below a certain threshold that are not common among multiple input modes, and determine the application with which the user is most likely intended to interact. For example, in FIG. 17A, the central runtime server 1650 may eliminate application B 1674 and application C 1676 because these two virtual objects are not identified by both head pose 1614 and eye gaze 1624 analysis. As another example, the central runtime server 1650 may aggregate the values ​​assigned to each application. The central runtime server 1650 may set a threshold confidence value equal to or greater than 80%. In this example, the aggregated value of application A 1672 is 100% (50%+50%), the aggregated value of application B 1674 is 50%, and the value of application C 1676 is 50%. Because the individual trust values ​​for applications B and C are below the threshold trust value, the central runtime server 1650 may be programmed to select application A 1672 rather than selecting applications B and C because the aggregate trust value of application A (100%) is above the threshold trust value.

[0178] 17A divides the values ​​associated with the input devices (e.g., confidence scores) equally among the candidate virtual objects, in an embodiment, the value distribution may not be equal among the candidate virtual objects. For example, if head pose 1614 has a value of 10, application A 1672 may receive a value of 7 and application B 1674 may receive a value of 3 (because the head pose is pointing more toward A 1672). As another example, if head pose 1614 has a qualitative grade of "A," application A 1672 may be assigned a grade of "A," while applications B 1674 and C 1676 receive nothing from head pose 1614.

[0179] The wearable system (e.g., the central runtime server 1650) can assign a focus indicator to the target virtual object so that the user may more easily perceive the target virtual object. The focus indicator can be a visual focus indicator. For example, the focus indicator can comprise a halo (substantially surrounding or near the object), a color, a perceived size or depth change (e.g., making the target object appear closer and / or larger when selected), or other visual effect that attracts the user's attention. The focus indicator can also include an audible or tactile effect, such as a vibration, ringing, beep, etc. The focus indicator can provide the user with useful feedback that the system is "working correctly" by confirming (via the focus indicator) to the user that the system has correctly determined the object associated with the command (e.g., correctly determined the "it" and "there" in the "move it there" command). For example, the identified target virtual object can be assigned a first focus indicator and the destination location (e.g., the "there" in the command) can be assigned a second focus indicator. In some cases, if the system incorrectly determines the target object, the user may override the system's determination, for example, by gazing (fixating) on ​​the correct object and providing a voice command such as "not that." Example of identifying a target user interface action

[0180] In addition to, or as an alternative to, identifying a target virtual object, the central runtime server 1650 can also determine a target user interface action based on multiple inputs received. FIG. 17B illustrates an example of determining a target user interface action based on a multimodal input. As depicted, the central runtime server 1650 can receive multiple inputs in the form of head poses 1614 and gestures 1612. The central runtime server 1650 can display multiple virtual objects associated with application A 1672 and application B 1674 to the user, for example. However, the head pose input mode itself may be insufficient to determine a desired user interface action because there is a 50% confidence (shown as modification option 1772) that the head pose is applied to a user interface action associated with application A 1672 and another 50% confidence (shown as modification option 1774) that the head pose is applied to another user interface action associated with application B 1674.

[0181] In various embodiments, certain applications or types of user interface actions may be programmed to be more responsive to certain input modes. For example, the HTML tags or Java script programming of application B 1674 may be set to be more responsive to gesture input than that of application A 1672. For example, application A 1672 may be more responsive to head poses 1672 than gestures 1612, while a "select" action may be more responsive to gestures 1612 (e.g., tap gestures) than head poses 1614 because a user is more likely to select an object using gestures than head poses, in some cases.

[0182] 17B, ​​gesture 1612 may be responsive to a certain type of user interface action in application B 1674. As shown, gesture 1612 may have a higher confidence associated with a user interface action for application B, while gesture 1612 may be inapplicable for an interface action in application A 1672. Thus, if the target virtual object is application A 1672, the input received from head pose 1614 may be the target user interface action. However, if the target virtual object is application B 1674, the input received from gesture 1612 (alone or in combination with input based on head pose 1614) may be the target user interface action.

[0183] As another example, because gesture 1612 has a higher confidence level than head pose 1614 when the user interacts with application B, gesture 1612 may be the primary input mode for application B 1674, while head pose 1614 may be the secondary input mode. Thus, input received from gesture 1612 may be associated with a higher weighting than head pose 1614. For example, if the head pose indicates that a virtual object associated with application B 1674 should remain stationary while gesture 1612 indicates that the virtual object should be moved to the left, the central runtime server 1650 may render the virtual object moving to the left. In one implementation, the wearable system may allow the user to interact with the virtual object using the primary input mode and may consider the secondary input mode if the primary input mode is insufficient to determine the user's action. For example, the user may primarily interact with application B 1674 using gesture 1612. However, if the HMD is unable to determine the target user interface action (e.g., because multiple candidate virtual objects may exist within application B 1674 or if the gesture 1612 is not clear), the HMD can use the head pose as input to identify the target virtual object or the target user interface action to be performed on application B 1674.

[0184] The scores associated with each input mode may be aggregated to determine a desired user interface action. FIG. 17C illustrates an example of aggregating confidence scores associated with input modes for a virtual object. As illustrated in this example, head pose input 1614 produces a higher confidence score for application A (80% confidence) than application B (30% confidence), while gesture input 1612 produces a higher confidence score for application B (60% confidence) than application A (30% confidence). The central runtime server 1650 can aggregate the confidence scores for each object based on the confidence scores derived from each user input mode. For example, the central runtime server 1650 can produce an aggregate score 110 for application A 1672 and an aggregate score 90 for application B 1674. The aggregated scores may be a weighted or unweighted average or other mathematical combination. Because application A 1672 has a higher aggregate score than application B 1674, the central runtime server 1650 may select application A as the application to be interacted with. Additionally, or alternatively, due to the higher aggregate score of application A 1672, the central runtime server 1650 may determine that application B is more "responsive" to gestures 1612 than application A, but that head pose 1614 and gestures 1612 are intended to perform a user interface action on application A 1672.

[0185] In this example, the central runtime server 1650 aggregates the resulting confidence scores by adding the confidence scores of the various inputs for a given object. In various other embodiments, the central runtime server 1650 can aggregate the confidence scores using techniques other than simple addition. For example, the input modes or scores may be associated with weights. As a result, the aggregation of confidence scores will take into account the weights assigned to the input modes or scores. The weights may be user adjustable, allowing the user to selectively adjust the "responsiveness" of the multimodal interaction with the HMD. The weights may also be contextual. For example, weights used in public places may emphasize head or eye postures over hand gestures to avoid possible social awkwardness of having the user gesture frequently while operating the HMD. As another example, in a subway, airplane, or train, voice commands may be given lower weights than head or eye postures because the user may not want to speak loudly to the HMD in such an environment. Environmental sensors (e.g., GPS) may assist the user in determining the appropriate context in which to operate the HMD.

[0186] Although the examples in Figures 17A-17C are illustrated with reference to two objects, the techniques described herein can also be applied when more or fewer objects are present. In addition, the techniques described with reference to these figures can also be applied to an application of the wearable system or to a virtual object associated with one or more applications. Furthermore, the techniques described herein can also be applied to direct or indirect input modes other than head pose, eye gaze, or gestures. For example, voice commands may also be used. In addition, although a central runtime server 1650 has been used throughout as an example to illustrate the processing of various input modes, the local processing and data module 260 of the HMD may also perform some or all of the operations in addition to or as an alternative to the central runtime server 1650. Exemplary Techniques for Calculating Confidence Scores

[0187] The wearable system can use one or a combination of various techniques to calculate the confidence score of an object. Figures 18A and 18B illustrate an example of calculating a confidence score for an object within a user's FOV. The user's FOV may be calculated based on the user's head pose or eye gaze, for example, during cone casting. The confidence scores in Figures 18A and 18B may be based on a single input mode (e.g., the user's head pose, etc.). Multiple confidence scores may be calculated (for some or all of the various multi-modal inputs) based on multi-modal user inputs and then aggregated to determine a user interface action or a target virtual object.

[0188] FIG. 18A illustrates an example in which a confidence score for a virtual object is calculated based on the portion of the virtual object that falls within a user's FOV 1810. In FIG. 18A, the user's FOV has portions of two virtual objects (represented by a circle 1802 and a triangle 1804). The wearable system may assign confidence scores to the circle and the triangle based on the percentage of the object's projected area that falls within the FOV 1810. As shown, approximately half of the circle 1802 falls within the FOV 1810, such that the wearable system may assign a confidence score of 50% to the circle 1802. As another example, approximately 75% of the triangle is within the FOV 1810, such that the wearable system may assign a confidence score of 75% to the triangle 1804.

[0189] The wearable system can use a regression analysis of the content in the FOV and the FOR to calculate the percentage of the virtual object within the FOV. As described with reference to FIG. 12B, the wearable system tracks the object within the FOR, but the wearable system may deliver the object (or a portion of the object) within the FOV to a rendering projector (e.g., display 220) for display within the FOV. The wearable system can determine the portion provided for the rendering projector, analyze the proportion delivered to the rendering projector relative to the virtual object as a whole, and determine the percentage of the virtual object within the FOV.

[0190] In addition to or as an alternative to calculating the confidence score based on the proportional area falling within the FOV, the wearable system can also analyze the space in the vicinity of the object within the FOV and determine the confidence score of the object. FIG. 18B illustrates an example of calculating the confidence score based on the uniformity of the space surrounding the virtual object within the FOV 1820. The FOV 1820 includes two virtual objects, as depicted by a triangle 1814 and a circle 1812. The space around each virtual object may be represented by a vector. For example, the space around virtual object 1812 may be represented by vectors 1822a, 1822b, 1822c, and 1822d, while the space around virtual object 1814 may be represented by vectors 1824a, 1824b, 1824c, and 1824d. The vectors may originate from the virtual object (or a boundary with the virtual object) and terminate at the edge of the FOV 1820. The system can analyze the distribution of vector lengths from objects to the edge of the FOV and determine objects that are located more toward the center of the FOV. For example, objects in the very center of a circular FOV will have a relatively uniform distribution of vector lengths, while objects very close to the edge will have a non-uniform distribution of vector lengths (because some vectors pointing to nearby edges will be shorter, while vectors pointing to the farthest edges will be longer). As depicted in FIG. 18B, the distribution of vector lengths from the virtual triangle 1814 to the edge of the field of view 1820 is more variable than the distribution of vector lengths from the circle 1812 to the edge of the field of view 1820, indicating that the virtual circle 1812 is closer to the center of the FOV 1820 than the virtual triangle 1814. The variability of the distribution of vector lengths may be represented by the standard deviation or variance (or other statistical measure) of the lengths. The wearable system can therefore assign a higher confidence score to the virtual circle 1812 than to the virtual triangle 1814.

[0191] In addition to the techniques described with reference to FIGS. 18A and 18B, the wearable system may assign confidence scores to virtual objects based on historical analysis of user interactions. As an example, the wearable system may assign higher confidence scores to virtual objects with which the user frequently interacts. As another example, one user may tend to move a virtual object using voice commands (e.g., "move it there"), while another user may prefer to use hand gestures (e.g., by reaching out and "grabbing" the virtual object and moving it to another position). The system may determine such user tendencies from historical analysis. As yet another example, an input mode may be frequently associated with a particular user interface action or a particular virtual object, such that the wearable system may increase the confidence score for a particular user interface action or a particular virtual object, even though there may be alternative user interface actions or virtual objects based on the same input.

[0192] As depicted in FIG. 18A or 18B, given either the field of view 1810 or 1820, the second input mode can facilitate selection of an appropriate virtual object or an appropriate user interface action on the virtual object. For example, a user can utter "magnify the triangle" to increase the size of the triangle in the field of view 1810. As another example, in FIG. 18A, a user may give a voice command such as "double it." The wearable system may determine that the subject (e.g., target object) of the voice command is the virtual object 1804 because the virtual object 1804 has a higher confidence score based on the head pose. Advantageously, in some embodiments, this reduces the specificity of the interaction required to produce a desired result. For example, a user does not need to utter "double the triangle" for the wearable system to accomplish the same interaction.

[0193] The triangles and circles in Figures 18A and 18B are for illustration purposes only. The various techniques described herein can also be applied to virtual content that supports more complex user interactions. Exemplary multimodal interactions in the physical environment

[0194] In addition to, or as an alternative to, interacting with virtual objects, the wearable system can also provide a wide range of interactions in the real-world environment. FIGS. 19A and 19B illustrate an example of interacting with a physical environment using multimodal input. In FIG. 19A, three input modes are illustrated: hand gestures 1960, head pose 1920, and input from a user input device 1940. Head pose 1920 can be determined using a posture sensor. The posture sensor may be an IMU, gyroscope, magnetometer, accelerometer, or other type of sensor described in FIG. 2. Hand gestures 1960 may be measured using an outward-facing imaging system 464, while user input device 1940 may be an embodiment of user input device 466 shown in FIG. 4.

[0195] In some embodiments, the wearable system can also measure the user's eye line of sight. The eye line of sight may include a vector extending from each of the user's eyes to a location where the two ocular lines of sight converge. The vector can be used to determine the direction the user is looking and can be used to select or identify virtual content at the convergence point or along the vector. Such eye lines of sight may be determined by eye tracking techniques such as, for example, flash detection, iris or pupil shape mapping, infrared illumination, or binocular imaging with regression of intersection points resulting from individual pupil orientations. The eye line of sight or head pose can then be considered as a source point for cone or ray casting for virtual object selection.

[0196] As described herein, an interaction event for moving selected virtual content in a user's environment (e.g., "put it there") may require the determination of a command action (e.g., "put"), a target (e.g., "it" as may be determined from the multimodal selection techniques described above), and a parameter (e.g., "there"). The command action (or command for short) and target (also referred to as a target object or target virtual object) may be determined using a combination of input modes. For example, a command to move a target 1912 may be based on a head pose 1920 change (e.g., head turning or nodding) or a hand gesture 1960 (e.g., a swipe gesture), alone or in combination. As another example, the target 1912 may be determined based on a combination of head pose and eye gaze. Thus, commands based on multimodal user input may also sometimes be referred to as multimodal input commands.

[0197] The parameters may also be determined using single or multi-modal inputs. The parameters may be associated with objects in the user's physical environment (e.g., a table or a wall) or with objects in the user's virtual environment (e.g., a movie application, an avatar, or a virtual building in a game). Identifying real-world parameters may enable faster and more accurate content placement responses in some embodiments. For example, a particular virtual object (or a portion of a virtual object) may be approximately planar with a horizontal orientation (e.g., the virtual object's normal is perpendicular to the floor of the room). When the user initiates an interaction that moves the virtual object, the wearable system may identify a real-world surface with a similar orientation (e.g., the surface of a table) and move the virtual object to the real-world surface. In some embodiments, such movement may be automatic. For example, a user may want to move a virtual book from its location on the floor. The only horizontal surface available in the room may be the user's desk. Thus, the wearable system may automatically move the virtual book to the desk surface in response to a voice command of "move it" without the user inputting any additional commands or parameters, since the desk surface is the most likely place where the user would want to move the book. As another example, the wearable system may identify a real-world surface of suitable size for a given content, thereby providing a better parameter match for the user. For example, if a user is watching a virtual video screen with a given display size and wants to move it to a particular surface using a simple voice command, the system may determine the real-world surface that provides the surface area required to best support the display size of the virtual video.

[0198] The wearable system can identify target parameters (e.g., target surfaces) using techniques described with reference to identifying a target virtual object. For example, the wearable system can calculate confidence scores associated with multiple target parameters based on indirect or direct user input. As an example, the wearable system can calculate a confidence score associated with a wall based on direct input (such as a user's head pose) and indirect input (such as a property of the wall (e.g., a vertical surface)). Exemplary Techniques for Identifying Real-World Parameters

[0199] The wearable system can use various techniques to determine parameters (such as target location) of a multi-modal input command. For example, the wearable system can use various depth sensing techniques, such as, for example, applying the SLAM protocol to environmental depth information (e.g., as described with reference to FIG. 9) or building or accessing a mesh model of the environment. In some embodiments, depth sensing determines the distance between known points in 3D space (e.g., the distance between sensors on the HMD) and points of interest ("POIs") on the surfaces of objects in the real world (e.g., walls for positioning virtual content). This depth information may be stored in the world map 920. Parameters for interaction may be determined based on a collection of POIs.

[0200] The wearable system can apply these depth sensing techniques to data obtained from a depth sensor to determine demarcations and boundaries of the physical environment. The depth sensor may be part of an outward-facing imaging system 464. In some embodiments, the depth sensor is coupled to an IMU. The data obtained from the depth sensor can be used to determine the orientation of multiple POIs relative to one another. For example, the wearable system can calculate a truncated signed distance function ("TSDF") for the POIs. The TSDF can include a numerical value for each POI. The numerical value may be zero when the point is within a given tolerance of a particular plane, positive when the point is spaced from the particular plane in a first direction (e.g., upward or outward), and negative when the point is spaced from the particular plane in a second (e.g., opposite) direction (e.g., downward or inward). The calculated TSDF can be used to define a 3-D volume grid of bricks or boxes aligned in, above, and below a particular plane, and aligned with an orientation as determined by the IMU, that creates or represents a particular surface.

[0201] POIs outside a given planar tolerance (e.g., with absolute values ​​of TSDFs above the tolerance) may be eliminated, leaving only POIs adjacent to each other within the given tolerance to create a virtual representation of a surface in the real-world environment. For example, the real-world environment may include a conference table. Various other objects (e.g., a phone, a laptop computer, a coffee mug, etc.) may be present on top of the conference table. For the conference table surface, the wearable system may keep the POIs associated with the conference table and remove POIs related to other objects. As a result, a planar map (outlining the surface of the conference table) may represent the conference table with only points that belong to the conference table. The map may exclude points associated with objects on top of the conference table. In an embodiment, the collection of remaining POIs in the planar map may be referred to as the "workable surface" of the environment, as these areas of the planar map represent a space in which virtual objects may be placed. For example, when a user wishes to move a virtual screen onto a table, the wearable system may identify suitable surfaces in the user's environment (such as a table top, a wall, etc.) while eliminating objects (such as a coffee mug or a pencil or a picture on a wall) or surfaces that are not suitable for placing the screen (such as the surface of a bookshelf). In this example, the identified suitable surfaces may be workable surfaces of the environment.

[0202] Referring back to the example shown in FIG. 19A, the environment 1900 may include a physical wall 1950. The HMD or user input device 1940 may house a depth sensor system (e.g., a time-of-flight sensor or a vertical cavity surface-emitting laser (VCSEL) or the like) and an attitude sensor (e.g., an IMU or the like). Data acquired by the depth sensor system may be used to identify various POIs within the user's environment. The wearable system may group together POIs that are approximately planar to form a bounding polygon 1910. The bounding polygon 1910 may be an exemplary embodiment of a workable surface.

[0203] In some embodiments, the outwardly facing imaging system 464 can identify a user gesture 1960, which may include pointing with a finger at an area within the real-world environment 1900. The outwardly facing imaging system 464 can identify the pre-measured bounding polygon 1910 by determining a sparse point vector structure of the finger pointing towards the bounding polygon 1910.

[0204] As illustrated in FIG. 19A, a virtual video screen 1930 can be present inside the bounding polygon 1910. A user can interact with a virtual object 1912 inside the virtual video screen 1930 using multimodal input. FIG. 19B depicts interaction using multimodal input of virtual content in a real-world environment. The environment in FIG. 19B includes a vertical surface 1915 (which may be part of a wall) and a tabletop surface 1917. In a first state 1970a, virtual content 1926 is initially displayed within a bounding polygon 1972a on the wall surface 1915. A user can select the virtual object 1926, for example, through cone casting or multimodal input (including two or more of a gesture 1960, a head pose 1920, an eye gaze, or input from a user input device 1940).

[0205] The user can use another input as part of the multimodal input to select surface 1917 as a destination. For example, the user can use a head pose combined with a hand gesture to indicate that surface 1917 is a destination. The wearable system can recognize surface 1917 (and polygon 1972b) by grouping POIs that appear to be on the same plane. The wearable system can also use other surface recognition techniques to identify surface 1917.

[0206] The user can also use multi-modal input to transport the virtual content 1126 to a bounding polygon 1972b on the surface 1917, as shown in the second state 1970b. For example, the user can move the virtual content 1926 through a combination of head pose changes and movement of the user input device 1940.

[0207] As another example, the user may utter "move it there" via the microphone 232 of the wearable system, which may receive an audio stream and parse the command therefrom (as described herein). The user may combine this voice command with head pose, eye gaze, gesture, or totem actuation. The wearable system may detect the virtual object 1926 as the target of the command because the virtual object 1926 is the highest confidence object (see, for example, the dashed line in the scene 1970a indicates the user's finger 1960, and the HMD 1920 and totem 1940 are oriented toward the object 1926). The wearable system may also identify the command action as "move" and determine that the parameter of the command is "there". The wearable system may further determine that "there" refers to the bounding polygon 1972b based on input modes other than voice (e.g., eye gaze, head pose, gesture, totem).

[0208] Commands in an interaction event may involve adjustment and calculation of multiple parameters. For example, the parameters may include a destination, location, orientation, appearance (e.g., size or shape), or animation of a virtual object. The wearable system may automatically calculate the parameters when changing the parameters even when direct input is not explicit. As an example, the wearable system may automatically change the orientation of a virtual object 1926 when it is moved from a vertical surface 1915 to a horizontal surface 1917. In a first state 1970a, the virtual content 1926 is approximately vertically oriented on the surface 1915. When the virtual content 1926 is moved to the surface 1917 in a second state 1970b, the orientation of the virtual content 1926 may be kept consistent (e.g., maintain a vertical orientation) as shown by the virtual object 1924. The wearable system may also automatically adjust the orientation of the virtual content 1926 to match the orientation of the surface 1917 so that the virtual content 1926 appears to be in a horizontal position, as illustrated by the virtual object 1922. In this example, the orientation may be automatically adjusted based on the environment tracking 1632 as an indirect input. The wearable system may automatically take into account the characteristics of the object (e.g., the surface 1917) when the wearable system determines that the object is a target destination object. The wearable system may adjust the parameters of the virtual object based on the characteristics of the target destination object. In this example, the wearable system automatically rotates the orientation of the virtual object 1926 based on the orientation of the surface 1917.

[0209] Additional embodiments of automatically placing or moving virtual objects are described in a patent application filed on August 9, 2017 entitled “AUTOMATIC PLACEMENT OF A VIRTUAL No. 15 / 673,135, entitled "OBJECT IN A THREE-DIMENSIONAL SPACE," the disclosure of which is incorporated herein by reference in its entirety.

[0210] In some implementations, the input may explicitly modify multiple parameters. A voice command of "Put it flat there" may alter the orientation of the virtual object 1926 in addition to identifying the surface 1917 as the destination. In this example, the word "flat" and the word "there" can both be parameter values, with "there" causing the wearable system to update the location of the target virtual object, while the word "flat" is associated with the orientation of the target virtual object at the destination location. To execute the parameter "flat", the wearable system can match the orientation of the virtual object 1926 to match the orientation of the surface 1917.

[0211] In addition to or as an alternative to selecting and moving virtual objects, multimodal input can interact with virtual content in other ways. FIG. 20 illustrates an example of automatically resizing a virtual object based on multimodal input. In FIG. 20, a user 1510 can wear an HMD 1502 and can interact with virtual objects using hand gestures and voice commands 2024. FIG. 20 illustrates four scenes 2000a, 2000b, 2000c, and 2000d. Each scene includes a display screen and a virtual object (illustrated by a smiley face).

[0212] In scene 2000a, the display screen has a size 2010 and the virtual object has a size 2030. The user can change a hand gesture from gesture 2020 to gesture 2022 to indicate that the user desires to adjust the size of either the virtual object or the display screen. The user can use voice input 2024 to indicate whether the virtual object or the display screen is the subject of manipulation.

[0213] As an example, a user may wish to enlarge both the display screen and the virtual object. Thus, the user may use input gesture 2022 as a command to enlarge. A parameter regarding the degree of enlargement may be represented by the degree of spread fingers. Meanwhile, the user may use voice input 2024 to determine the object of interaction. As shown in scene 2000b, the user may utter "all" and produce an enlarged display 2012 and an enlarged virtual object 2032. As another example, in scene 2000c, the user may utter "content" and produce an enlarged virtual object 2034, while the size of the display screen remains the same as in scene 2000a. As yet another example, in scene 2000d, the user may utter "display" and produce an enlarged display screen 2016, while the virtual object remains the same size as in scene 2000a. Example of indirect input as an input mode

[0214] As described herein, the wearable system can be programmed to allow a user to interact with direct and indirect user inputs as part of a multimodal input. Direct user inputs may include head pose, eye gaze, voice input, gestures, input from a user input device, or other inputs directly from a user. Indirect inputs may include various environmental factors, such as, for example, the user's position, the user's characteristics / preferences, characteristics of objects, characteristics of the user's environment, etc.

[0215] As described with reference to FIG. 2, the wearable system may include a location sensor, such as, for example, a GPS or a radar or lidar. The wearable system may determine a target of a user's interaction as a function of the proximity of the object to the user. FIG. 21 illustrates an example of identifying a target virtual object based on the location of the object. FIG. 21 diagrammatically illustrates a bird's-eye view 2100 of a user's FOR. The FOR may include multiple virtual objects 2110a-2110q. The user may wear an HMD, which may include a location sensor. The wearable system may determine candidate target objects based on the proximity of the object to the user. For example, the wearable system may select virtual objects within a threshold radius (e.g., 1 m, 2 m, 3 m, 5 m, 10 m, or more) from the user as candidate target virtual objects. In FIG. 21, the virtual objects (e.g., virtual objects 2110o, 2110p, 2110q) fall within a threshold radius (illustrated by dashed circle 2122) from the user's position 2120. As a result, the wearable system can set the virtual objects 2110o-2110q as candidate target virtual objects. The wearable system can further refine the selection based on other inputs (e.g., the user's head pose, etc.). The threshold radius can depend on contextual factors such as the user's location. For example, the threshold radius can be smaller if the user is in his / her office than if the user is outdoors in a park. Candidate objects can be selected from a portion of the region 2122 within the threshold radius from the user. For example, only those objects within both the circle 2122 and the user's FOV (e.g., generally in front of the user) can be candidates, while objects within the circle 2122 but outside the user's FOV (e.g., behind the user) cannot be candidates. As another example, multiple virtual objects can be along a common line of sight. For example, cone casting may select multiple virtual objects. The wearable system can use the user's location as another input to determine a target virtual object or parameters for user interaction.For example, cone casting may select objects that correspond to different depth planes, while the wearable system may be configured to identify the target virtual object as an object within reach of the user.

[0216] Similar to direct inputs, indirect inputs may also be assigned values ​​that may be used to calculate a confidence score for a virtual object. For example, multiple targets or parameters were within the general confidence of the selection, but the indirect inputs may be used as a confidence factor as well. Referring to FIG. 21, a virtual object in the circle 2122 may have a higher confidence score than a virtual object between the circle 2122 and the circle 2124 because an object closer to the user's position 2120 is more likely to be an object the user is interested in interacting with.

[0217] In the example shown in Figure 21, the dashed circles 2122, 2124 are conveniently illustrated to represent the projection of a sphere of corresponding radius onto the plane shown in Figure 21. This is for illustration purposes and not limitation. In other implementations, regions of other shapes (e.g., polygons) may be selected.

[0218] 22A and 22B illustrate another example of interacting with the user's environment based on a combination of direct and indirect inputs. These two figures show two virtual objects, namely virtual object A 2212 and virtual object B 2214, within the world camera's FOV 1270, which may be larger than the user's FOV 1250. Virtual object A 2212 is also within the user's FOV 1250. For example, virtual object A 2212 may be a virtual document that the user is currently viewing, while virtual object B 2214 may be a virtual sticky note on the wall. However, while the user interacts with virtual object A 2212, the user may want to look at virtual object B 2214 to obtain additional information from virtual object B 2214. As a result, the user may turn his head to the right (changing the FOV 1250) to view virtual object B 2214. Advantageously, in some embodiments, rather than turning the head, the wearable system may detect a change in the user's gaze direction (towards virtual object B 2214), so that the wearable system can automatically move virtual object B 2214 within the user's FOV without the user having to change their head pose. Virtual object B may overlay (or be contained within) virtual object A, or object B may be located within the user FOV 1250 but at least partially spaced apart from object A (such that object A is also at least partially visible to the user).

[0219] As another example, virtual object B2214 may be on another user interface screen. A user may wish to switch between a user interface screen with virtual object A2212 and a user interface screen with virtual object B2214. The wearable system may perform the switching without changing the user's FOV 1250. For example, in response to detecting a change in eye gaze or an actuation of a user input device, the wearable system may automatically move the user interface screen with virtual object A2212 out of the user's FOV 1250 while moving the user interface screen with virtual object B2214 into the user's FOV 1250. As another example, the wearable system may automatically overlay the user interface screen with virtual object B2214 on top of the user interface screen with virtual object A2212. Once the user provides an indication that they are finished interacting with the virtual user interface screen, the wearable system may automatically move the virtual user interface screen out of the FOV 1250.

[0220] Advantageously, in some embodiments, the wearable system can identify virtual object B2214 as a target virtual object to be moved into the FOV based on the multi-modal input. For example, the wearable system can make the determination based on the user's eye line of sight and the location of the virtual object. The wearable system can set the target virtual object as the object that is in the user's line of sight and is the object closest to the user. Exemplary Process for Interacting with Virtual Objects Using Multimodal User Input

[0221] 23 illustrates an example process for interacting with a virtual object using multimodal input. The process 2300 can be performed by the wearable system described herein. For example, the process 2300 may be performed by the local processing and data module 260, the remote processing module 270, and the central runtime server 1650, either alone or in combination.

[0222] In block 2310, the wearable system can optionally detect a start condition. The start can be a user-initiated input, which can provide an indication that the user intends to issue a command to the wearable system. The start condition can be predetermined by the wearable system. The start condition can be a single input or a combination input. For example, the start condition can be a voice input, such as by uttering the phrase "Hey Magic Leap". The start condition can also be gesture-based. For example, the wearable system can detect the presence of a start condition when it detects the user's hand within the world camera's FOV (or the user's FOV). As another example, the start condition can be a specific hand movement, such as, for example, a finger snap. The start condition can also be detected when the user actuates a user input device. For example, the user can click a button on the user input device, indicating that the user will issue a command. In some implementations, the start condition can be based on a multi-modal input. For example, both voice commands and hand gestures may be required for the wearable system to detect the presence of a starting condition.

[0223] Block 2310 is optional. In some embodiments, the wearable system may receive and begin analysis of multimodal input without detection of a start condition. For example, when a user is watching a video, the wearable system may capture the user's multimodal input and adjust volume, fast forward, rewind, skip to the next episode, etc., without requiring the user to first provide a start condition. Advantageously, in some embodiments, the user may not need to wake up the video screen before the user can interact with the video screen using the multimodal input (e.g., so that the video screen may present time or volume adjustment tools).

[0224] In block 2320, the wearable system can receive multimodal input for user interaction. The multimodal input may be direct or indirect input. Exemplary input modes may include voice, head pose, eye gaze, gestures (on the user input device or in the air), inputs on the user input device (e.g., a totem, etc.), properties of the user's environment, or objects (physical or virtual objects) in 3D space.

[0225] In block 2330, the wearable system can analyze the multimodal input and identify targets, commands, and parameters for user interaction. For example, the wearable system can assign confidence scores to candidate target virtual objects, target commands, and target parameters, and select the targets, commands, and parameters based on the highest confidence scores. In some embodiments, one input mode can be a primary input mode while another input mode can be a secondary input mode. Input from the secondary input mode can complement the input from the primary input mode and confirm the target targets, commands, or parameters. For example, the wearable system can set head pose as the primary input mode and voice command as the secondary input mode. The wearable system can first interpret the input from the primary input mode as much as possible and then interpret additional input from the secondary input mode. If the additional input is interpreted to suggest a different interaction than that of the primary input, the wearable system can automatically provide a disambiguation prompt to the user. The disambiguation prompt can request the user to select a desired task from alternative options based on the interpretation of the primary input or the interpretation of the secondary input. Although the present embodiment is described with reference to a primary input mode and a secondary input mode, in various circumstances there may be more than two input modes, and the same techniques may also be applicable with a tertiary input mode, a fourth input mode, etc.

[0226] In block 2340, the wearable system can execute the user interaction based on the object, the command, and the parameter. For example, the multimodal input may include eye gaze and a voice command "put it there." The wearable system can determine that the object of the interaction is the object with which the user is currently interacting, the command is "put it," and the parameter is the center of the user's fixation field (determined based on the user's eye gaze direction). Thus, the user can move the virtual object with which the user is currently interacting to the center of the user's fixation field. Example of setting direct input mode associated with user interaction

[0227] In some situations, such as when a user interacts with a wearable system using posture, gestures, or voice, there is a risk that other people in the user's vicinity may "hijack" the user's interaction by issuing commands using these direct inputs. For example, user A is standing in the vicinity of user B in a park. User A may use voice commands to interact with the HMD. User B may hijack user A's experience by uttering "take a photo." This voice command issued by user B may cause user A's HMD to take a photo even if user A did not intend to take a photo. As another example, user B may perform a gesture within the FOV of the world camera of user A's HMD. This gesture may cause user A's HMD to go to, for example, a homepage while user A is playing a video game.

[0228] In some implementations, the input can be analyzed to determine whether the input originated from a user. For example, the system can apply speaker recognition techniques to determine whether the command "take a picture" was uttered by user A or hijacker B. The system may also apply computer vision techniques to determine whether a gesture was made by the hand of user A or the hand of hijacker B.

[0229] Additionally or alternatively, to prevent security breaches and disruption of a user's interaction with the wearable system, the wearable system can automatically set the available direct input modes based on the indirect inputs, or require multiple modes of direct input before a command is issued. FIG. 24 illustrates an example of setting the direct input modes associated with user interaction. Three direct inputs are illustrated in FIG. 24: voice 2412, head pose 2414, and hand gesture 2416. As explained further below, slider bars 2422, 2424, and 2426 represent the amount each input is weighted in determining a command. When the slider is at the far right, the input is given full weighting (e.g., 100%), when the slider is at the far left, the input is given zero weighting (e.g., 0%), and when the slider is between these extreme settings, the input is given partial weighting (e.g., 20% or 80% or some other intermediate value such as a value between 0 and 1). In this example, the wearable system can be configured to require both a voice command 2422 and a hand gesture 2426 (while not using head pose 2414) before a command is executed. Thus, the wearable system cannot execute a command if the voice command 2442 and the gesture 2426 indicate different user interactions (or virtual objects). By requiring both types of input, the wearable system can reduce the likelihood that others will hijack the user's interaction.

[0230] As another example, one or more input modes may be disabled. For example, when a user interacts with a document processing application, head pose 2414 may be disabled as an input mode, as shown in FIG. 24, where head pose slider 2424 is set to 0.

[0231] Each input may be associated with an authentication level. In FIG. 24, voice 2412 is associated with authentication level 2422, head pose 2414 is associated with authentication level 2424, and hand gesture 2416 is associated with authentication level 2426. The authentication level may be used to determine whether an input is required for a command to be executed, or whether the input is disabled or given partial weighting (between fully enabled and fully disabled). As illustrated in FIG. 24, the authentication levels of voice 2412 and hand gesture 2416 are set to the rightmost (associated with the maximum authentication level), suggesting that these two inputs are required to issue a command. As another example, the authentication level of head pose is set to the leftmost (associated with the minimum authentication level). This suggests that head pose 2414 is not required to issue a command, even though head pose 2414 may still be used to determine a target virtual object or a target user interface action. In some circumstances, by setting the authentication level to a minimum, the wearable system may disable head pose 2414 as an input mode.

[0232] In some implementations, the authentication level may also be used to calculate a confidence level associated with the virtual object. For example, the wearable system may assign higher values ​​to input modes with higher authentication levels, while assigning lower values ​​to input modes with lower authentication levels. As a result, when aggregating confidence scores from multiple input modes to calculate an aggregated confidence score for a virtual object, input modes with higher authentication levels may have a greater weighting in the aggregated confidence score than input modes with lower authentication levels.

[0233] The authentication level can be set by the user (through input or via a settings panel) or automatically by the wearable system, for example based on indirect input. The wearable system may require more input modes when the user is in a public place, while requiring fewer input modes when the user is in a private place. For example, the wearable system may require both voice 2412 and hand gestures 2416 when the user is in the subway. However, when the user is at home, the wearable system may require only voice 2412 to issue commands. As another example, the wearable system may disable voice commands when the user is in a public park, thereby providing privacy to the user's interactions. However, voice commands may still be available when the user is at home.

[0234] Although these embodiments are described with reference to setting a direct input mode, similar techniques can also be applied to setting an indirect input mode as part of a multimodal input. For example, when a user is using public transportation (e.g., a bus, etc.), the wearable system may be configured to disable geographic location as an input mode because the wearable system may not know exactly where the user is specifically wearing or standing on the public transportation. Additional Exemplary User Experiences

[0235] In addition to the examples described herein, this section describes additional user experiences with multimodal input. As a first example, multimodal input can include voice input. For example, a user can issue a voice command such as "Hey Magic Leap, call her," which is received by the audio sensor 232 on the HMD and analyzed by the HMD system. In this command, the user can say, "Hey Magic Leap, call her," which is received by the audio sensor 232 on the HMD and analyzed by the HMD system. The wearable system may initiate the task (or provide a starting condition) by uttering "Leap." "Call" may be a pre-programmed word, so the wearable system knows to make a call (rather than initiate a video call). In some implementations, these pre-programmed words may also be referred to as "hot words" or "carrier phrases" that the system recognizes as indicating that the user desires to take a particular action (e.g., "call"), which may alert the system to receive further input to complete the desired action (e.g., identifying the person ("her") or phone number after the word "call"). The wearable system may use the additional input to identify who "her" is. For example, the wearable system may use eye tracking to see which virtual contact list or contacts on the user's phone the user is looking at. The wearable system may also use head pose or eye tracking to determine if the user is looking directly at the person the user desires to call. In some embodiments, the wearable system may utilize face recognition techniques (e.g., using the object recognizer 708) to determine the identity of the person the user is looking at.

[0236] As a second example, a user can place a virtual browser directly on a wall (e.g., the wearable system's display 220 can project the virtual browser as if it were overlaid on the wall). The user can reach out and provide a tap gesture on a link in the browser. As the browser appears to be on the wall, the user may tap on the wall or tap in space such that a projection of the user's finger appears to tap on the wall and provides an indication. The wearable system can use multimodal input to identify a link that the user intends to click. For example, the wearable system can use gesture detection (e.g., via data obtained by the outward-facing imaging system 464), head pose-based cone casting, and eye gaze. In this example, gesture detection can be less than 100% accurate. The wearable system can use data obtained from head pose and eye gaze to refine gesture detection and increase the accuracy of gesture tracking. For example, the wearable system can identify the radius where the eyes are most likely to be focused based on data obtained by the inward-facing imaging system 462. In an embodiment, the wearable system can identify the user's field of fixation based on the eye gaze. The wearable system can also use indirect inputs such as environmental features (e.g., wall locations, browser or web page characteristics, etc.) to improve gesture tracking. In this example, walls may be represented by a planar mesh (which may be pre-stored in the map 920 of the environment), and the wearable system can determine the location of the user's hands against the planar mesh to determine the link that the user targets and selects. Advantageously, in various embodiments, by combining multiple input modes, the accuracy required for one input mode for user interaction can be reduced compared to a low single input mode.For example, the FOV camera may not need to have very high resolution for hand gesture recognition because the wearable system can complement the hand gestures with head pose or eye gaze to determine the intended user interaction.

[0237] Although the multimodal input in the above examples includes audio input, audio input is not required for the multimodal input interaction described above. For example, a user can use a 2D-touch swipe gesture (e.g., on a totem) to move a browser window from one wall to a different wall. The browser may initially be on the left wall. The user can select the browser by activating the totem. The user can then look at the right wall and perform a right swipe gesture on the totem's touchpad. Swiping on a touchpad is unrestrained and imprecise because 2D swipes themselves do not translate easily / well into 3D movement. However, the wearable system can detect the wall (e.g., based on environmental data obtained by an outward-facing imaging system) and detect the point on the wall that the user is specifically looking at (e.g., based on eye gaze). Using these three inputs (touch-swipe, gaze, and environmental features), the wearable system can smoothly place the browser wherever the user wants to move the browser window with a high degree of reliability. Additional implementation of head pose as multimodal input.

[0238] In various embodiments, multimodal input can support totem-free experiences (or experiences in which totems are used less frequently). For example, multimodal input can include a combination of head pose and voice control, which can be used to share or search for virtual objects. Multimodal input can also use a combination of head pose and gestures to navigate various user interface planes and virtual objects within the user interface planes. A combination of head pose, voice, and gestures can be used to move objects, perform social networking activities (e.g., starting and conducting a telepresence session, sharing a post), browse information about a web page, or control a media player.

[0239] FIG. 25 illustrates an example of a user experience using multimodal input. In the example scene 2500a, a user 2510 can use head pose to target and select applications 2512 and 2514. The wearable system can display a focus indicator 2524a and use head pose to indicate that the user is currently interacting with a virtual object. Once the user selects an application 2514, the wearable system may show a focus indicator 2524a for the application 2514 (e.g., a target graphic as shown in FIG. 25, a halo around the application 2514, or making the virtual object 2514 appear closer to the user, etc.). The wearable system can also change the appearance of the focus indicator from focus indicator 2524a to focus indicator 2524b (e.g., an arrow graphic shown in scene 2500b) to indicate that interaction with the user input device 466 is also available after the user selects with the virtual object 2514. Voice and gesture interaction also extends this interaction pattern of head pose plus hand gestures. For example, when a user issues a voice command, an application targeted with the head pose may respond to or be manipulated by the voice command. Additional examples of virtual objects interacting with a combination of head pose, hand gestures, and voice recognition, for example, are described in U.S. Application No. 15 / 296,869, filed October 18, 2016, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE," and published as U.S. Patent Publication No. 2017 / 0109936, the disclosure of which is incorporated herein by reference in its entirety.

[0240] Head pose may be integrated with voice control, gesture recognition, and environmental information (e.g., mesh information) to provide hands-free browsing. For example, a voice command of "Search Fort Lauderdale" will be handled by the browser if the user targets the browser using head pose. If the user does not target a specific browser, the wearable system can also handle this voice command without going through the browser. As another example, if the user utters "Share this with Karen," the wearable system will perform a share action on the application the user has targeted (e.g., using head pose, eye gaze, or gestures). As another example, voice control can perform browser window functions, such as "Go to Bookmarks," while gestures may be used to perform basic navigation of web pages, such as clicking and scrolling.

[0241] Multimodal input can also be used to launch and move virtual objects without the need for a user input device. The wearable system can place content naturally in the vicinity of the user and the environment using multimodal input such as gestures, voice, and gaze. For example, a user can use voice to open an application that is not launched when the user interacts with the HMD. The user can issue a voice command by saying "Hey Magic Leap, launch my browser." In this command, the initiation condition includes the presence of the invocation phrase "Hey Magic Leap." The command can be interpreted to include "launch" or "open" (which may be interchangeable commands). The target of this command is the application name, e.g., "browser." However, this command does not require parameters. In some embodiments, the wearable system can automatically apply default parameters, such as, for example, placing a browser in the user's environment (or the user's FOV).

[0242] Multimodal input can also be used to implement basic browser controls, such as opening bookmarks, opening new tabs, navigating history, etc. The ability to browse web content in a hands-free or hands-full multitasking scenario can make a user more informed and productive. For example, user Ada is a radiologist reading films in her office. Ada can navigate the web and retrieve reference materials using voice and gestures while reading the films, reducing the need to move the mouse back and forth and switch between the film and the reference materials on the screen. As another example, user Chris is cooking a new recipe from a virtual browser window. The virtual browser window can be placed on the cabinet. Chris can use voice commands to start shearing food while pulling up the bookmarked recipe.

[0243] 26 illustrates an example user interface with various bookmarked applications. A user can select an application on the user interface 2600 by uttering the name of the application. For example, a user can utter "open food" to launch the food application. As another example, a user can utter "open this." The wearable system can determine the user's gaze direction and identify applications on the user interface 2600 that intersect with the user's gaze direction. The wearable system can thus open the identified application.

[0244] The user can also use voice to issue a search command. The search command can be implemented by the application currently targeted by the user. If the object does not currently support a search command, the wearable system may implement a search in the wearable system's data storage or a search for the information via a default application (e.g., via a browser, etc.). FIG. 27 illustrates an exemplary user interface 2700 when a search command is issued. The user interface 2700 shows both an email application and a media viewing application. The wearable system may determine that the user is currently interacting with the email application (based on the user's head pose). As a result, the wearable system may automatically convert the user's voice command into a search command within the email application.

[0245] Multimodal input can also be used for media control. For example, the wearable system can use voice and gesture control to issue commands such as play, pause, mute, fast forward, and rewind, for example, to control a media player within an application (such as a screen). The user can use voice and gesture control for media applications and leave out the totem.

[0246] Multimodal input can also be used in social networking contexts: for example, users can initiate conversations and share experiences (e.g., virtual images, documents, etc.) without using a user input device. As another example, a user can join a telepresence session and set up a private context so that the user may feel comfortable to navigate the user interface using voice.

[0247] Thus, in various implementations, the system may utilize multi-modal inputs such as head pose + voice (e.g., for information sharing and general application search), head pose + gestures (e.g., for navigation within an application), or head pose + voice + gestures (e.g., for "put it there" functionality, media player control, social interaction, or browser applications). Additional Examples of Gesture Control as Part of Multimodal Input

[0248] There may be two non-limiting and non-exclusive classes of gesture interactions: event gestures and dynamic hand tracking. Event gestures may be responsive to events while a user is interacting with the HMD, such as a catcher's signal to a pitcher in a baseball game or a "like" signal in a browser window to cause the wearable system to open a share dialog. The wearable system may follow one or more gesture patterns performed by the user and respond to the events accordingly. Dynamic hand tracking may involve tracking of the user's hands with low latency. For example, a user may move their hands across the user's FOV and a virtual character may follow the movement of the user's fingers.

[0249] The quality of gesture tracking may depend on the type of user interaction. Quality may involve multiple factors, for example, robustness, responsiveness, and ergonomics. In some embodiments, event gestures may be nearly perfectly robust. The threshold for minimum acceptable gesture performance may be slightly lower for social experiences, pre-production interactions, and third party applications, as the aesthetics of these experiences may tolerate glitches, interruptions, low latency, etc., but gesture recognition can still remain highly performant and responsive in these experiences.

[0250] To increase the likelihood that the wearable system will respond to a user's gestures, the system can reduce or minimize latency for gesture detection (for both event gestures and dynamic hand tracking). For example, the wearable system can reduce or minimize latency by detecting when the user's hand is within the field of view of a depth sensor, automatically switching the depth sensor to an appropriate gesture mode, and then providing feedback to the user when a gesture can be performed.

[0251] As described herein, gestures can be used in combination with other input modes to launch, select, and navigate applications. Gestures can also be used to interact with virtual objects within an application by tapping in the air or on a surface (e.g., on a table or wall), scrolling, etc.

[0252] In an embodiment, the wearable system can implement social networking tools that can support gestural interactions. Users can perform semantic event gestures to enrich communication. For example, a user can wave their hand in front of the FOV camera, and an animation of the hand wave can be sent appropriately to the person with whom the user is chatting. The wearable system can also provide a virtualization of the user's hand using dynamic hand tracking. For example, a user can lift their hand in front of their FOV and get visual feedback that their hand is being tracked to animate their avatar's hand.

[0253] Hand gestures can also be used as part of multimodal input for media player control. For example, a user can use hand gestures to play or pause a video stream. A user can perform gesture operations away from the device (e.g., a television) that is playing the video. In response to detecting the user's gestures, the wearable system can remotely control the device based on the user's gestures. The user can also look at a media panel, and the wearable system can use the user's hand gestures in combination with the user's gaze direction to update parameters of the media panel. For example, a pick (OK) gesture can suggest a "play" command and a fist gesture can suggest a "pause" command. A user can also close a menu by waving one of their arms in front of the FOV camera. An example of a hand gesture 2080 is shown in FIG. 20. Additional Examples of Interacting with Virtual Objects

[0254] As described herein, the wearable system can support a variety of multi-modal interactions with objects (physical or virtual) in the user's environment. For example, the wearable system can support direct input for interaction with found objects, such as targeting, selecting, controlling (e.g., movement or properties) the found objects. Interaction with found objects can also include interaction with found object geometry or interaction with found object connecting surfaces.

[0255] Direct input is also supported for interaction with flat surfaces, such as targeting and selecting a wall or table top. A user can also initiate various user interface events, such as, for example, a touch event, a tap event, a swipe event, or a scroll event. A user can manipulate 2D user interface elements (e.g., a panel) using direct interactions, such as, for example, panel scrolling, swiping, and selecting elements (e.g., virtual objects or user interface elements such as buttons) within a panel. A user can also move or resize a panel using one or more direct inputs.

[0256] Direct input can also be used to manipulate objects at different depths. The wearable system can set various threshold distances (from the user) and determine the area of ​​the virtual object. With reference to FIG. 21 , objects within the dashed circle 2122 can be considered as objects in the near distance, objects within (but outside) the dashed circle 2124 can be considered as objects in the middle distance, and objects outside the dashed circle 2124 can be considered as objects in the far distance. The threshold distance between the near and far distances may be, for example, 1 m, 2 m, 3 m, 4 m, 5 m, or more, and may depend on the environment (e.g., larger in an outdoor park than an indoor office subdivision).

[0257] The wearable system can support various 2D or 3D manipulations of virtual objects within a close distance. Exemplary 2D manipulations may include moving or resizing. Exemplary 3D manipulations may include setting the virtual object in 3D space by pinching, drawing, moving, or rotating the virtual object, etc. The wearable system can also support interactions with virtual objects within a medium distance, such as panning and repositioning the object in the user's environment, performing radial motion of the object, or moving the object into a close or far distance.

[0258] The wearable system can also support continuous fingertip interaction. For example, the wearable system can enable a user's finger to point like an attractor or pinpoint an object and perform a push interaction on the object. The wearable system can further support high-speed pose interaction, such as hand surface interaction or hand contour interaction. Additional Examples of Voice Commands in a Social Network and Sharing Context

[0259] The wearable system may support voice commands as input for social networking (or messaging) applications, for example, to share information with a contact or to call a contact.

[0260] As an example of initiating a call with a contact, a user can use a voice command such as "Hey Magic Leap, call Karen," where "Hey Magic Leap" is the invocation phrase, the command is "Call," and the command parameter is the name of the contact. The wearable system can automatically use a messenger application (as the target) to initiate the call. The command "Call" may be associated with a task, for example, "Initiate a call with," "Start a chat with," etc.

[0261] If the user says "Initiate a call" and then a name, the wearable system may attempt to recognize the name. If the wearable system does not recognize the name, it may communicate a message to the user for the user to confirm the name or contact information. If the wearable system recognizes the name, it may present a dialog prompt that allows the user to confirm / reject (or cancel) the call or provide an alternative contact.

[0262] A user can also initiate a call with some contacts with a list of friends. For example, a user can utter "Hey, start a group chat with Magic Leap, Karen, Cole, and Kojo." The group chat command may be derived from the phrase "start a group chat" or may be from a list of friends provided by the user. While the user is on a call, the user can add another user to the conversation. For example, a user can utter "Hey, Magic Leap, invite Karen," and the phrase "invite" can be associated with the invite command.

[0263] The wearable system can use voice commands to share virtual objects with contacts. For example, a user can utter, "Hey Magic Leap, share my screen with Karen" or "Hey Magic Leap, share it with David and Tony." In these examples, the word "share" is the share command. The words "screen" or "it" can refer to an object that the wearable system can determine based on multimodal input. The names, such as "Karen," "David and Tony," are parameters of the command. In some embodiments, when a voice command provided by a user includes an application reference and the word "share" with a contact, the wearable system may provide a confirmation dialog and ask the user to confirm whether the user wants to share the application itself or the object referenced through the application. When a user issues a voice command that includes the word "share," an application reference, and a contact, the wearable system can determine whether the application name is recognized by the wearable system or whether the application exists on the user's system. If the system does not recognize the name or the application does not exist in the user's system, the wearable system may provide a message to the user, which may suggest that the user try the voice command again.

[0264] If the user provides a deictic or anaphoric reference (e.g., "this" or "that") in the voice command, the wearable system can use multimodal input (e.g., the user's head pose) to determine if the user is interacting with an object that can be shared. If the object cannot be shared, the wearable system may prompt the user with an error message or transition to a second input mode, such as gestures, to determine the object to be shared.

[0265] The wearable system may also determine whether the contact with whom the object is to be shared can be recognized (e.g., as part of the user's contact list). If the wearable system recognizes the name of the contact, the wearable system may provide a confirmation dialog to confirm that the user wants to proceed with sharing. If the user confirms, the virtual object can be shared. In some embodiments, the wearable system may share multiple virtual objects associated with an application. For example, the wearable system may share an entire album of photos or share the most recently viewed photo in response to a user's voice command. If the user declines to share, the share command is canceled. If the user indicates that the contact is incorrect, the wearable system may prompt the user to speak the name of the contact again or to select the contact from a list of available contacts.

[0266] In one implementation, if a user utters "share" and utters an application reference but does not specify a contact, the wearable system may locally share the application with people in the user's environment who have access to the user's files. The wearable system may also reply and prompt the user to enter a name using one or more of the input modes described herein. Similar to the social networking embodiment, the user can issue a voice command to share a virtual object with a contact or a group of contacts.

[0267] A challenge in making a call via voice is when the voice user interface incorrectly recognizes or fails to recognize the contact's name. This can be particularly problematic for less common or non-English names, such as, for example, lsi, Ileana, etc. For example, when a user issues a voice command that includes the contact's name (such as "share my screen with lly"), the wearable system may be unable to identify the name "lly" or its pronunciation. The wearable system may open a contact dialog with a prompt, such as, for example, "Who are you?" The user may again attempt to use voice to define "Ily", to say the spelling of the name "ILY" using voice or a user input device, or to quickly select a name from a panel of available names using a user input device. The name "Ily" may be a nickname for Ileana, who has an entry in the user's contacts. Once the user tells the system that "Ily" is a nickname, the system may be configured to "remember" the nickname by automatically associating the nickname (or the pronunciation or audio pattern associated with the nickname) with the friend's name. Additional Examples of Selecting and Moving Virtual Objects Using Voice Commands

[0268] A user can naturally and quickly manage the location of virtual objects in the user's environment using multimodal input, such as a combination of eye gaze, gestures, and voice. For example, a user named Lindsay is seated at a table and ready to perform a task. She opens her laptop and starts a desktop monitor app on the computer. As the computer loads, she reaches her hand above the laptop screen and utters, "Hey, Magic Leap, put the monitors here." In response to this voice command, the wearable system can automatically bring up the monitor screens and place them above the laptop. However, when Lindsay looks across to a wall on the other side of the room and utters, "Put the screens there," the wearable system can automatically place the screens on the wall across from her. Lindsay can also look at her desk and utter, "Put Halcyon here." Halcyon was initially on the kitchen table, but in response to a voice command, the wearable system can automatically move it to the table surface. As she progresses, she can use her totem to interact with these objects and adjust their scale to her liking.

[0269] A user can use voice to open an application that is not launched at any time in the user's environment. For example, the user can say, "Hey Magic Leap, launch a browser." In this command, "Hey Magic Leap" is the invocation word, the word "launch" is the launch command, and the word "browser" is the target application. The "launch" command may be associated with the words "launch," "open," and "play." For example, the wearable system can still identify the launch command when the user says "open browser." In an embodiment, the application may be an immersive application, which can provide a 3D virtual environment to the user as if the user were part of the 3D virtual environment. As a result, when the immersive application is launched, the user can be positioned as if they are present in the 3D virtual environment. In one implementation, the immersive application also includes a store application. When the store application is launched, the wearable system can provide a 3D shopping experience for the user, such that the user can feel as if they are shopping in a physical store. In contrast to an immersive application, an application may be a landscape application, which when launched may be placed relative to where it would be placed if launched via a totem in the launcher, so that the user can interact with the landscape application but may not feel like they are part of it.

[0270] A user can also use voice commands to launch a virtual application at a defined location within the user's FOV, or to move an already placed virtual application (e.g., a landscape application) to a specific location within the user's FOV. For example, a user can utter "Hey Magic Leap, put my browser here," "Hey Magic Leap, put my browser there," "Hey Magic Leap, put this here," or "Hey Magic Leap, put that there." These voice commands include an invocation word, a placement command, an application name (which is the object), and a location cue (which is a parameter). The object may be referenced based on the audio data, for example, based on the name of the application spoken by the user. The object may also be identified based on head pose or eye gaze when the user utters the word "this" or "it" instead. To facilitate this voice interaction, the wearable system can, for example, make two inferences: (1) the application to launch and (2) the location to place the application.

[0271] The wearable system can use the installation command and the application name to infer the application to launch. For example, if the user speaks an application name that the wearable system does not recognize, the wearable system may provide an error message. If the user speaks an application name that the wearable system recognizes, the wearable system can determine whether the application is already installed in the user's environment. If the application is already shown in the user's environment (e.g., within the user's FOV, etc.), the wearable system can determine the number of instances of the application that exist in the user's environment (e.g., the number of open browser windows). If only one instance of the target application exists, the wearable system can move the application to a location specified by the user. If more than one instance of the spoken application exists in the environment, the wearable system can move all instances of the application to a specified location, or the most recently used instance to a specified location. If the virtual application is not already installed in the user's environment, the system can determine whether the application is a landscape application, an immersive application, or a store application (where the user can download or purchase other applications). If the application is a landscape application, the wearable system may launch a virtual application at a specified location. If the application is an immersive application, the wearable system may place a shortcut to the application at a specified location, since immersive applications do not support launching at a specified location within the user's FOV. If the application is a store application, the system may place a mini store at a specified location, since store applications may require full 3D immersion of the user into the virtual world and therefore do not support launching at a concrete location within the user's environment.The mini store may include overviews or icons of virtual objects within the store.

[0272] The wearable system can use various inputs to determine where to place an application. The wearable system can analyze syntax in the user's command (e.g., "here" or "there"), determine the intersection of head pose-based ray casting (or cone casting) with virtual objects in the user's environment, determine the position of the user's hands, determine a planar surface mesh or an environmental plane mesh (e.g., a mesh associated with a wall or table), etc. As an example, if the user utters "here," the wearable system can determine the user's hand gestures, such as whether an open hand is within the user's FOV. The wearable system can place an object in a rendering plane near the position of the user's open hand and within the reach of the user's hand. If the open hand is not within the FOV, the wearable system can determine whether the head pose (e.g., the direction of the head pose-based cone casting) intersects with a surface plane mesh within the reach of the user's arm. If a surface plane mesh exists, the wearable system can place the virtual object at the intersection of the head pose direction and the surface plane mesh in a rendering plane within the user's arm's reach. The user can place the object flat on the surface. If a surface plane mesh does not exist, the wearable system can place the virtual object in a rendering plane that has a distance between the arm's reach and the optimal reading distance. If the user utters "there", the wearable system can perform a similar action as when the user utters "here", but if a surface plane mesh does not exist within the user's arm's reach, the wearable system may place the virtual object in a rendering plane within a medium distance.

[0273] Once the user utters "Put application on...", the wearable system can immediately provide predictive feedback to the user, indicating where the virtual object would be placed based on available input if the user utters either "here" or "there". This feedback can be in the form of a focus indicator. For example, the feedback may include a hand, mesh, or planar surface that intersects with the user's head pose direction in a rendering plane within arms reach of the user, a small floating text bubble uttering "here". The planar surface may be located in the near distance if the user's command is "here", while it may be located in the middle or far distance if the user's command is "there". This feedback can be visualized like a shadow or outline of a visual object.

[0274] The user can also cancel an interaction, which may be canceled in two ways at various times: (1) failure to complete the command due to an n-second timeout, or (2) by entering a cancel command, such as saying "no," "never mind," or "cancel." Examples of interacting with text using a combination of user inputs

[0275] Free-form text entry in mixed reality environments, especially long string sequences using traditional interaction modalities, can be problematic. As an example, especially in "hands-free" environments, systems that lack input or interface devices such as keyboards, handheld controllers (e.g., totems), or mice, and that rely entirely on automatic speech recognition (ASR), can have difficulty using text editing (e.g., to correct ASR errors inherent in the speech recognition technology itself, such as incorrect transcription of a user's speech). As another example, a virtual keyboard in a "hands-free" environment can require elaborate user control and can result in fatigue if used as the primary form of user input.

[0276] The wearable system 200 described herein can be programmed to allow a user to naturally and quickly interact with virtual text using multimodal inputs, such as a combination of two or more of voice, eye gaze, gesture, head posture, totem input, etc. The term "text," as used herein, can include type, characters, words, phrases, sentences, paragraphs, or other types of free-form text. Text can also include graphics or animations, such as emojis, ideograms, emoticons, smileys, symbols, etc. Interaction with virtual text can include, alone or in combination, composing, selecting (e.g., selecting some or all of the text), or editing (e.g., modifying, copying, cutting, pasting, deleting, clearing, undoing, redoing, inserting, replacing, etc.) the text. By utilizing a combination of user inputs, the system described herein provides significant improvements in speed and convenience over single-input systems.

[0277] The multimodal text interaction techniques described herein can be applied in any dictation scenario or application (e.g., the system simply transcribes user utterances rather than applying any semantic evaluation, even if the transcription is part of another task that does not rely on semantic evaluation). Some example applications can include messaging applications, word processing applications, gaming applications, system configuration applications, etc. Examples of use cases can include a user writing a text message to be sent to a contact who may or may not be in the user's contact list, a user writing a letter, article, or other text content, a user posting and sharing content on a social media platform, and a user completing or otherwise filling out a form using the wearable system 200.

[0278] Systems utilizing a combination of user inputs need not be wearable systems: as desired, such systems may be any suitable computing system, such as a desktop computer, laptop, tablet, smartphone, or another computing device having multiple user input channels, such as a keyboard, trackpad, microphone, eye or gaze tracking system, gesture recognition system, etc. Example of composing text using multimodal user input

[0279] 28A-28F illustrate an example user experience of composing and editing text based on a combination of inputs, such as voice commands or eye gaze. As described herein, the wearable system can determine the user's gaze direction based on images obtained by the inwardly facing imaging system 462 shown in FIG. 4. The inwardly facing imaging system 462 may determine the orientation of one or both of the user's pupils and may extrapolate the user's monocular or binocular line of sight or lines of sight. By determining the user's binocular line of sight, the wearable system 200 can determine the three-dimensional location in space where the user is looking.

[0280] The wearable system can also determine voice commands based on data obtained from an audio sensor 232 (e.g., a microphone) shown in Figure 2. The system may have an automatic speech recognition (ASR) engine that converts the spoken input 2800 to text. The speech recognition engine may use natural language understanding in converting the spoken input 2800 to text, including isolating and extracting the message text from the longer utterance.

[0281] As shown in FIG. 28A, the audio sensor 232 can receive a phrase 2800 spoken by a user. As illustrated in FIG. 28A, the phrase 2800 can include a command, such as "send a message to John Smith saying...", as well as parameters of the command, such as to compose and send the message, and the destination of the message as John Smith. The phrase 2800 can also include the content of the message to be composed. In this example, the message content can include "I'm boarding a flight from Boston, arriving around 7:00. Meet me on the corner near my office". Such content can be obtained by parsing the audio data using an ASR engine, which can implement natural language understanding and isolate and extract the message content and punctuation (e.g., ".") from the user's utterance. In some examples, the punctuation can be processed for presentation in the context of the transcribed string (e.g., "two o'clock" can be presented as "2:00", or a "question mark" can be presented as "?"). The wearable system can also tokenize the text string, such as by isolating discrete words within the text string, and display the results, such as by displaying the discrete words in a mixed reality environment.

[0282] However, automatic speech recognition can be prone to errors in some circumstances. As illustrated in FIG. 28B, a system using an ASR engine may produce results that do not precisely match the user's spoken input for a variety of reasons, including poor or idiosyncratic pronunciation, environmental noise, homonyms and other similar-sounding words, hesitation or stuttering, and vocabulary that is not in the ASR's dictionary (e.g., foreign words, technical terms, jargon, slang, etc.). In the example of FIG. 28B, the system properly interpreted the command aspect of phrase 2800 and generated a message with header 2802 and body 2804. However, in the body 2804 of the message, the system incorrectly interpreted the user's utterance of "corner" as the somewhat similar-sounding "quarter." A system that relies entirely on voice input may have difficulty quickly replacing a misrecognized word (or phrase) with the intended word (or phrase) for the user. However, the wearable system 200 described herein can advantageously allow a user to quickly correct the error, as illustrated in FIGS. 28C-28F.

[0283] The ASR engine in the wearable system may produce text results including at least one word associated with the user's utterance and may produce an ASR score associated with each word (or phrase) in the text results. A high ASR score may indicate a high confidence or high likelihood that the ASR engine correctly transcribed the user's utterance into text, while a low ASR score may indicate a low confidence or low likelihood that the ASR engine correctly transcribed the user's utterance into text. In some embodiments, the system may display words with low ASR scores (e.g., ASR scores below an ASR threshold) in an emphasized manner (e.g., with background highlighting, italic or bold font, different color font, etc.), which may make it easier for a user to identify or select an incorrectly recognized word. A low ASR score for a word may indicate that a user is more likely to select the word for editing or replacement because there is a reasonable likelihood that the ASR engine misrecognized the word.

[0284] As shown in Figures 28C and 28D, the wearable system may allow the user to select the misrecognized word (or phrase) using an eye-tracking system, such as inward-facing imaging system 462 of Figure 4. In this example, the selected word may be an example of a target virtual object as described above with reference to the preceding figures.

[0285] The wearable system 200 can determine the gaze direction based on the inward facing imaging system 462 and can cast a cone 2806 or light beam in the gaze direction. The wearable system can select one or more words that intersect with the user's gaze direction. In an implementation, a word may be selected when the user's gaze dwells on an incorrect word for at least a threshold time. As described above, the incorrect word may be determined, at least in part, by being associated with a low ASR score. The threshold time may be any amount of time sufficient to indicate that the user desires to select a particular word, but is a length that does not unnecessarily delay the selection. The threshold time may also be used to determine a confidence score that indicates that the user desires to select a particular virtual word. For example, the wearable system can calculate a confidence score based on how long the user has been gazing at a certain direction / object, and the confidence score may increase as the time duration for looking at a certain direction / object increases. The confidence score may also be calculated based on a multi-modal input, as described herein. For example, the wearable system may determine if both a user's hand gestures and eye gaze indicate that a word should be selected, with a higher confidence score (than a confidence score derived from eye gaze alone).

[0286] As another example, the wearable system may calculate a confidence score based in part on the ASR score, which may indicate the relative confidence of the ASR engine of the translation of a particular word, as discussed in more detail herein. For example, a low ASR engine score may indicate that the ASR engine has relatively low confidence that it correctly transcribed the spoken word. Thus, there may be a higher probability that the user will likely select that word to edit or replace. If the user's gaze lingers on a word with a low ASR score for longer than a threshold time, the system may assign a higher confidence score to reflect that the user selected that word for at least two reasons: first, the length of the eye gaze on the word, and second, the fact that the word was likely transcribed incorrectly by the ASR engine (both of which tend to indicate that the user will want to edit or replace the word).

[0287] A word may be selected if the confidence score reaches a threshold criterion. As an example, the threshold time may be 0.5 seconds, 1 second, 1.5 seconds, 2 seconds, 2.5 seconds, 1-2 seconds, 1-3 seconds, etc. Thus, a user can easily and quickly select the incorrect word, "quarter," simply by looking for a sufficient amount of time. A word may be selected based on a combination of eye gaze (or gesture) time and an ASR score above the ASR threshold, both criteria providing an indication that the user intends to select that particular word.

[0288] As an example, if the results of the ASR engine include a first word with a high ASR score (e.g., a word that the ASR engine is relatively confident that it recognized correctly) and a second word with a low ASR score (e.g., a word that the ASR engine is relatively confident that it recognized incorrectly), and these two words are displayed adjacent to one another by the wearable system, the wearable system may assume that the user's gaze input encompassing both the first and second words is in fact an attempt by the user to select the second word based on its relatively low ASR score, because the user is more likely to want to edit the incorrectly recognized second word than the correctly recognized first word. In this way, the word produced by the ASR engine with the low ASR score, which is more likely to be incorrect and require editing, may be significantly easier for the user to select for editing, and thus encourage editing by the user.

[0289] Although this example describes using eye gaze to select the misrecognized word, other multimodal inputs can also be used to select the word. For example, cone casting can identify multiple words such as "around," "7:00," "the," and "quarter" because they also intersect with a portion of the virtual cone 2806. As will be further described with reference to FIGS. 29-31 , the wearable system can combine eye gaze input with another input (such as a gesture, voice command, or input from the user's input device 466) and select the word "quarter" as the word for further editing.

[0290] In response to selecting the word 2808, the system can enable editing of the selected word. The wearable system can enable the user to edit the word using various techniques, such as, for example, modify, cut, copy, paste, delete, clear, undo, redo, insert, replace, etc. As shown in FIG. 28D, the wearable system can enable the user to change the word 2808 to another word. The wearable system can support various user inputs to edit the word 2808, such as, for example, by receiving additional spoken input through a microphone to replace or delete the selected word, displaying a virtual keyboard to allow the user to type a replacement word, or receiving user input via a user input device. In some implementations, the inputs may be associated with a specific type of text editing. For example, a waving gesture may be associated with deleting selected text, while a gesture involving pointing with a finger at a location in the text may cause the wearable system to insert additional text at that location. The wearable system can also support combinations of user inputs to edit words. As will be further explained with reference to Figures 32-35, the system can support eye gaze in combination with other input modes to edit words.

[0291] In the example of Figures 28D and 28E, the system may automatically present the user with an array of suggested alternatives, such as alternatives 2810a and 2810b, in response to selection of word 2808. The suggested alternatives may be generated by an ASR engine or other language processing engine in the system and may be based on the original speech input (which may also be referred to as voice input in some embodiments), natural language understanding, context learned from user behavior, or other suitable sources. In at least some embodiments, the suggested alternatives may be alternative hypotheses generated by the ASR engine, hypotheses generated by a predictive text engine (which may attempt to "fill in the blanks" using the context of adjacent words and the user's historical patterns of text), homonyms of the original transformation, may be generated using a thesaurus, or may be generated using other suitable techniques. In the illustrated example, the suggested alternatives for "quarter" include "corner" and "courter," which may be provided by the language engine as words that are similar in sound to "quarter."

[0292] FIG. 28E illustrates how the system may enable a user to select a desired alternative word, such as "corner" using eye gaze. The wearable system may select the alternative word using similar techniques, such as those described with reference to FIG. 28C. For example, the system may track the user's eyes using an inward-facing imaging system 462 and determine that the user's gaze 2812 has been focused on a particular alternative, i.e., alternative 2810A or "corner", for at least a threshold time. After determining that the user's gaze 2812 has been focused on an alternative for a threshold time, the system may revise the text (message) by replacing the originally selected word with the selected alternative word 2814, as shown in FIG. 28F. In some implementations where the wearable system uses cone casting to select words, the wearable system may dynamically adjust the size of the cone based on the density of the text. For example, the wearable system may present a cone with a larger aperture (and therefore with a larger surface area away from the user) as shown in Figure 28E to select an alternative word for editing because there are several available options, but the wearable system may present a cone with a smaller aperture in Figure 28C to select word 2808 because it is surrounded by other words, and the smaller cone may reduce the error rate of accidentally selecting another word.

[0293] The wearable system can provide feedback (e.g., visual, auditory, tactile, etc.) to the user throughout the course of the action. For example, the wearable system can present a focus indicator to facilitate the user's recognition of the target virtual object. For example, as shown in FIG. 28E, the wearable system can provide a contrasting background 2830 around the word "quarter" to indicate that the word "quarter" has been selected and that the user is currently editing the word "quarter". As another example, as shown in FIG. 28F, the wearable system can change the font of the word "corner" 2814 (e.g., to a bold font) to indicate that the wearable system has confirmed the replacement of the word "quarter" with the present alternative word "corner". In other implementations, the focus indicator can include crosshairs, a circle or oval surrounding the selected text, or other graphical techniques to highlight or emphasize the selected text. Example of Word Selection Using Multimodal User Input

[0294] The wearable system can be configured to support and utilize multiple modes of user input to select words. Figures 29-31 illustrate an example of selecting a word based on a combination of eye gaze and another input mode. However, in other examples, input other than eye gaze can also be used in combination with another input mode to interact with text.

[0295] FIG. 29 illustrates an example of selecting a word based on input from a user input device and line of sight. As shown in FIG. 29, the system may combine the user's line of sight 2900 (which may be determined based on data from the inward-facing imaging system 462) with user input received via the user input device 466. In this example, the wearable system may perform cone casting based on the user's gaze direction. The wearable system may confirm the selection of the word "quarter" based on input from the user input device. For example, the wearable system may identify that the word "quarter" is the word closest to the user's gaze direction, and the wearable system may confirm that the word "quarter" was selected based on the user's actuation of the user input device 466. As another example, the cone casting may capture multiple words, such as, for example, "around," "7:00," "the," and "quarter." The user may select a word from among the multiple words for further editing via the user input device 466. By receiving input independent of the user's gaze, the system may not have to wait long before confidently identifying a particular word as one the user desires to edit. After selecting and thus editing a word, the system may present alternatives (as discussed in connection with FIG. 28E) or otherwise allow the user to edit the selected word. The same process of combining the user's gaze and user input received via the totem may be applied to the step of selecting a desired replacement word (e.g., selecting the word "corner" from among the alternatives to replace the word "quarter"). Some implementations may utilize a confidence score to determine the text that has been selected by the user. The confidence score may aggregate multiple input modalities and provide a better determination of the selected text.For example, the confidence score may be based on how long the user gazed at the text, whether the user actuated the user input device 466 while gazed at the text, whether the user pointed towards the selected text, etc. If the confidence score reaches a threshold, the wearable system may determine, with increased confidence, that the system correctly selected the user's desired text. For example, to select text using only eye gaze, the system may be configured to select the text if the gaze time exceeds 1.5 seconds. However, if the user only gazes at the text for 0.5 seconds but simultaneously actuates the user input device, the system may be able to more quickly and confidently determine the selected text, which may improve the user experience.

[0296] FIG. 30 illustrates an example of selecting a word for editing based on a combination of voice and gaze input. The wearable system can determine a target virtual object based on the user's gaze. As shown in FIG. 30, the system may determine that the user's gaze 3000 is directed at a particular word (in this case, "quarter"). The wearable system can also determine an action to be performed on the target virtual object based on the user's voice command. For example, the wearable system may receive the user's spoken input 3010 via the audio sensor 232, recognize the spoken input 3010 as a command, combine the two user inputs into a command, and apply the command action ("edit") to the target virtual object (e.g., the word ("quarter") on which the user is focusing his gaze). As mentioned above, the system may present alternative words after the user selects a word for editing. The same process of combining the user's gaze and spoken input may be applied to select a desired replacement word from among the alternative words to replace the word "quarter". As described herein, a term such as "edit" represents a context-specific wake-up word that serves to invoke a constrained system command library associated with editing for each of one or more different user input modalities. That is, such a term, when received as spoken input by the system, may cause the system to evaluate a subsequently received user input against a limited set of references to recognize edit-related commands provided by the user with improved accuracy. For example, in the context of spoken input, the system may consult a limited command-specific vocabulary of terms and perform speech recognition on the subsequently received spoken input. In another example, in the context of gaze or gesture input, the system may consult a limited command-specific library of template images and perform image recognition on the subsequently received gaze or gesture input.Terms like "edit" are sometimes referred to as "hot words" or "carrier phrases," and the system may include a number of pre-programmed (and optionally user-configurable) hot words, such as edit (in the editing context), cut, copy, paste, bold, italics, delete, move, etc.

[0297] FIG. 31 illustrates an example of selecting a word for editing based on a combination of gaze and gesture input. As illustrated in the example of FIG. 31, the system may use a gesture input 3110 together with an eye gaze input 3100 to select a word for editing. In particular, the system may determine the eye gaze input 3100 (e.g., based on data acquired by the inwardly facing imaging system 462) and identify the gesture input 3110 (e.g., based on images acquired by the outwardly facing imaging system 464). An object recognizer, such as the recognizer 708, may be used in detecting a part of the user's body, such as the hand, making a gesture associated with identifying the word for editing.

[0298] Gestures may be used alone or in combination with eye gaze to select a word. For example, cone casting may capture multiple words, but the wearable system may nevertheless identify the word "quarter" as the target virtual object because it is identified from both the cone casting and the user's hand gestures (e.g., a confidence score based on the eye gaze cone casting plus hand gestures exceeds a confidence threshold indicating that the user selected the word "quarter"). As another example, cone casting may capture multiple words, but the wearable system may nevertheless identify the word "quarter" as the target virtual object because it is identified from the cone casting and is the word with the lowest ASR score from the ASR engine that is within (or near) the cone casting. In some implementations, gestures may be associated with command actions, such as "edit" or other hot words described herein. As an example, the system may recognize when a user is pointing to the same word with their gaze and interpret these user inputs as a request to edit the same word. If desired, the system may also utilize additional user input, such as a voice command for "edit," at the same time as determining that the user wishes to edit a particular word. Example of editing a word using multimodal user input

[0299] Once a user selects a word for editing, the system can utilize any desired mode of user input to edit the selected word. The wearable system can allow the user to modify or replace the selected word by displaying a list of potential alternatives and receiving user gaze input 2812 to select an alternative word to replace the original word (see example illustrated in FIG. 28E). FIGS. 32-34 illustrate additional examples of editing a selected word, where the selected word can be edited using multimodal input.

[0300] FIG. 32 illustrates an example of replacing words based on a combination of eye gaze and speech input. In FIG. 32, the system receives speech input 3210 from a user (through audio sensor 232 or other suitable sensor). The speech input 3210 may contain a desired replacement word (which may or may not be a replacement word from the list of suggested replacements 3200). In response to receiving the speech input 3210, the wearable system may parse the input (e.g., to retrieve a carrier phrase such as "change this to..."), identify the word spoken by the user, and replace the word "corner" as spoken by the user with the selected word "quarter." In this example, the replacement is a word, but in some implementations the wearable system may be configured to replace the word "quarter" with a phrase or a sentence or some other element (e.g., an emoji). In embodiments where multiple words are contained within the eye gaze cone casting, the wearable system may automatically select the word within the eye gaze cone that is closest to the replacement word (e.g., "quarter" is closer to "corner" than "the" or "7:00").

[0301] FIG. 33 illustrates an example of modifying a word based on a combination of voice and gaze input. In this example, the wearable system can receive speech input 3310 and determine a user's gaze direction 3300. As shown in FIG. 33, the speech input 3310 includes the phrase "change it to 'corner'". The wearable system can analyze the speech input 3310 and determine that the speech input 3310 includes a command action "change" (which is an example of a carrier phrase), a target "it", and parameters of the command (e.g., the resulting word "corner"). This speech input 3310 can be combined with the eye gaze 3300 to determine the target of the action. As described with reference to FIGS. 28A and 28B, the wearable system can identify the word "quarter" as the target of the action. Thus, the wearable system can modify the target ("quarter") to the resulting word "corner".

[0302] FIG. 34 illustrates an example of editing a selected word 3400 using a virtual keyboard 3410. The virtual keyboard 3410 can be controlled by user gaze input, gesture input, input received from a user input device, and the like. For example, a user may type a replacement word by moving an eye gaze direction 3420 across a virtual keyboard 3410 displayed to the user by a display of the wearable system 200. The user may type each letter in the replacement word by pausing their gaze across individual keys for a threshold time period, or the wearable system may recognize a change in the direction of the user's gaze 3420 across a particular key as an indication that the user desires to select that key (thereby eliminating the need for the user to hold their focus stationary on each individual key as they type the word). As described with reference to FIG. 28D, in some implementations, the wearable system may vary the size of the cone based on the size of the key. For example, in a virtual keyboard 3410 where the size of each key is relatively small, the wearable system may reduce the size of the cone to allow the user to more accurately identify letters in a replacement word (so that cone casting will not accidentally capture multiple possible keys). If the size is relatively large, the wearable system can increase the size of the keys accordingly so that the user does not need to precisely indicate the gaze direction (which can reduce fatigue).

[0303] In some implementations, after a word is selected, the wearable system may present a set of possible actions in addition to or as an alternative to displaying a list of suggested alternative words to replace the selected word. The user 210 may select an action and edit the selected word using techniques described herein. FIG. 35 illustrates an exemplary user interface displaying possible actions to apply to the selected word. In FIG. 35, in response to selecting a word 3500 for editing, the wearable system may present a list 3510 of options for editing, including (in this example) options to: (1) change the word (using any of the techniques described herein for editing), (2) cut the word and, optionally, store it in a clipboard or copy the word and store it in a clipboard, or (3) paste the word or phrase from the clipboard. Additional or alternative options that may be presented include options to delete the selection, redo, undo, select all, insert here, and replace. The various options may be selected using gaze input, totem input, gesture input, etc., as described herein. Example of Interacting with Phrases Using Multimodal User Input

[0304] While the preceding examples have described steps for selecting and editing words using multi-modal input, this is intended for purposes of illustration, and the same or similar processes and inputs may be used in selecting and editing phrases or sentences or paragraphs, generally containing multiple words or characters.

[0305] 36(i)-36(iii) illustrate examples of interacting with a phrase using multimodal input. In FIG. 36(i), the wearable system may determine a user's gaze 3600 direction and perform cone casting based on the user's gaze direction. In FIG. 36(ii), the system may recognize that the user's 210 gaze 3600 is focused on a first word 3610 (e.g., "I'm"). The system may make such a determination of the first word 3610 using any of the techniques discussed herein, including, but not limited to, recognizing that the user's gaze 3600 rests (e.g., dwells) on a particular word for a threshold time period, that the user is providing a voice, gesture, or totem input at the same time that the user's gaze 3600 is on a particular word, etc. The wearable system may also display a focus indicator (e.g., a contrasting background, as shown) over the selected word "I'm" 3610 to indicate that the word was determined from eye gaze cone casting. While viewing the first word 3610, the user may activate a totem 3620 (an example of a user input device 466). This activation may indicate that the user intends to select a phrase or sentence starting with the first word 3610.

[0306] In FIG. 36(iii), after actuation of the user input device 466, the user can see the last intended word (e.g., the word "there") and indicate that the user desires to select a phrase starting with the word "I'm" and ending with the word "there". The wearable system can also detect that the user has stopped actuating the totem 3620 (e.g., the user releases a previously pressed button), thus selecting the entire range 3630 of the phrase "I'm flying in from Boston and will be there". The system can indicate the selected phrase using a focus indicator (e.g., by extending a contrasting background to all words in the phrase).

[0307] The system may use various techniques to determine that a user desires to select a phrase rather than another word for editing. As an example, the system may determine that a user desires to select a phrase rather than undoing the selection of the first word when the user selects a second word shortly after the user selects a first word. As another example, the system may determine that a user desires to select a phrase when the user selects a second word that appears after the first word, and that the user has not yet edited the first selected word. As yet another example, a user may press a button on the totem 3620 when their gaze is focused on the first word 3610, and then hold the button until their gaze settles on the last word. When the system recognizes that the button was pressed when the gaze 3610 was focused on the first word, but released only after the user's gaze 3610 shifted to the second word, the system may recognize the multi-modal user input as a selection of a phrase. The system may then identify all of the words in the phrase, including the first word, the last word, and all words in between, and may allow editing of the phrase as a whole. The system may use a focus indicator to highlight the selected phrase so that it stands out from the non-selected text (e.g., highlight, emphasized text (e.g., bold or italic or a different color), etc.). The system may then contextually display appropriate options for editing the selected phrase, such as options 3510, a virtual keyboard, such as keyboard 3410, alternative phrases, etc. The system may receive additional user input, such as spoken input, totem input, gesture input, etc., to determine how to edit the selected phrase 3630.

[0308] 36 illustrates a user selecting the first word 3610 at the beginning of a phrase, although the system may also allow the user to select backwards from the first word 3610. In other words, a user may select a phrase by selecting the last word of the phrase (e.g., "there") and then the first word of the desired phrase (e.g., "I'm").

[0309] 37A-37B illustrate another example of interacting with text using multimodal input. In FIG. 37A, the user 210 utters a sentence ("I want to sleep"). The wearable system can capture the user's utterance as speech input 3700. For this speech input, the wearable system can display both primary and secondary results from an automatic speech recognition (ASR) engine on a word-by-word basis, as shown in FIG. 37B. The primary results on a word-by-word basis may represent the ASR engine's best guess for the word spoken by the user in the speech input 3700 (e.g., the word with the highest ASR score for representing the word the user actually spoke), while the secondary results may represent similar-sounding alternatives or words with lower ASR scores than the ASR engine's best guess. In this FIG. 37B, the primary results are displayed as a sequence 3752. In some embodiments, the wearable system may present alternative results or hypotheses as alternative phrases and / or entire sentences, as opposed to alternative words. As an example, the wearable system may provide a primary result "four score and seven years ago" along with a secondary result "force caring seven years to go," where there is no one-to-one correspondence between the discrete words in the primary and secondary results. In such an embodiment, the wearable system may support input from a user (in any of the manners described herein) to select alternative or secondary phrases and / or sentences.

[0310] As shown in Figure 37B, each word from a user's speech input 3700 may be displayed as a collection of primary and secondary results 3710, 3720, 3730, 3740. This type of arrangement may allow a user to quickly swap out incorrect primary results and any correct errors introduced by the ASR engine. The primary results 3752 may be highlighted with a focus indicator (e.g., each word is bold text surrounded by a bounding box in the example in Figure 37B) to distinguish them from the secondary results.

[0311] The user 210 may pause on a secondary result, such as a secondary word, phrase, or sentence, if the primary word is not the one intended by the user. As an example, the primary result of the ASR engine in the collection 3740 is "slip," while the correct transcript is actually the first secondary result, "sleep." To correct this error, the user may focus their gaze on the correct secondary result, "sleep," and the system may recognize that the user's gaze has remained on the secondary result for a threshold time period. The system may translate the user gaze input as a request to replace the selected secondary result, "sleep," with the primary result, "slip." Additional user input may be received in conjunction with selecting a desired secondary result, such as user speech input (e.g., the user may ask the system to "edit," "use this," or "replace" while viewing the desired secondary result).

[0312] Once the user is finished editing the phrase "I want to sleep" or has verified that the transcript is correct, the phrase can be added to the body of text using any mode of user input described herein. For example, the user can utter a hot word such as "end" to cause the edited phrase to be added back to the body of text. Exemplary Process for Interacting with Text Using a Combination of User Inputs

[0313] 38 is a process flow diagram of an example method 3800 of interacting with text using multiple modes of user input. The process 3800 can be implemented by the wearable system 200 described herein.

[0314] In block 3810, the wearable system may receive spoken input from the user. The spoken input may include a user's speech containing one or more words. In one embodiment, the user may dictate a message and the wearable system may receive the dictated message. This may be accomplished through any suitable input device, such as the audio sensor 232.

[0315] In block 3820, the wearable system may convert the spoken input to text. The wearable system may utilize an automatic speech recognition (ASR) engine to convert the user's spoken input to text (e.g., a transcript), and may further leverage natural language processing techniques to convert such text into a semantic representation indicating intent and concepts. The ASR engine may be optimized for free-form text input.

[0316] In block 3830, the wearable system may tokenize the text into discrete actionable elements, such as words, phrases, or sentences. The wearable system may also display the text for the user using a display system, such as display 220. In some embodiments, the wearable system does not need to understand the meaning of the text during the tokenization system. In other embodiments, the wearable system is equipped with the ability to understand the meaning of the text (e.g., one or more natural language processing models or other probabilistic statistical models), or simply to distinguish between (i) words, phrases, and sentences that represent a user-configured message or part thereof, and (ii) words, phrases, and sentences that do not represent a user-configured message or part thereof, but instead correspond to commands to be executed by the wearable system. For example, the wearable system may need to grasp the meaning of the text in order to recognize a command action or parameters of a command spoken by the user. Examples of such text may include context-specific wake-up words, also referred to herein as hot words, that serve to invoke one or more constrained system command libraries associated with edits for one or more different user input modalities.

[0317] A user can interact with one or more of the actionable elements using a multi-modal user input. In block 3840, the wearable system can select one or more elements in response to a first indication. The first indication can be a user input or a combination of user inputs as described herein. The wearable system may receive input from the user selecting one or more of the elements of the text string for editing. The user may select a single word or multiple words (e.g., a phrase or sentence). The wearable system may receive user input selecting the elements for editing in any desired form, including, but not limited to, speech input, gaze input (e.g., via the inwardly facing imaging system 462), gesture input (e.g., as captured by the outwardly facing imaging system 464), totem input (e.g., via actuation of the user input device 466), or any combination thereof. As an example, the wearable system may receive user input in the form of a user's gaze lingering on a particular word for a threshold time period, or may receive user input via a microphone or totem simultaneously with the user's gaze on a particular word indicating selection of that particular word for editing.

[0318] In block 3850, the wearable system can edit the selected element in response to the second indication. The second indication can be received via a single input mode or a combination of input modes as described with the preceding figures, including, but not limited to, user gaze input, spoken input, gesture input, and totem input. The wearable system may receive user input indicating how the selected element should be edited. The wearable system may edit the selected element according to the user input received in block 3850. For example, the wearable system can replace the selected element based on spoken input. The wearable system can also present a list of suggested alternatives and select from among the selected alternatives based on the user's eye gaze. The wearable system can also receive input via user interaction with a virtual keyboard or via a user input device 466 (e.g., a physical keyboard or a handheld device, etc.).

[0319] In block 3860, the wearable system may display the results of editing the selected element. In some implementations, the wearable system may provide a focus indicator on the element being edited.

[0320] As indicated by arrow 3870, the wearable system may repeat blocks 3840, 3850, and 3860 if the user provides additional user input to edit additional elements of the text. Additional aspects

[0321] In a first aspect, a method for interacting with virtual content includes receiving a first direct user input, determining a first confidence score correlated with the first direct user input, receiving a second direct user input, determining a second confidence score correlated with the second direct user input, calculating an aggregate confidence score from at least the first confidence score and the second confidence score, and manipulating a target virtual object and satisfying a parameter condition.

[0322] In a second aspect, the method of aspect 1, wherein the first direct user input is selected from the group consisting of speech input, hand gestures, head pose, and eye gaze.

[0323] In a third aspect, the method of any one of aspects 1-2, wherein the second direct user input is selected from the group consisting of speech input, hand gestures, head pose, and eye gaze.

[0324] In a fourth aspect, the method of any one of aspects 1-3, wherein the parameter is a position of the target virtual object.

[0325] In a fifth aspect, the method of any one of aspects 1-4, wherein the parameter is a size of the target virtual object.

[0326] In a sixth aspect, the method of any one of aspects 1-5, wherein the parameter is a functionality of the target virtual object.

[0327] In a seventh aspect, a method for interacting with virtual content includes receiving a direct user input, determining a first confidence score correlated with the direct user input, receiving an indirect user input, determining a second confidence score correlated with the indirect user input, calculating an aggregate confidence score from at least the first confidence score and the second confidence score, and manipulating a target virtual object to satisfy a parameter condition.

[0328] In an eighth aspect, the method of aspect 7, wherein the direct user input is selected from the group consisting of speech input, hand gestures, head pose, and eye gaze.

[0329] In a ninth aspect, the method of aspect 8, wherein the indirect user input is a geographic location.

[0330] In a tenth aspect, the method of any one of aspects 7-8, wherein the parameter is a position of the target virtual object.

[0331] In an eleventh aspect, the method of any one of aspects 7-10, wherein the parameter is a size of the target virtual object.

[0332] In a twelfth aspect, the method of any one of aspects 7-11, wherein the parameter is a functionality of the target virtual object.

[0333] The method according to any one of aspects 1-12 above may be performed under the control of the wearable system 200 described herein.

[0334] In a thirteenth aspect, a system for interacting with objects for a wearable device includes a display system for the wearable device configured to present a three-dimensional (3D) view to a user and enable the user to interact with objects within the user's field of view (FOR), the FOR comprising a portion of the user's surrounding environment that can be perceived by the user through the display system, and a hardware processor in communication with a sensor and the display system, the hardware processor being programmed to receive multi-modal input for user interaction, analyze the multi-modal input, identify targets, commands, and parameters of the user interaction, and perform the user interaction based on the targets, commands, and parameters.

[0335] In a fourteenth aspect, the system of aspect 13 includes at least one of a microphone, an inertial measurement unit, an outwardly facing imaging system, or an inwardly facing imaging system.

[0336] In a fifteenth aspect, the system of aspect 14, wherein the multi-modal input comprises direct input comprising at least one of head pose, eye gaze, hand gesture, or speech input.

[0337] In a sixteenth aspect, the system of any one of aspects 13-14, wherein the multi-modal input comprises at least one of input from a user input device or indirect input, the indirect input comprising a location of a user or a location of an object within the user's environment.

[0338] In a seventeenth aspect, the system of any one of aspects 13-16, wherein the target comprises a target virtual object with which the user intends to interact, the command comprises an action to be performed by the wearable system on the target virtual object, and the parameters comprise characteristics of the action.

[0339] In an eighteenth aspect, the system of aspect 17, wherein the parameters include placement, orientation, and destination position of the target virtual object.

[0340] In a 19th aspect, the system of any one of aspects 13-18 is programmed to identify a target of user interaction, wherein the wearable device accesses a first value for a first input of the multi-modal input, accesses a second value for a second input of the multi-modal input, calculates a first confidence score for a first candidate virtual object based on the first or second value, calculates a second confidence score for a second candidate virtual object based on the first or second value, and sets the target to either the first candidate virtual object or the second candidate virtual object based on a comparison of the first confidence scores.

[0341] In a twentieth aspect, the system of aspect 19, wherein the first input is associated with a first weighting and the second input is associated with a second weighting, and the first value and the second value are assigned based on the first weighting and the second weighting, respectively.

[0342] In a twenty-first aspect, the computer processor is further configured to detect a start condition associated with a user interaction, the start condition comprising a call phrase or a hand gesture, the system being described in any one of aspects 13-20.

[0343] In a twenty-second aspect, the system of aspect 21, wherein the starting condition comprises a combination of two input modes, the two input modes consistently indicating user interaction.

[0344] In a twenty-third aspect, the system of any one of aspects 13-22, wherein the processor is further programmed to receive environmental data from an environmental sensor of the wearable device and automatically set an authentication level associated with the input mode based on the environmental data.

[0345] In a twenty-fourth aspect, the system of any one of aspects 13-23, wherein the user interaction comprises at least one of selecting or moving a virtual object, conducting a telepresence session, modifying the virtual object, or sharing the virtual object with another wearable device.

[0346] In a twenty-fifth aspect, the system of any one of aspects 13-24, wherein the multi-modal input comprises head pose and speech input.

[0347] In a twenty-sixth aspect, the system of aspect 25, wherein the multi-modal input further comprises gestures or eye gaze.

[0348] In a twenty-seventh aspect, a system includes a first sensor configured to obtain first user input data in a first input mode, a second sensor configured to obtain second user input data in a second input mode, the second input mode being different from the first input mode, and a hardware processor in communication with the first and second sensors. The system may be a wearable system for interacting with an object, and the first and second sensors may be part of the wearable system. The hardware processor of the system can be programmed to receive a multi-modal input comprising first user input data in a first input mode and second user input data in a second input mode, identify a first set of candidate objects for interaction based on the first user input data, identify a second set of candidate objects for interaction based on the second user input data, determine a target virtual object from the first set of candidate objects and the second set of candidate objects based on a combination of the first user input data and the second user input data, determine a user interface action on the target virtual object based on at least one of the first user input data or the second user input data, and generate a multi-modal input command that causes the user interface action to be performed on the target virtual object.

[0349] In a twenty-eighth aspect, the system of aspect 27, wherein the multi-modal input comprises at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice.

[0350] In a 29th aspect, the system of aspect 27 or 28, wherein the user interface action comprises at least one of selecting, moving, or resizing the target virtual object.

[0351] In a 30th aspect, the hardware processor is further configured to determine at least one of a target location, orientation, or movement for a target virtual object in a user interface operation, the system being described in any one of aspects 27-29.

[0352] In a thirty-first aspect, the system described in aspect 30, wherein to determine a target location for the target virtual object, the hardware processor is programmed to identify a workable surface within the physical environment for placing the target virtual object.

[0353] In a thirty-second aspect, the system of aspect 31, wherein the workable surface is identified by calculating a distance function for points of interest (POIs) on physical objects in the physical environment, eliminating one or more of the POIs that are outside of a planar tolerance, and outlining the workable surface based on the remaining POIs.

[0354] In a thirty-third aspect, the system of aspect 31 or 32, wherein the hardware processor is programmed to automatically orient the target virtual object to match an orientation of the target location.

[0355] In a thirty-fourth aspect, the system of any one of aspects 27-33, wherein the operation is determined based on first user input data in a first input mode, and at least one of the objects or parameters is determined based on a combination of the first input mode and the second input mode.

[0356] In a thirty-fifth aspect, the system of any one of aspects 27-34, wherein the first input mode includes an indirect input mode based on location information of a user of the wearable system.

[0357] In a thirty-sixth aspect, the system of aspect 35, wherein the hardware processor is programmed to identify the virtual object as a target virtual object from the first set of objects and the second set of objects in response to determining that the object is within a threshold range of the user.

[0358] In a thirty-seventh aspect, the system described in aspects 27-36, wherein the user interface operation is associated with a virtual application, and the virtual application is programmed to be more responsive to one of the first sensor or the second sensor.

[0359] In a thirty-eighth aspect, the system described in aspects 27-37, wherein to determine a target virtual object from the first set of candidate objects and the second set of candidate objects, the hardware processor is programmed to perform a tree-based analysis on the first set of candidate objects and the second set of candidate objects based on the first user input data and the second user input data.

[0360] In a thirty-ninth aspect, the system of aspect 38 is programmed to determine a target virtual object from the first set of candidate objects and the second set of candidate objects, wherein the hardware processor is programmed to: calculate a first confidence score for a candidate object in the first set of candidate objects based on first user input data, calculate a second confidence score for the candidate object based on second user input data, calculate an aggregated score for the candidate object from at least the first confidence score and the second confidence score, and set the candidate object as the target virtual object in response to determining that the aggregated score satisfies a threshold condition.

[0361] In a fortieth aspect, the system of any one of aspects 27-39 is programmed to calculate a confidence score for the virtual object by calculating at least one of: uniformity of space around the virtual object in the field of view; a proportional area for a first portion of the virtual object within the user's field of view relative to a second portion of the virtual object that is outside the user's field of view; or a historical analysis of the user's interactions with the virtual object, to determine the target virtual object.

[0362] In a forty-first aspect, the system of any one of aspects 27-40, wherein the hardware processor is further programmed to detect a start condition for an interaction event based on the multi-modal input, triggering the hardware processor to determine a target virtual object and user interface action.

[0363] In a forty-second aspect, the system of aspect 41, wherein the initiation condition comprises a trigger phrase.

[0364] In a forty-third aspect, the first input mode is a primary input mode, the second input mode is a secondary input mode, and the hardware processor is programmed to resolve ambiguity in at least one of the target virtual object and the user interface action based on the second user input data. The system of any one of aspects 27-42.

[0365] In a forty-fourth aspect, the system of any one of aspects 27-43, wherein the first user input data comprises a deictic or anaphoric reference to the virtual object, and the hardware processor is programmed to identify the virtual object as a target virtual object of interest based on the second user input data.

[0366] In a forty-fifth aspect, the hardware processor is further programmed to automatically enable, disable, or adjust the sensitivity of the first input mode, the second input mode, or both, based at least in part on user settings or the user's environment, in the system described in aspects 27-44.

[0367] In a 46th aspect, the hardware processor is programmed to identify a target virtual object outside the user's field of view based, at least in part, on the multi-modal input, and automatically move the virtual object into the field of view for user interaction, a system as described in aspects 27-45.

[0368] In a forty-seventh aspect, a method is described. The method can be for interacting with an object and can be implemented under control of a hardware processor of a wearable system in communication with a plurality of sensors configured to obtain user input data. The method can include receiving user input data from the plurality of sensors regarding a user-environment interaction event, analyzing the user input data and identifying a multi-modal input for interacting with the environment, the multi-modal input comprising a first input in a first input channel and a second input in a second input channel, determining a multi-modal input command comprising one or more of an object, a parameter, or an action to describe the interaction event with the environment based on the first input and the second input, the multi-modal input command comprising one or more of an object, a parameter, or an action, the at least one of the object, the parameter, or the action being identified based on a combination of the first input and the second input, and causing the wearable device to execute the multi-modal input command for the interaction event.

[0369] In a forty-eighth aspect, the method of aspect 47, wherein the action comprises at least one of selecting, moving, or resizing the object, the object comprising a target virtual object with which the user intends to interact, or the parameters comprise at least a target location, orientation, or movement of the object.

[0370] In a forty-ninth aspect, the method of aspect 48, wherein the target location is determined by calculating a distance function for points of interest (POIs) on the physical object, eliminating one or more of the POIs that are outside a given planar tolerance, and outlining a workable surface on the physical object based on the remaining POIs.

[0371] In a 50th aspect, the method of aspect 48 or 49 includes a step of automatically orienting the virtual object to match that of the target location.

[0372] In a fifty-first aspect, the method of any one of aspects 47-50, wherein the multi-modal input comprises at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice.

[0373] In a fifty-second aspect, the method of aspect 51, wherein the first input channel is speech and the second input channel is head pose or hand gesture.

[0374] In a 53rd aspect, the method of any one of aspects 47-52, wherein at least one of the actions, objects, or parameters is further identified based on the user's environment or location information.

[0375] In a fifty-fourth aspect, the method of aspect fifty-three, wherein the object is selected from a group of objects within a threshold range of a user.

[0376] In a 55th aspect, a method according to any one of aspects 47-54, wherein the interaction event occurs within a virtual application, and the virtual application is programmed to be more responsive to one of the first input channel or the second input channel.

[0377] In a 56th aspect, the method of any one of aspects 47-55, wherein the target is identified by analyzing the first input and the second input using a lattice tree analysis and identifying a virtual object to set as the target.

[0378] In aspect 57, the method of any one of aspects 47-56, wherein the target or parameter in the multi-modal input command is determined by calculating, for a candidate object or parameter, a first confidence score correlating with the first input and a second confidence score correlating with the second input, calculating an aggregated score for the candidate object or parameter from at least the first confidence score and the second confidence score, and setting the candidate object or parameter, respectively, as the target or parameter for the interaction event based, at least in part, on the aggregated score.

[0379] In a fifty-eighth aspect, the candidate virtual object is identified as being of interest based on a confidence score of the virtual object in the user's field of view, the confidence code being calculated based on at least one of a uniformity of space around the candidate virtual object in the field of view, a proportional area of ​​a first portion of the candidate virtual object in the user's field of view relative to a second portion of the candidate virtual object outside the user's field of view, or a historical analysis of the user's interactions with the candidate virtual object.

[0380] In a fifty-ninth aspect, the method of any one of aspects 47-58 further includes detecting a start condition for an interaction event based on data received from one or more sensors of the plurality of sensors, the start condition triggering the receiving, analyzing, determining, and generating steps.

[0381] In a 60th aspect, a first input from a first input channel is a primary input and a second input from a second input channel is a secondary input, and the method includes the steps of parsing the first input and identifying objects, parameters, and actions, and resolving ambiguities in at least one of the objects, parameters, or actions based on the second input, and generating a multi-modal input command, a method according to any one of aspects 47-59.

[0382] In a 61st aspect, the method described in any one of aspects 47-60, further comprising a step of automatically enabling, disabling, or adjusting sensitivity of an input channel based, at least in part, on user settings or the user's environment.

[0383] In a 62nd aspect, the method of any one of aspects 47-61 further includes identifying a virtual object outside the user's field of view as a target based, at least in part, on user interaction, and automatically moving the virtual object into the field of view for user interaction.

[0384] In a 63rd aspect, the method of any one of aspects 47-62, wherein the first input comprises a deictic or anaphoric reference to the target, and the method further comprises a step of selecting a target object as the target based on the second input.

[0385] In a sixty-fourth aspect, a system includes a head-mounted display (HMD) configured to present three-dimensional (3D) virtual content to a user, two or more user input components configured to receive user input of respective modalities, one of the user input components including an audio sensing device configured to capture sound, and a hardware processor communicatively coupled to the display and the two or more user input components. The HMD may be part of a wearable system. The hardware processor may be programmed to receive, from the audio sensing device, speech data encoding an utterance of one or more words spoken by the user; obtain a transcript for the one or more words spoken by the user based on at least the received speech data; control a display to present to the user a string of text characters representing the obtained transcript; receive user input data indicative of user input of another, different modality from another of the two or more user input components; determine that the user input data received from the other user input component represents a command to select a particular subset of the text characters for editing; and, in response to determining that the user input data received from the other user input component represents a command to select a particular subset of the text characters for editing, determine whether subsequently received data from any of the two or more user input components represents a command to modify the particular subset of the text characters in a particular manner.

[0386] In a sixty-fifth aspect, the system of aspect 64, wherein the other user input component comprises an eye gaze tracking device configured to obtain data indicative of the user's eye gaze direction.

[0387] In aspect 66, the hardware processor is further programmed to determine, based on at least the data received from the eye tracking device, that the user has fixated the particular subset of text characters for a period of time longer than a threshold, and in response to determining that the user has fixated the particular subset of text characters for a period of time longer than the threshold, determine that the user input data received from the other user input component represents a command to select the particular subset of text characters for editing.

[0388] In a 67th aspect, the hardware processor is further pro...

Claims

1. 1. A system comprising: a head mounted display (HMD) of a wearable system configured to present three-dimensional (3D) virtual content to a user; two or more user input components, the two or more user input components comprising at least an audio sensing device configured to capture sound, an eye gaze tracking device configured to detect eye gaze direction, and a gesture tracking device configured to obtain data indicative of hand gestures of the user; a hardware processor communicatively coupled to the head mounted display and to the two or more user input components; Equipped with The hardware processor includes: receiving speech data from the audio sensing device encoding an utterance of one or more words spoken by the user; obtaining a transcript of the one or more words spoken by the user based at least on the received speech data; and controlling the head mounted display to present to the user a string of text characters representing the captured transcript; receiving an eye gaze direction from the eye gaze tracking device; receiving hand gesture data from the gesture tracking device; determining that the user performed one or more predefined hand gestures while fixating on a particular subset of the text characters based at least on the eye gaze direction received from the eye gaze tracking device and the hand gesture data received from the gesture tracking device; determining that the eye gaze direction and the hand gesture data represent a command to select the particular subset of the text characters for editing in response to determining that the user has performed the one or more predetermined hand gestures while fixating on the particular subset of the text characters; determining whether subsequently received data from any of the two or more user input components represents a command to modify the particular subset of the text characters in a particular manner; The system is programmed to:

2. The hardware processor further comprises: determining, based at least on data received from the eye tracking device, that the user is fixating on the particular subset of text characters for a period longer than a threshold duration; determining that the hand gesture data received from a user input component represents the command to select the particular subset of the text characters for editing in response to determining that the user has fixated the particular subset of the text characters for longer than a threshold duration; The system of claim 1 , programmed to:

3. The hardware processor further comprises: receiving additional speech data from the audio sensing device encoding an utterance of a phrase spoken by the user; determining, based at least on the data received from the eye tracking device and the additional speech data received from the audio sensing device, that the user uttered one or more predefined hot words while fixating on the particular subset of text characters; determining that the data received from the eye tracking device and the additional speech data received from the audio sensing device represent the command to select the particular subset of the text characters for editing in response to determining that the user has uttered one or more predefined hot words while fixating on the particular subset of the text characters; The system of claim 1 , programmed to:

4. the two or more user input components further comprise a touch-sensitive device configured to obtain data indicative of a physical interaction therewith; The hardware processor further comprises: receiving data indicative of the eye gaze direction from the eye gaze tracking device; receiving data from the touch sensitive device indicative of the physical interaction with the touch sensitive device; determining whether the user provided one or more predefined touch inputs while fixating on the particular subset of text characters based at least on the data received from the eye tracking device and the data received from the touch sensitive device; in response to determining that the user has provided one or more predefined touch inputs while fixating on the particular subset of the text characters, determining that the data received from the eye tracking device and the touch sensitive device represents the command to select the particular subset of the text characters for editing; The system of claim 1 , programmed to:

5. The system of claim 1 , wherein the hardware processor is programmed to implement an automatic speech recognition (ASR) engine to obtain the transcript.

6. 6. The system of claim 5, wherein the ASR engine is configured to generate a score associated with one or more words in the string of text that indicates a likelihood that the ASR engine correctly transcribed the one or more words.

7. The system of claim 6 , wherein the hardware processor is further programmed to cause the HMD to highlight words whose likelihood of being a correct transcription is below a threshold level.

8. 1. A system comprising: a display of the wearable system configured to present virtual content to a user; an audio sensing device configured to capture words spoken by the user and generate speech data; an eye gaze tracking device of the wearable system configured to track the gaze of the user; a hardware processor communicatively coupled to the display, the audio sensing device, and the eye tracking device; Equipped with The hardware processor includes: obtaining a text transcription of one or more words spoken by the user based at least in part on the speech data from the audio sensing device; and controlling the display to present the text to the user; determining that the user has provided a command to select the portion of the presented text for editing based on data from the eye-tracking device and data from another user input device indicating that a user input device received user input while the user's gaze was focused on the portion of the presented text presented by the display; performing an editing action on the portion of the presented text; and The system is programmed to:

9. 10. The system of claim 8, wherein the hardware processor is further programmed to determine that the user has given the command to select a word from the one or more words of the presented text for editing based on data from the eye tracking device indicating that the user's gaze remained on the portion of the presented text presented by the display for at least a threshold duration.

10. 10. The system of claim 8, wherein the hardware processor is programmed to determine that the user has given the command to select the portion of the presented text for editing based on data from the audio sensing device and data from the eye gaze tracking device indicating that the audio sensing device received a voice command while the user's gaze was focused on the portion of the presented text presented by the display.

11. 10. The system of claim 8, further comprising an imaging system that images at least one hand of the user, and wherein the hardware processor is configured to determine that the user has given the command to select the portion of the presented text for editing based on data from the imaging system and data from the eye tracking device indicating that the user made a command gesture with their hand while the user's gaze was focused on the portion of the presented text presented by the display.

12. 10. The system of claim 8, wherein the hardware processor is further programmed to, in response to the command to select a word from the one or more words of the presented text for editing, control the display to present an alternative transcript of the portion of the presented text.

13. The hardware processor further comprises: determining, based on additional data received from the eye tracking device, that the user has provided a command to replace the portion of the presented text with a selected alternative transcript; and revising the text to replace the portion of the presented text with the selected alternative transcript; controlling the display to present the revised text to the user; The system of claim 8 , programmed to:

14. 10. The system of claim 8, wherein the hardware processor is further programmed to generate a score associated with one or more words in the text that indicates a likelihood that the one or more words were correctly transcribed.

15. 15. The system of claim 14, wherein the hardware processor is further programmed to cause the display to highlight words whose likelihood of being a correct transcript is below a threshold level.

16. 1. A method comprising: Under the control of a hardware processor, Receiving spoken input from a user via a microphone; converting the spoken input into text comprising a plurality of words; causing a wearable display to present the text to the user; determining a selection of the portion of the presented text based at least on the user's gaze being focused on a portion of the presented text within the presented text while receiving data from a gesture tracking system indicating that the user has made a predefined command gesture requesting editing; providing the user with an opportunity to edit the portion of the presented text; The method of claim 1,

17. receiving a selection of a word from the plurality of words included in the presented text based on at least one data from the gesture tracking system; providing the user with an opportunity to edit the portion of the presented text or a phrase formed from an additional portion of the text; 20. The method of claim 16, further comprising:

18. 17. The method of claim 16, wherein at least a portion of the presented text is highlighted on the display, the portion being associated with a low confidence that a conversion of the spoken input to a corresponding portion of the presented text is correct.

19. 1. A method comprising: receiving a multi-mode input, the multi-mode input comprising: a first user input from a component of the wearable device, the first user input being associated with a first mode of user input; and a second user input from an eye gaze tracking component of the wearable device configured to detect an eye gaze direction, the second user input being associated with a user input of a second mode; and and identifying a multi-mode input command based on the first user input and the second user input, the multi-mode input command comprising at least: an object comprising at least a portion of a text; a command action comprising an interaction with the text; and executing the multi-modal input command and displaying the results of the multi-modal input to a user; A method comprising:

20. The object comprises a sentence, and the command action comprises selecting the sentence for editing, the method further comprising: performing a first user interface action including extracting the sentence from a body of text; processing the sentence to generate a sequence of tokens; displaying the sequence of tokens as a primary result, the primary result comprising the sentence as transcribed from the user's utterance; and further displaying a secondary result, the secondary result comprising alternative words for tokens in the sequence; receiving another multi-modal input comprising a third mode of input and a fourth mode of input for editing the sequence word by word; responsive to an indication that editing of the sequence is complete, inserting the edited sequence back into the body of the text; 20. The method of claim 19, comprising:

Images