Watch Timer - based Extension of the Functionality of User Input Devices

Abstract

A wearable computing system (100) including a head-mounted display (110) implements a gaze timer feature to enable a user to temporarily expand the functionality of a handheld controller or other user input device (104). In one embodiment, when a user gazes at or near the handheld controller (104) for a predetermined period of time, the functionality of one or more input elements (130) of the handheld controller (104) is temporarily modified. For example, a function associated with a particular controller button (130) may be modified to enable a user to use the button to open a particular menu (506).

Description

Technical Field

[0001] The present disclosure relates to augmented reality and mixed reality systems, and more particularly, to user interfaces and processes for augmenting the functionality of handheld controllers or other user input devices in such systems.

Background Art

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", and "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user so as to appear as if they were real. Virtual reality, i.e., "VR" scenarios, typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality, i.e., "AR" scenarios, typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the actual world surrounding the user. Mixed reality, i.e., "MR" scenarios, typically involve fusing the real and virtual worlds to produce a new environment in which physical and virtual objects coexist and interact in real time.

[0003] Some systems, such as those available from Magic Leap, include a headset and a handheld controller. The headset typically includes a tracking camera for mapping the user's environment, an inward-facing eye-tracking camera, and a lens with a waveguide that forms an inward-facing display. The headset may also include one or more other types of sensors. The controller typically includes one or more types of input elements, such as buttons and / or touch pads, that are used to control an MR or AR session.

[0004] A system that supports MR and AR sessions may also include a separate computing device, such as a wearable device, connected to the headset by a cable. The computing device may execute various software components, such as an operating system and applications. One or more of these software components may map a particular controller input element or action to a particular function that can be invoked by the user. For example, the operating system may map a particular button on a handheld controller to a particular function, or map a specific touchpad action (such as a single or double tap action) to another function.

Summary of the Invention

Means for Solving the Problems

[0005] One problem with existing MR and AR systems is that the number of functions available to the user via a handheld controller or any other input device or object is limited by the number of user input buttons or surfaces included on the controller (or device or object), and / or by the number of user actions (such as single and double tap actions and gestures) that can be performed on such an input device. The functionality of the controller can sometimes be extended by using the session context to expose new controller-based functions, but in many cases, it is desirable to give the user an intuitive way to selectively extend the functionality of the controller.

[0006] Various use cases, such as networked co - experience for sharing app content and chatting with both local users and remote users represented as avatars located in the same place, can benefit from a flexible on - demand interface that does not get in the user's way. Such sessions may require the user to track different elements, such as avatars, shared content, menus, etc. The disclosure herein describes systems and methods that enable the user to access the session menu without diverting the user's attention from these other social activities and the communication goals of the overall experience. In particular, what is described below is, in some embodiments, a method for accessing a context menu without any existing UI or world - based criteria, so that the user can access the session user interface without the inconvenience of having to move around, collaborate and communicate with others, and still need to return to a specific world location (e.g., the session menu may be statically located). In one embodiment, the session user interface is configured to follow the user regardless of where they are located (using, for example, the systems and methods of U.S. Provisional Application No. 62 / 965,708, filed on January 24, 2020, entitled "CONTENT MOVEMENT AND INTERACTION USING A SINGLE CONTROLLER", the entire disclosure of which is incorporated herein by reference). Thus, important session tools (e.g., mute, block, invite, end) are always close to the user, discretely and in an easily accessible location.

[0007] Within these shared co - experience sessions, access to session tools (e.g., within a session menu) may be hidden until requested by the user through, for example, the gaze timer functionality discussed herein. This gaze timer functionality provides a way to extend the functionality of existing user input devices, without limiting the functionality of input devices available in the shared co - experience, and enables the user to request the session menu as needed.

[0008] The present disclosure addresses the above and other problems by providing a user interface feature (also referred to as a "gaze timer" or "focus timer" feature) that enables a user to temporarily modify the functionality of one or more buttons or surfaces of a handheld controller (or other user input device) by performing a defined visual or "gaze" action over a predetermined period of time. For example, in a preferred embodiment, when a user gazes at a handheld controller or a predetermined area within it for a predetermined period (such as 0.5 seconds or 1 second), the mapping of controller actions to functions is temporarily modified to make one or more additional functions or selections available, or to temporarily switch to an alternative mapping for one or more user input devices. The predetermined amount of time may be selected to reduce the likelihood of accidental activation. In some embodiments, the modified mapping remains in effect until the gaze action ends or the user performs a specific action (e.g., presses the button whose function has been modified).

[0009] As an example, the functionality associated with a particular button (or other user input element) of a handheld controller may be temporarily modified in response to a controller gaze action such that pressing the button causes the headset to display a menu that would otherwise not be displayable via the controller. Thus, a user desiring to display this menu can do so by gazing at the controller for a predetermined period of time and then pressing the button (optionally, regardless of the session context). The system may restore the functionality of the button when the user presses the button or when the user closes the menu. In some embodiments, the system may additionally or alternatively restore the functionality of the button when the user aborts the controller gaze action.

[0010] When the gaze action begins, the headset may display a "gaze timer" icon and / or another type of visual indicator to indicate that a controller gaze action has been detected. The appearance of this icon or visual indicator may then be updated substantially continuously to graphically depict the amount of time that the gaze action needs to continue before the functionality of the controller will be modified. Once the timer expires (i.e., reaches a predetermined gaze duration), the display may be updated to indicate that the functionality of the controller has been modified. For example, in the menu example above, a message may be briefly displayed to indicate that the user can display the menu by pressing a particular button.

[0011] The fixation timer may be implemented in software, executed by one or more components of a wearable computing system that supports MR and / or AR sessions. The wearable computing system may detect a fixation action based on head pose, eye pose, or both. In some embodiments, the target of the fixation action may be an object other than the controller. For example, the user may perform a hand fixation action over a predetermined period while performing a specific hand gesture or movement.

[0012] One aspect of the present disclosure is thus a computerized process implemented under the control of program instructions executed by one or more hardware processors. The process includes detecting a controller fixation event in which a user is fixating on a handheld controller or a defined area within it based on sensor data collected during a mixed reality or augmented reality session, starting a timer used to measure whether the duration of the controller fixation event meets a condition in response to detecting the controller fixation event to temporarily modify the functionality of the handheld controller, and modifying the functionality of a user input element of the handheld controller in response to a determination that the duration of the controller fixation event meets the condition. The sensor data may include head tracking data reflecting the user's head orientation, eye tracking data, controller position data reflecting the position of the controller, and / or data collected by a head-mounted camera. In some embodiments, modifying the functionality of the user input element includes mapping a button of the handheld controller to a function of displaying a menu.

[0013] The computerized process may also include one or more of the following, namely: (1) displaying a visual indication that a timer has been started on a display generated by a head-mounted display worn by a user; (2) updating the visual indication during a fixation event and indicating the amount of time that the fixation event needs to continue until the function of a user input element is modified; (3) detecting the end of a controller fixation event based on sensor data after modifying the function of a user input device, and in response to detecting the end of the controller fixation event, restoring the previous function of the user input element; and (4) displaying a visual indication of a new function assigned to a user input element in response to a determination that the duration of the controller fixation event meets a condition.

[0014] Another aspect of the present disclosure is a wearable computing system comprising a head-mounted display system having a display and one or more sensors, a user input device configured to receive user input, and one or more processors programmed to implement a process including detecting an object fixation event in which a user's posture is directed at a physical object or a predetermined area thereof, at least in part based on sensor data from the one or more sensors, determining whether a duration of the object fixation event satisfies a defined condition, and modifying a function of the user input device when the duration satisfies the condition. The physical object may be the user input device. The process may also include, hereinafter, namely, (1) causing the head-mounted display system to display a visual indication that an object fixation event has been detected, (2) updating the visual indication during the object fixation event and indicating an amount of time that the object fixation event needs to continue until the function of the user input device is modified, (3) detecting an end of the object fixation event after modifying the function of the user input device and restoring a previous function of the user input device in response to detecting the end of the object fixation event, (4) displaying a visual indication of a new function assigned to the user input device on the display in response to determining that the duration of the object fixation event satisfies the condition, (5) detecting the object fixation event based on the user's head posture regardless of the eye posture, and (6) including one or more of detecting a fixation event, at least in part based on the user's perceived eye posture. In some embodiments, modifying the function of the user input device includes causing buttons of the user input device to be mapped to a function of displaying a menu on the display.

[0015] Another aspect of the present disclosure is a computerized process, implemented under the control of program instructions, executed by one or more hardware processors, the process comprising detecting a fixation event in which a user is fixating on a physical object or a defined area thereof based on data collected during a mixed reality or augmented reality session; starting a timer used to measure whether the duration of the fixation event meets a condition for temporarily modifying the functionality of a user input device in response to the detection of the fixation event; and modifying the functionality of the user input device in response to a determination that the duration of the fixation event meets the condition. The present invention provides, for example, the following. (Item 1) A computerized process performed under the control of program instructions executed by one or more hardware processors, the process comprising: Detecting a controller gaze event in which a user is gazing at a handheld controller or within its defined area based on sensor data collected during a mixed reality or augmented reality session; In response to detecting the controller gaze event, starting a timer used to measure whether the duration of the controller gaze event meets a condition in order to temporarily modify the functionality of the handheld controller; In response to determining that the duration of the controller gaze event meets the condition, modifying the functionality of the user input element of the handheld controller A process including. (Item 2) The process according to item 1, further comprising displaying a visual indication on a display generated by a head-mounted display worn by the user that the timer has been started. (Item 3) The process according to item 2, further comprising updating the visual indication during the gaze event and indicating the amount of time the gaze event needs to continue until the functionality of the user input element is modified. (Item 4) After modifying the functionality of the user input device, Detecting the end of the controller gaze event based on the sensor data; Restoring the previous functionality of the user input element in response to detecting the end of the controller gaze event The process according to item 1, further including. (Item 5) The process according to item 1, wherein modifying the functionality of the user input element includes mapping a button of the handheld controller to a function of displaying a menu. (Item 6) The process according to item 1, further comprising displaying a visual indication of a new function assigned to the user input element in response to determining that the duration of the controller gaze event meets the condition. (Item 7) The process according to item 1, wherein the sensor data comprises at least head tracking data reflecting the head orientation of the user. (Item 8) The sensor data is the process according to item 1, comprising eye-tracking data. (Item 9) The sensor data is the process according to item 1, comprising controller position data reflecting the position of the controller. (Item 10) The sensor data is the process according to item 1, comprising data collected by a head-mounted camera. (Item 11) A wearable computing system, A head-mounted display system comprising a display and one or more sensors, A user input device configured to receive user input, One or more processors, At least partially detecting an object fixation event in which the user's posture is directed at a physical object or an object within a predetermined area thereof based on sensor data from the one or more sensors; Determining whether a condition defined by the duration of the object fixation event is satisfied; When the duration satisfies the condition, modifying the function of the user input device One or more processors programmed to implement a process including A wearable computing system comprising. (Item 12) The physical object is the user input device in the wearable computing system according to item 11. (Item 13) The process further includes causing the head-mounted display system to display a visual indication that the object fixation event has been detected in the wearable computing system according to item 11. (Item 14) The process further includes updating the visual indication during the object fixation event and indicating an amount of time for which the object fixation event needs to continue until the function of the user input device is modified in the wearable computing system according to item 11. (Item 15) After the process further modifies the function of the user input device, Detecting the end of the object fixation event; Restoring the previous function of the user input device in response to detecting the end of the object fixation event Including the wearable computing system according to item 11. (Item 16) The wearable computing system according to item 11, wherein modifying the function of the user input device includes causing the button of the user input device to be mapped to the function of displaying a menu on the display. (Item 17) The wearable computing system according to item 11, wherein the process further includes, in response to a determination that the duration of the object fixation event satisfies the condition, displaying a visual indication of a new function assigned to the user input device on the display. (Item 18) The wearable computing system according to item 11, wherein the process includes detecting the object fixation event based on the user's head posture regardless of the eye posture. (Item 19) The wearable computing system according to item 11, wherein the process includes detecting the fixation event at least partially based on the user's perceived eye posture. (Item 20) A computerized process implemented under the control of program instructions executed by one or more hardware processors, the process comprising: detecting a fixation event in which a user is fixating on a physical object or a defined area thereof based on data collected during a mixed reality or augmented reality session; starting a timer used to measure whether the duration of the fixation event satisfies a condition for temporarily modifying the functionality of a user input device in response to the detection of the fixation event; modifying the function of the user input device in response to a determination that the duration of the fixation event satisfies the condition and including a process.

Brief Description of the Drawings

[0016]

Figure 1

[0017]

Figure 2

[0018]

Figure 3

[0019]

Figure 4

[0020]

Figure 5-1

Figure 5-2

[0021] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION

[0022] Detailed Description I. Exemplary System Components Figures 1-3 illustrate the primary components of a wearable computing system 100 that implements fixation timer features, according to one embodiment. The wearable computing system 100 includes a headset 102, a handheld controller 104, and a wearable computing device 106 that is connected to the headset by cable 108 or wirelessly. Other configurations of wearable computing systems, such as systems where some or all components of the wearable computing device 106 are integrated within the headset 102 and / or the handheld controller 104 (or another type of user input device), are also possible. As described below, in the illustrated embodiment, the fixation timer is implemented within software executed by the wearable computing device 106 to enable the user to efficiently and intuitively extend the functionality of the handheld controller by performing a specific type of fixation action over a defined period. The disclosed process can also be used to extend the functionality of other types of user input devices, such as keyboards, wands, pointing devices, electronic drawing pads, headsets, joysticks, and mobile phones.

[0023] As shown in FIG. 1, the headset 102 includes a display 110, one or more inward-facing cameras 112, one or more outward-facing cameras 114, other sensors 116, one or more hardware processors 118, and an array of memories 120. As shown in FIG. 2, the headset 102 may be similar to a pair of goggles or glasses having translucent or transparent lenses that form the display 110. In some embodiments, the headset may be any suitable head-mounted display or device, and may have a single lens and / or display for one eye, two lenses and / or displays for both eyes, or a single lens and / or display for both eyes. The display 110 may be implemented using waveguides incorporated within the lenses, such as those disclosed in U.S. Patent Publication No. 2018 / 0157398, the disclosure of which is incorporated herein by reference. The headset 102 may alternatively be in the form of a helmet that partially or fully covers the user's head. During a mixed reality or augmented reality session, the display 110 typically augments the user's view of the real-world environment (as seen through the lenses) with digital objects and information.

[0024] The inward-facing camera or cameras 112, if present, are used to track the movement and position of one or both of the user's eyes. The tracked state of the user's eyes is referred to as the "eye pose." Other types of sensors besides cameras may also be used, in addition to or instead of, to track the user's eyes. A fixation timer may be implemented based on head pose only, without tracking eye movement, in which case the inward-facing camera 112 may be omitted or used for other functions only.

[0025] Other sensors 116 depicted in FIG. 1 may include various types of sensors for sensing head position, movement, and orientation. Examples include accelerometers, inertial measurement units (IMUs), compasses, wireless devices, and gyroscopes. The sensed position and orientation of the user's head is referred to as the "head pose." Processor 118 may include a general-purpose processor that executes code stored in memory 120, and / or may include one or more specialized processors such as a display processor / controller or a head pose processor. In some embodiments, the specialized processor may be implemented as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or other specialized hardware device. Although not shown in FIG. 1, headset 102 may also include other types of input and output devices such as microphones and speakers.

[0026] As shown in FIG. 1, the handheld controller 104 includes a set of user input elements 130 such as buttons, touch pads, joysticks, rotatable wheels, microphones, etc. These user input elements 130 may be used to control various aspects of an MR (or VR or AR session). In the embodiment shown in FIG. 3, the input elements of the controller include a circular touch pad 130a, a home / back button 130b, a bumper button 130c, and a trigger button 130d. As will be described below, the wearable computing device 106 (or another component of the system 100) may store mapping data that maps specific user input elements 130 and associated user actions to specific functions. Typically, the home / back button 130b is used to move backward within an application or return to the launcher, the bumper button 130c is used to display a context menu, and the trigger button is used for gripping and selection interactions. The touch pad 130a is capable of detecting various gestures that can be performed using the thumb, and each gesture can trigger an individual function or operation. As will be recognized, the handheld controller 104 can have different types and numbers of input elements 130 than those shown in FIG. 3, and / or may have different functionality associated with each input element 130.

[0027] The handheld controller 104 also includes a set of one or more sensors 132, such as sensors for sensing the position and orientation of the controller (referred to as "controller pose"). Examples include accelerometers, inertial measurement units (IMUs), compasses, wireless devices, and gyroscopes. The controller 104 is preferably a multi-degree-of-freedom (DOF) controller (e.g., a 3DOF or 6DOF controller), meaning that it can sense user input in multiple translations (e.g., left / right, forward / backward, or up / down) and / or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translations can be referred to as a 3DOF controller, while a multi-DOF controller that supports both translations and rotations can be referred to as a 6DOF controller.

[0028] As further shown in FIG. 1, the handheld controller 104 may also include a wireless transceiver 134 (or multiple wireless transceivers), such as a Bluetooth® or WIFI transceiver, or a transceiver not based on a wireless standard. The wireless transceiver 134 enables the handheld controller 104 to communicate wirelessly with the wearable computing device 106 and possibly other devices. For example, when a user performs an input operation (e.g., pressing a button) on the controller 104, the wireless transceiver 134 may notify the wearable computing device 106 of the input event. In other embodiments, the wireless transceiver 134 may be omitted, in which case the handheld controller 104 may be connected to the wearable computing device 106 by a cable. As shown, the handheld controller 104 also includes one or more hardware processors 140 connected to the aforementioned controller components 130, 132, 134, and a memory 142 that stores program code executed by the processor 140. The controller 104 may also include other types of components, such as one or more haptic devices for providing haptic feedback to the user, and an orientation processor for determining the orientation (position and orientation) of the controller.

[0029] As further shown in FIG. 1, the wearable computing device 106 includes one or more hardware processors 150, one or more wireless transceivers 152 (e.g., Bluetooth®, WIFI, cellular, etc.), and a memory 154 such as an array of solid state memory devices. The memory 154 stores operating system (OS) files of an operating system that supports composite and augmented reality applications and sessions, and includes application files of one or more applications 162 such as composite and / or augmented reality applications. As described above, in some implementations, components and / or functionality discussed with reference to the wearable computing device 106 may be implemented within the headset 102 such that a separate wearable computing device 106 may be omitted from the wearable computing system.

[0030] As shown in FIG. 1, the memory of the wearable computing device 106 also stores a set of mapping tables 164, 166 that map input operations on the handheld controller 104 to corresponding functions. These tables may be maintained and used by the operating system 160, the application 162, or both. The primary mapping table 164 stores mappings that are used when the functionality of the controller is not augmented by the gaze timer feature. Each entry in this table 164 maps an input operation on the user controller 104 (e.g., activation of a particular button or a particular gesture or tap sequence on the touchpad) to a corresponding function to be performed by the wearable computing system 100.

[0031] The secondary mapping table 166 stores mappings that are used when the functionality of the controller is temporarily extended using the gaze timer feature. In some embodiments, the extension in functionality may apply only to a single user input element 130 of the handheld controller 104. For example, the functionality of a single button (e.g., 130b, 130c, or 130d in FIG. 3) may be modified while the functionality of other user input elements 130 of the controller remains unchanged. Thus, the primary and secondary mapping tables 164, 166 may differ in only a single mapping, or the secondary mapping 166 may include only mappings for input that differ from those in the primary mapping 164. In other embodiments, two or more of the mappings may differ between these two mapping tables 164, 166. For example, the functionality of two or more different controller buttons or two or more different touchpad gestures may be modified.

[0032] The illustrated embodiments use mapping tables, but the gaze timer feature may alternatively be implemented without a mapping table. For example, changes in controller functionality may be hardcoded within the executable code that implements the gaze timer feature. Additionally, only one primary mapping table 164 and one secondary mapping table 166 are shown, but multiple primary tables and multiple secondary tables may be stored and used. For example, different applications 162 may use different primary / secondary mapping table pairs, and / or different controllers (or other user input devices or objects) may use different primary / secondary mapping table pairs.

[0033] In some embodiments, the wearable computing system 100 may interact with one or more types of external devices. For example, the system 100 may communicate with one or more radio frequency devices (e.g., beacons) mounted within a fixed location in the user's room or building for the purpose of tracking the location and orientation of the headset 102 and the controller 104. As another example, the wearable computing system 100 may communicate with a remote computing system such as a remote server or a cloud-based system. II. Fixation Timer State Diagram

[0034] FIG. 4 is a state diagram illustrating one embodiment of a process that may be implemented by the wearable system 100, and in particular, by the wearable computing device 106, to implement a gaze timer feature. The process may be implemented by the operating system 160 and / or by one or more of the applications 162. In the embodiment of FIG. 4, the user modifies the functionality of the controller by gazing at the handheld controller 104 over a set period of time. As described below, in some embodiments, the user may expand the functionality of the controller by gazing at another real-world object, such as the user's hand not holding the controller, or the user's hand performing a particular gesture.

[0035] As illustrated by the "start" arrow, when a session such as a mixed or augmented reality session is started, the process enters a monitoring state 400 and monitors for the start of a controller gaze event. More specifically, the process monitors sensor data that reflects the user's head pose and / or eye pose and the position of the handheld controller 104 to determine whether the user is gazing at the controller or within a predetermined distance thereof. Methods for detecting whether a user is gazing at an object are well known in the art.

[0036] When a controller gaze event is detected, the process transitions to the "Display Countdown Timer" state 402 and remains in this state until the gaze event ends or the countdown timer expires. The main purpose of the timer is to avoid modifying the functionality of the controller as a result of spurious controller gaze events. In other words, the main purpose of the timer is to reduce "false detections". In some embodiments, this purpose is facilitated by displaying a visual indication or other user feedback to the user that a controller gaze event has been detected, such that the user can avoid controller modification simply by looking away from the controller before the timer expires. The system may also, additionally or alternatively, notify the user of the detection of a controller gaze event via other types of user feedback, such as audible and / or tactile signals. For example, the controller (or a specific button on the controller) may vibrate when the gaze event is first detected.

[0037] In some embodiments, the countdown period is 0.5 seconds, although shorter and longer predetermined periods may also be used. Preferably, the period is in the range of 0.3 seconds to 2 seconds, more preferably in the range of 0.4 seconds to 1.2 seconds. The real-time state of the countdown timer is preferably graphically displayed to the user, such as by displaying a real-time timer object (see FIG. 5B discussed below). As illustrated by the arrow returning from state 402 to state 400, if the controller gaze event ends before the timer expires, the process returns to state 400 and monitors for the next controller gaze event. The timer is also reset when this state transition occurs.

[0038] If the timer expires before the fixation event ends, the process transitions from state 402 to state 404 and the functionality of the controller is temporarily modified. The process may notify the user of this transition via user feedback, such as visual, tactile, and / or audible notifications. If the modification applies only to a single controller button, the notification may identify the button (e.g., by outputting a tactile signal via the button) and / or identify the new function assigned to the button (e.g., by displaying a text message on display 110 as shown in FIG. 5C). While in this state 404, the wearable system 100 uses the secondary mapping table 166 (FIG. 1). Thus, if a controller input event occurs while the process is in this state 404, the function to be performed in response to the input event will be determined based on the secondary mapping table 166. As described above, different secondary mapping tables may be used by different applications 162 such that the modified functionality of the controller is application-dependent in some embodiments. In some embodiments, additional secondary mapping tables may be used and may be associated with other controllers, devices, and / or objects. For example, the user may gaze at a phone in the user's hand, which may trigger a focus timer, which may access a secondary mapping table associated with the phone. The secondary mapping table may be independent of the user's environmental context (recognizing the room the user is in, historical data, etc.).

[0039] In the illustrated embodiment, the process remains in state 404 until a restoration event occurs. A restoration event may be, for example, one or more of the following: (1) the user uses the enhanced controller functionality (e.g., presses a modified controller button), (2) the user closes a menu or other user interface opened using the enhanced controller functionality, (3) the user aborts a controller gaze action without using the enhanced controller functionality. Any combination of these and other types of restoration events may be implemented within the system. Once a restoration event occurs, the process returns to the monitoring state 400 and resumes its use of the primary mapping table 164.

[0040] The process described above is useful for handling "overmapping" scenarios where controller input events (or specific types of input events such as button press events) are not available for assignment to specific functions. Such scenarios can occur, for example, when all buttons are already assigned to individual functions, or when all possible controller input events (button presses, touchpad gestures, etc.) are already assigned to individual functions. The process may also be used in non-overmapping scenarios. For example, some controller input events (such as certain touchpad gestures) may remain unmapped, but a developer may desire to temporarily assign a given function to a more commonly or easily used input event, such as pressing a specific button 130b - 130d. As an example, the function of a bumper or trigger button may be temporarily modified to display a specific menu that is not context-dependent on the world.

[0041] The process of FIG. 4 may be modified such that the gaze timer is called by a gaze event directed at another type of real-world object other than the controller 104. For example, the user may expand the functionality of the controller by gazing at a hand not holding the controller, a hand performing a specific gesture such as a pointing gesture, a specific virtual or real-world object, or any other object over a set period of time. Further, the target of the gaze event may determine the controller modification that occurs in response to timer expiration. For example, the system may be configured to perform one modification when the user gazes at the hand performing the first gesture and a different modification when the user gazes at the hand performing the second gesture. The real-time object recognition process may be applied to a video / image stream from an outward-facing camera to determine whether the target of the user's gaze is something that calls the gaze timer feature. III. Exemplary User Interface

[0042] Figures 5A - 5D depict, in sequence, views seen by the user through the headset 102 during the use of the gaze timer feature according to one embodiment. In these drawings, the following elements are digital objects that extend the real-world view seen by the user through the lens, namely, the controller location icon 500 in FIG. 5A, the gaze timer graphic 502 in FIG. 5B, the text message 504 in FIG. 5C, and the menu 506 in FIG. 5D. All other objects within these views are actual physical objects seen through the lens.

[0043] In this particular embodiment of FIGS. 5A-5D, the user gazes at the handheld controller 104 over a set period of time, at which point the trigger button 130d (FIG. 3) of the controller is reconfigured to initiate the display of a menu 506 (or other user interface) that, based on a secondary mapping table, would not otherwise be displayable via the controller (using the primary mapping table). FIG. 5A illustrates a view as seen by the user before the user begins the controller-gazing operation. In this example, the user is standing or sitting within a room that includes a desk, chair, table, and window as seen through the lenses of the headset 102. The view in FIG. 5A also shows the hand of the user holding the controller 104. The headset, in this example, displays a controller position icon 500 that represents the real-time location and orientation of the controller 104 as detected by the wearable system 100 based on sensor data. As the user moves the controller 104, the icon 500 moves with the controller such that it appears at a fixed distance in front of the controller within 3D space. The icon 500 may, in some embodiments, be omitted and / or supplemented with or replaced by a line or beam that indicates the direction in which the controller is oriented.

[0044] In Figure 5A, the controller 104 is visible to the user but is not yet the target of a fixation operation, and thus the fixation timer has not yet been started. Referring to the state diagram of Figure 4, the system is in state 400 where the start of a controller fixation event is being monitored. In one embodiment, the user can start a controller fixation operation by fixating on either the icon 500 or the controller 104. As discussed above, the wearable system may determine whether the user is performing a controller fixation operation based on head pose, eye pose, or both. In some embodiments, controller fixation may be based on detecting whether the user is fixating on the controller position icon 500 or a predetermined distance therefrom. Methods for determining whether a user is looking at a digital or physical object based on head pose and / or eye pose are known in the art.

[0045] When a controller fixation event is first detected, the system transitions from state 400 to state 402 (Figure 4), and the wearable system 100 may notify the user by, for example, briefly changing the appearance of the icon 500, outputting an audible signal, and / or outputting a tactile signal via the handheld controller 104. As shown in Figure 5B, the wearable system may also replace the timer graphic 502 that graphically depicts the remaining amount of time until the timer expires (i.e., the amount of time that the controller fixation operation needs to continue to modify the functionality of the controller). The appearance of this timer icon 502 may change in real time as the timer counts down to zero.

[0046] As shown in FIG. 5C, once the attention timer expires, the system transitions from state 402 to 404 (FIG. 4), and the modified controller functionality is activated based on the secondary mapping table. In the example of FIG. 5C, the headset displays text message 504, indicating that the user can launch a menu by pressing the trigger button of the controller. The message may alternatively identify the particular button that has been remapped without specifying the newly assigned function (e.g., "press the trigger to continue"). In a use case where multiple controller buttons are remapped, the text message may, if applicable, indicate the new function assigned to each such button. If the user looks away from the controller 104 before pressing the trigger button, in some embodiments, the previous functionality of the trigger button is restored (e.g., the system returns to state 400 in the example of the state diagram of FIG. 4), and the display will revert to a view of the type shown in FIG. 5A. On the other hand, if the user presses the trigger button before looking away, a two-way menu 506 is displayed as shown in FIG. 5D. Menu 506 is not otherwise displayable via the trigger button in this example (i.e., the trigger button is normally mapped to a different function). Menu 506 may be context-dependent, but it need not be. Menu 506 is shown in FIG. 5D as being substantially opaque, but a translucent menu may be displayed so that the user can see the physical object appearing behind the menu.

[0047] In some embodiments, the display of the menu 506 continues after the user stops the controller-gaze operation and until the user performs another operation to close the menu. For example, the menu 506 may follow the controller 104 by being moved by the user, and / or by maintaining a position near the controller as the user moves around the environment. The user may interact with the menu using a finger, using the controller 104, and / or using other known methods. In some embodiments, the display location of the menu 506 tracks the location of the controller 104, and thus the menu remains accessible to the user as the user looks around or moves around, for example, in a room. The user may also be able to "drop" or "leave" the menu at a specific real-world location, to which the user can later return as desired. IV. Conclusion

[0048] The gaze timer feature as described above may be implemented within program code (instructions) executed by one or more of the components of the wearable computing system 100. For example, the gaze timer may be implemented within program instructions executed by the processor 150 (or processors) of the wearable computing device 106 and / or within program instructions executed by the processor 118 (or processors) of the headset. The gaze timer feature may also be implemented, in part or in whole, within software executed by a device or system that is not part of the wearable system 100. For example, some or all of the software for implementing the gaze timer feature may be launched on a local or remote computing device that communicates with the wearable system 100 via a wireless link.

[0049] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0050] In particular, conditional clauses used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes certain features, elements, and / or steps, other embodiments do not include them, unless specifically stated otherwise or understood otherwise in the context in which they are used. Thus, such conditional clauses generally do not imply that the features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more than one" or "at least one", unless otherwise defined.

[0051] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless specifically stated otherwise, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to imply that an embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0052] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not required for the desired results to be achieved, that such operations may be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations need to be performed. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results.

Claims

1. A computerized process executed under the control of program instructions executed by one or more hardware processors, the process comprising: Detecting a controller gaze event based on sensor data collected during a mixed reality or augmented reality session as to whether the user is gazing at a handheld controller or is gazing within a predetermined distance of the handheld controller; In response to detecting the controller gaze event, starting a timer used to measure whether the duration of the controller gaze event meets a condition to temporarily modify the functionality of the handheld controller; In response to determining that the duration of the controller gaze event meets the condition, modifying the functionality of the user input element of the handheld controller A process including.

2. The process according to claim 1, further comprising displaying a visual indication that the timer has been started on a display generated by a head-mounted display worn by the user.

3. The process according to claim 2, further comprising indicating, by updating the visual indication during the controller gaze event, the amount of time the controller gaze event needs to continue until the functionality of the user input element is modified.

4. The process includes After modifying the functionality of the user input element, Detecting the end of the controller gaze event based on the sensor data; and Restoring the previous functionality of the user input element in response to detecting the end of the controller gaze event The process according to claim 1, further comprising.

5. Modifying the functionality of the user input element includes mapping the buttons of the handheld controller to the function of displaying a menu, the process according to claim 1.

6. The process according to claim 1, further comprising displaying a visual indication of the new functionality assigned to the user input element in response to determining that the duration of the controller gaze event meets the condition.

7. The process according to claim 1, wherein the sensor data comprises at least head tracking data reflecting the orientation of the user's head.

8. The process according to claim 1, wherein the sensor data comprises eye tracking data.

9. The process according to claim 1, wherein the sensor data comprises controller position data reflecting the position of the handheld controller.

10. The process according to claim 1, wherein the sensor data comprises data collected by a head-mounted camera.

11. A wearable computing system, the wearable computing system comprising: A head-mounted display system comprising a display and one or more sensors; A user input device configured to receive user input; One or more processors and wherein the one or more processors are programmed to implement a process, the process comprising: Detecting an object fixation event based at least in part on sensor data from the one or more sensors, whether the user's pose is directed at a physical object or whether the user's pose is directed within a predetermined distance of the physical object; Determining whether a defined condition for the duration of the object fixation event is met; Modifying the functionality of the user input device when the duration meets the condition A wearable computing system.

12. The wearable computing system according to claim 11, wherein the physical object is the user input device.

13. The wearable computing system according to claim 11, wherein the process further comprises causing the head-mounted display system to display a visual indication that the object fixation event has been detected.

14. The wearable computing system according to claim 13, wherein the process further comprises indicating, by updating the visual indication during the object fixation event, the amount of time the object fixation event needs to continue until the functionality of the user input device is modified.

15. After modifying the functionality of the user input device, detecting the end of the object fixation event, and in response to detecting the end of the object fixation event, restoring the previous functionality of the user input device The wearable computing system according to claim 11, further comprising.

16. Modifying the functionality of the user input device includes mapping the buttons of the user input device to a function of displaying a menu on the display. The wearable computing system according to claim 11.

17. The process further includes displaying, on the display, a visual indication of a new function assigned to the user input device in response to determining that the duration of the object fixation event meets the condition. The wearable computing system according to claim 11.

18. The process includes detecting the object fixation event based on the user's head pose regardless of the eye pose. The wearable computing system according to claim 11.

19. The process includes detecting the object fixation event based at least in part on the user's perceived eye pose. The wearable computing system according to claim 11.

20. A computerized process executed under the control of program instructions executed by one or more hardware processors, the process comprising: detecting a fixation event that determines whether the user is fixating on a physical object or whether the user is fixating within a predetermined distance of the physical object based on data collected during a mixed reality or augmented reality session; starting a timer used to measure whether the duration of the fixation event meets the condition for temporarily modifying the functionality of the user input device in response to detecting the fixation event; modifying the functionality of the user input device in response to determining that the duration of the fixation event meets the condition A process, including.

Images