Artificial reality system with sliding menu

Abstract

The invention relates to an artificial reality system with a sliding menu. An artificial reality system is described that renders, presents, and controls user interface elements in an artificial reality environment and performs actions in response to detected one or more gestures of a user. An artificial reality system may include a menu that may be activated and interacted with one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system may cause a menu to be rendered. A menu swipe gesture (e.g., horizontal motion) of a hand may be used to move slidably engaged user interface (UI) elements along a horizontal dimension of the menu while the horizontal location of the UI menu remains unchanged. Movement of the hand perpendicular to the menu swipe gesture (e.g., non-horizontal movement) may cause the menu to be repositioned. Implementation of an artificial reality system indeed requires both hands or other input devices to interact with the artificial reality system.

Description

[0001] This application is a divisional application of an application with a filing date of June 3, 2020, application number 202080035337.X, and invention name “Artificial reality system with sliding menu”. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 434,919, filed on June 7, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to artificial reality systems, such as virtual reality, mixed reality, and / or augmented reality systems, and more particularly to user interfaces for artificial reality systems. background

[0004] Artificial reality systems are becoming increasingly common, with applications in many fields such as computer gaming, health and safety, industry, and education. Artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks, as a few examples. Typically, an artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof.

[0005] A typical artificial reality system includes one or more devices for rendering and displaying content to a user. As an example, an artificial reality system may incorporate a head mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include entirely generated content or generated content combined with captured content (e.g., real-world video and / or images). During operation, a user typically interacts with the artificial reality system to select content, launch applications, or otherwise configure the system. Overview

[0006] In general, this disclosure describes artificial reality systems and, more particularly, describes graphical user interface elements and techniques for presenting and controlling user interface elements in an artificial reality environment.

[0007] For example, an artificial reality system is described that generates and renders a graphical user interface element for display to a user in response to detecting one or more predefined gestures of a user, such as a particular movement, configuration, position, and / or orientation of a hand, finger, thumb, or arm of the user, or a combination of predefined gestures. In some examples, the artificial reality system may also trigger the generation and rendering of a graphical user interface element in response to the detection of a particular gesture in combination with other conditions, such as the position and orientation of the particular gesture in the physical environment relative to the user's current field of view, which may be determined by real-time gaze tracking of the user, or relative to a pose of an HMD worn by the user.

[0008] In some examples, the artificial reality system can generate and present a graphical user interface element as an overlay element relative to artificial reality content currently being rendered within a display of the artificial reality system. The graphical user interface element can be, for example, a graphical user interface, such as a menu or submenu with which a user interacts to operate the artificial reality system, or a single graphical user interface element that can be selected and operated by a user, such as a toggle element, a drop-down element, a menu selection element, a two-dimensional or three-dimensional shape, a graphical input key or keyboard, a content display window, etc.

[0009] The technical problem of some HMDs is the lack of input devices that can be used to interact with various aspects of the artificial reality system, for example, positioning a selection user interface element within a menu. In some systems, the artificial reality system can use both hands of the user to provide the user with a menu or icon. However, the technical problem with this type of interaction is that one hand may block the other hand, making it difficult for the artificial reality system to accurately determine the user's intention. In addition, some users may have disabilities, which may prevent them from using both hands to interact with the artificial reality system. As a technical solution to the aforementioned technical problems, some aspects include menus that can be activated and interacted with with one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system can cause the menu to be rendered. A menu sliding gesture (e.g., horizontal motion) of the hand can be used to move a slidably engageable user interface (UI) element along the horizontal dimension of the menu while keeping the horizontal positioning of the menu unchanged. In some aspects, the movement of the hand substantially perpendicular to the menu sliding gesture (e.g., non-horizontal motion) can cause the menu to be repositioned. Implementations of artificial reality systems do not require the use of both hands or use of other input devices to interact with the artificial reality system, and thus this technical improvement over conventional artificial reality implementations may provide one or more practical applications, such as improving ease of use, providing individuals with disabilities with capabilities associated with using one hand to interact with the system, and the ability to accurately determine a user's interaction with a menu or other user interface element.

[0010] In one or more example aspects, an artificial reality system includes an image capture device configured to capture image data; a head mounted device (HMD) configured to output artificial reality content; a gesture detector configured to identify a menu activation gesture from the image data, the menu activation gesture comprising a hand configuration with a hand in a substantially upward orientation and a pinch configuration of a thumb and fingers of the hand; a UI engine configured to generate a menu interface and a slidably engageable UI element in a first position relative to the menu interface in response to the menu activation gesture; and a rendering engine configured to render the artificial reality content, the menu interface, and the slidably engageable UI element for display at the HMD.

[0011] In one or more further example aspects, a method includes obtaining image data via an image capture device through an artificial reality system including a head-mounted device (HMD); identifying, by the artificial reality system, a menu activation gesture from the image data, the menu activation gesture comprising a hand configuration with a hand in a substantially upward orientation and a pinch configuration of a thumb and fingers of the hand; generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engageable UI element in a first position relative to the menu interface; and rendering, by the artificial reality system, artificial reality content, the menu interface, and the slidably engageable UI element for display at the HMD.

[0012] In one or more additional example aspects, a non-transitory computer-readable medium includes instructions that, when executed, cause one or more processors of an artificial reality system to: capture image data via an image capture device; identify a menu activation gesture including a hand configuration from the image data; generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface in response to the menu activation gesture; identify a menu sliding gesture including the hand configuration combined with movement of the hand after the menu activation gesture; translate the slidably engageable UI element to a second position relative to the menu interface in response to the menu sliding gesture; and render artificial reality content, the menu interface, and the slidably engageable UI element for display at a head mounted device (HMD).

[0013] In one example, an artificial reality system is provided, comprising: an image capture device configured to capture image data; a head-mounted display (HMD) configured to output artificial reality content; a gesture detector configured to identify a menu activation gesture from the image data, the menu activation gesture comprising a configuration of a hand in a substantially upward orientation and a pinch configuration of a thumb and fingers of the hand; a user interface (UI) engine configured to generate a menu interface and a slidably engageable UI element in a first position relative to the menu interface in response to the menu activation gesture; and a rendering engine configured to render the artificial reality content, the menu interface, and the slidably engageable UI element for display at the HMD.

[0014] In some embodiments, the menu interface preferably includes one or more menu items arranged along a dimension of the menu interface, and wherein the UI engine is configured to highlight one of the menu items based on a position of the slidably engageable UI element relative to the menu interface.

[0015] In some embodiments, one or more menu items preferably correspond to respective applications executing on the artificial reality system.

[0016] In some embodiments, the gesture detector is preferably further configured to identify a menu slide gesture from the image data, the menu slide gesture comprising a hand configuration combined with a hand movement, and wherein the UI engine is further configured to translate the slidably engageable UI element to a second position relative to the menu interface in response to the menu slide gesture.

[0017] In some embodiments, the menu interface preferably includes one or more menu items arranged along a dimension of the menu interface, wherein in order to translate the slidably engaged UI element to a second position relative to the menu interface, the UI engine is configured to slide the slidably engaged UI element along the dimension of the menu interface to the second position relative to the menu interface.

[0018] In some embodiments, the menu sliding gesture preferably includes movement of the hand in a substantially first direction, wherein the gesture detector is configured to identify from the image data movement of the hand in a substantially second direction after the menu activation gesture, the substantially second direction being substantially perpendicular to the substantially first direction, wherein the UI engine is further configured to translate the slidably engageable UI element and the menu interface in response to the movement of the hand in the substantially second direction while maintaining a position of the slidably engageable UI element relative to the menu interface.

[0019] In some embodiments, the artificial reality system preferably also includes: an application engine for executing one or more artificial reality applications, wherein the gesture detector is configured to identify a selection gesture subsequent to the menu sliding gesture from the image data, and wherein the application engine is configured to perform an action in response to the selection gesture.

[0020] In some embodiments, the selection gesture preferably includes one of: (1) movement of different fingers of the hand, (2) translation of the hand in a direction substantially perpendicular to the menu interface, or (3) reconfiguring the thumb and fingers of the hand to no longer be in a pinch configuration.

[0021] In some embodiments, the gesture detector is preferably further configured to identify a menu hint gesture preceding the menu activation gesture from the image data, and wherein the UI engine is further configured to generate a menu hint element in response to the menu hint gesture.

[0022] In some embodiments, the menu prompt gesture preferably includes the hand being configured in a substantially upward position with a space between the thumb and fingers, and wherein the UI engine generates the menu prompt element in the space between the thumb and fingers of the virtual hand.

[0023] In some embodiments, the menu prompt element preferably comprises a line between the thumb and fingers of the virtual hand.

[0024] In some embodiments, the image capture device is preferably integrated with the HMD.

[0025] In some examples, a method is provided, including: obtaining image data via an image capture device through an artificial reality system including a head-mounted display (HMD); recognizing, by the artificial reality system, a menu activation gesture from the image data, the menu activation gesture comprising a hand configuration with a hand in a substantially upward orientation and a pinch configuration of a thumb and fingers of the hand; generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engageable UI element in a first position relative to the menu interface; and rendering, by the artificial reality system, artificial reality content, the menu interface, and the slidably engageable UI element for display at the HMD.

[0026] In some embodiments, the menu interface preferably includes one or more menu items arranged along a dimension of the menu interface, and the method further includes: highlighting one of the menu items by the artificial reality system based on the position of the slidably engageable UI element relative to the menu interface.

[0027] In some embodiments, the method preferably further includes: identifying, by the artificial reality system, a menu sliding gesture from the image data, the menu sliding gesture comprising a hand configuration combined with hand movement; and translating, by the artificial reality system in response to the menu sliding gesture, the slidably engageable UI element to a second position relative to the menu interface.

[0028] In some embodiments, translating the slidably engaged UI element to the second position relative to the menu interface preferably includes sliding the slidably engaged UI element along a dimension of the menu interface to the second position relative to the menu interface.

[0029] In some embodiments, the menu sliding gesture preferably includes movement of the hand in substantially a first direction, and the method further includes: identifying, by the artificial reality system from image data, movement of the hand in substantially a second direction after the menu activation gesture, the substantially second direction being substantially perpendicular to the substantially first direction; and in response to the movement of the hand in substantially the second direction, translating, by the artificial reality system, the slidably engageable UI element and the menu interface while maintaining a position of the slidably engageable UI element relative to the menu interface.

[0030] In some embodiments, the method preferably further includes: recognizing, by the artificial reality system, a menu prompt gesture from the image data; and generating, by the artificial reality system, a menu prompt element in response to the menu prompt gesture.

[0031] In some examples, a non-transitory computer-readable medium is provided that includes instructions that, when executed, cause one or more processors of an artificial reality system to: capture image data via an image capture device; identify a menu activation gesture that includes a hand configuration from the image data; generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface in response to the menu activation gesture; identify a menu sliding gesture that includes a hand configuration combined with movement of the hand after the menu activation gesture; translate the slidably engageable UI element to a second position relative to the menu interface in response to the menu sliding gesture; and render artificial reality content, the menu interface, and the slidably engageable UI element for display at a head-mounted display (HMD).

[0032] In some embodiments, the instructions further cause one or more processors to: identify a menu prompt gesture from the image data, the menu prompt gesture comprising a hand configured in a substantially upward position with a space between the thumb and fingers; and generate a menu prompt element in the space between the thumb and fingers of the virtual hand in response to the menu prompt gesture.

[0033] It should be understood that the features of the present disclosure described as suitable for incorporation into one or more examples or embodiments of the present disclosure are intended to be generalizable to any and all examples and embodiments of the present disclosure. The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objects and advantages of the present invention will be apparent from the specification and drawings, as well as the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1A is a diagram depicting an example artificial reality system for presenting and controlling user interface elements in an artificial reality environment in accordance with techniques of this disclosure.

[0035] Figure 1B is a diagram depicting another example artificial reality system in accordance with techniques of this disclosure.

[0036] Figure 2 is an illustration depicting an example HMD operating in accordance with the techniques of this disclosure.

[0037] Figure 3 It is shown Figure 1A and Figure 1B A block diagram of an example implementation of a console and HMD for an artificial reality system.

[0038] Figure 4 It is a description of the technology according to the present disclosure. Figure 1A and Figure 1B A block diagram of an example of an artificial reality system HMD performing gesture detection and user interface generation.

[0039] Figure 5 is a flow chart illustrating the operation of an example method for activating a menu prompt or UI menu in accordance with aspects of the present disclosure.

[0040] Figure 6 is a flow diagram illustrating the operation of an example method for locating and interacting with a UI menu in accordance with aspects of the present disclosure.

[0041] Figure 7A-7G is an example HMD display illustrating positioning of and interaction with UI menus according to aspects of the present disclosure.

[0042] Figure 8 is an example HMD display showing menu prompts according to aspects of the present disclosure.

[0043] Like reference characters refer to like elements throughout the drawings and the specification. Detailed Description

[0044] Figure 1A is a diagram depicting an example artificial reality system 10 for presenting and controlling user interface elements in an artificial reality environment in accordance with the techniques of this disclosure. In some example implementations, the artificial reality system 10 generates and renders graphical user interface elements to the user 110 in response to one or more detected gestures performed by the user 110. That is, as described herein, the artificial reality system 10 presents one or more graphical user interface elements 124, 126 in response to detecting one or more specific gestures performed by the user 110, such as specific movements, configurations, positions, and / or orientations of the user's hand, fingers, thumb, or arm. In other examples, the artificial reality system 10 presents and controls user interface elements specifically designed for user interaction and manipulation in an artificial reality environment, such as dedicated toggle elements, drop-down elements, menu selection elements, graphical input keys or keyboards, content display windows, and the like.

[0045] exist Figure 1AIn the example of , the artificial reality system 10 includes a head mounted device (HMD) 112, a console 106, and one or more external sensors 90 in some examples. As shown, the HMD 112 is typically worn by the user 110 and includes an electronic display and optical components for presenting artificial reality content 122 to the user 110. In addition, the HMD 112 includes one or more sensors (e.g., accelerometers) for tracking the movement of the HMD 112, and may include one or more image capture devices 138 (e.g., cameras, linescanners, etc.) for capturing image data of the surrounding physical environment. In this example, the console 106 is shown as a single computing device, such as a game console, a workstation, a desktop computer, or a laptop computer. In other examples, the console 106 may be distributed across multiple computing devices, such as a distributed computing network, a data center, or a cloud computing system. As shown in this example, the console 106, the HMD 112, and the sensor 90 may be communicatively coupled via a network 104, which may be a wired or wireless network, such as WiFi, a mesh network, or a short-range wireless communication medium. Although in this example, the HMD 112 is shown as being in communication with the console 106, such as being tethered to the console or in wireless communication with the console, in some implementations, the HMD 112 operates as a standalone mobile artificial reality system.

[0046] Typically, the artificial reality system 10 uses information captured from the real world 3D physical environment to render artificial reality content 122 for display to the user 110. Figure 1A In the example of , the user 110 views artificial reality content 122 constructed and rendered by an artificial reality application executed on the console 106 and / or the HMD 112. As an example, the artificial reality content 122 may be a consumer gaming application in which the user 110 is rendered as an avatar 120 with one or more virtual objects 128A, 128B. In some examples, the artificial reality content 122 may include a mix of real-world images and virtual objects, such as mixed reality and / or augmented reality. In other examples, the artificial reality content 122 may be, for example, a video conferencing application, a navigation application, an educational application, a training or simulation application, or other types of applications that implement artificial reality.

[0047] During operation, the artificial reality application constructs artificial reality content 122 for display to the user 110 by tracking and calculating pose information of a reference frame, typically the viewing angle of the HMD 112. Using the HMD 112 as a reference frame, and based on a current field of view 130 determined by the current estimated pose of the HMD 112, the artificial reality application renders 3D artificial reality content, which in some examples may be at least partially overlaid on the real-world 3D physical environment of the user 110. In this process, the artificial reality application uses sensory data received from the HMD 112, such as motion information and user commands, and in some examples, uses data from any external sensors 90 (e.g., external cameras) to capture 3D information in the real-world, physical environment, such as the motion of the user 110 and / or feature tracking information about the user 110. Based on the sensed data, the artificial reality application determines the current pose of the reference frame of the HMD 112, and renders the artificial reality content 122 according to the current pose.

[0048] In addition, according to the techniques of the present disclosure, based on the sensed data, the artificial reality application detects gestures performed by the user 110, and in response to detecting one or more specific gestures, generates one or more user interface elements, such as a UI menu 124 and a UI element 126, which can be overlaid on the underlying artificial reality content 122 presented to the user. In this regard, the user interface elements 124, 126 can be considered as part of the artificial reality content 122 presented to the user in the artificial reality environment. In this way, the artificial reality system 10 dynamically presents one or more graphical user interface elements 124, 126 in response to detecting one or more specific gestures of the user 110, such as a specific movement, configuration, position and / or orientation of the user's hand, finger, thumb or arm. Example configurations of the user's hand can include a fist, one or more fingers extended, relative and / or absolute positions and orientations of one or more individual fingers of the hand, the shape of the palm, etc. For example, the user interface element may be a graphical user interface, such as a menu or submenu that the user 110 interacts with to operate the artificial reality system, or a single user interface element that the user 110 may select and manipulate, such as an icon element, a toggle element, a drop-down element, a menu selection element, a two-dimensional or three-dimensional shape, a graphical input key or keyboard, a content display window, etc. Although depicted as a two-dimensional element, for example, the UI element 126 may be a two-dimensional or three-dimensional shape that may be manipulated by a user performing gestures to translate, scale, and / or rotate the shape in the artificial reality environment.

[0049] In addition, as described herein, in some examples, the artificial reality system 10 can trigger the generation and rendering of graphical user interface elements 124, 126 in response to other conditions, such as the current state of one or more applications being executed by the system, or the position and orientation of a particular detected gesture in the physical environment relative to the user's 110 current field of view 130 (which can be determined by real-time gaze tracking of the user or other conditions).

[0050] More specifically, as further described herein, the image capture device 138 of the HMD 112 captures image data representing objects in the real-world physical environment that are within the field of view 130 of the image capture device 138. The field of view 130 generally corresponds to the viewing angle of the HMD 112. In some examples, for example Figure 1A In the example shown, the artificial reality application renders the portion of the user's 110 hand 132 within the field of view 130 as a virtual hand 136 within the artificial reality content 122. In other examples, the artificial reality application may present a real-world image of the user's 110 hand 132 and / or arm 134 in the artificial reality content 122, including mixed reality and / or augmented reality. In either example, the user 110 is able to view the portion of their hand 132 and / or arm 134 within the field of view 130 as an object within the artificial reality content 122. In other examples, the artificial reality application may not render a representation of the user's hand 132 or arm 134.

[0051] In any case, during operation, the artificial reality system 10 performs object recognition within the image data captured by the image capture device 138 of the HMD 112 to identify the hand 132, including optionally identifying individual fingers or thumbs of the user 110, and / or all or part of the arm 134. In addition, the artificial reality system 10 tracks the position, orientation, and configuration of the hand 132 (optionally including specific fingers of the hand) and / or the portion of the arm 134 over a sliding time window. The artificial reality application analyzes any tracked motion, configuration, position, and / or orientation of the portion of the hand 132 and / or arm 134 to identify one or more gestures performed by a specific object (e.g., the hand 132 (including specific fingers of the hand) and / or the portion of the arm 134 of the user 110). To detect a gesture, the artificial reality application may compare the motion, configuration, position, and / or orientation of the portion of the hand 132 and / or arm 134 to gesture definitions stored in a gesture library of the artificial reality system 10, where each gesture in the gesture library may be mapped to one or more actions. In some examples, detecting movement may include tracking the position of one or more fingers (individual fingers and thumb) of hand 132, including whether any defined combination of fingers (e.g., index finger and thumb) are placed together touching or nearly touching in the physical environment. In other examples, detecting movement may include tracking the orientation of hand 132 (e.g., fingers pointing toward HMD 112 or away from HMD 112) and / or the orientation of arm 134 relative to the current pose of HMD 112 (i.e., the normal of the arm faces toward HMD 112). The position and orientation of hand 132 (or a portion thereof) may alternatively be referred to as the pose of hand 132 (or a portion thereof).

[0052] In addition, the artificial reality application may analyze the configuration, position, and / or orientation of the hand 132 and / or arm 134 to identify gestures that include the hand 132 and / or arm 134 being maintained in one or more specific configurations, positions, and / or orientations for at least a threshold period of time. As an example, one or more specific positions of the hand 132 and / or arm 134 remaining substantially stationary within the field of view 130 for at least a configurable period of time may be used by the artificial reality system 10 as an indication that the user 110 is attempting to perform a gesture intended to trigger a desired response from the artificial reality application, such as triggering the display of a particular type of user interface element 124, 126 (such as a menu). As another example, one or more specific configurations of the fingers and / or palm of the hand 132 and / or arm 134 being maintained within the field of view 130 for at least a configurable period of time may be used by the artificial reality system 10 as an indication that the user 110 is attempting to perform a gesture. Although in Figure 1AOnly the right hand 132 and the right arm 134 of the user 110 are shown in FIG, but in other examples, the artificial reality system 10 may recognize the left hand and / or left arm of the user 110 or recognize the right hand and / or right arm of the user 110 and the left hand and / or left arm of the user 110. In this way, the artificial reality system 10 may detect single-handed gestures, two-handed gestures, or arm-based gestures performed by a hand in the physical environment and generate associated user interface elements in response to the detected gestures.

[0053] According to the technology of the present disclosure, the artificial reality application determines whether the recognized gesture corresponds to a gesture defined by one of multiple entries in a gesture library of the console 106 and / or HMD 112. As described in more detail below, each entry in the gesture library can define a different gesture as a specific movement, configuration, position and / or orientation of a user's hand, digit (finger or thumb) and / or arm over time, or a combination of these attributes. In addition, each defined gesture can be associated with a desired response in the form of one or more actions to be performed by the artificial reality application. As an example, one or more defined gestures in the gesture library can trigger the generation, conversion and / or configuration of one or more user interface elements (e.g., UI menu 124) to be rendered and overlaid on artificial reality content 122, where the gesture can define the position and / or orientation of UI menu 124 in artificial reality content 122. As another example, one or more defined gestures may indicate interaction of user 110 with a particular user interface element, such as selection of UI element 126 of UI menu 124, to trigger changes to the presented user interface, presentation of a submenu of the presented user interface, etc.

[0054] In some aspects, the artificial reality application may analyze the configuration, position, and / or orientation of the hand 132 and / or arm 134 to identify a menu activation gesture that includes the hand 132 being held in a particular configuration and orientation for at least a threshold period of time. In some aspects, the menu activation gesture may be, for example, the hand being held in a substantially upward position while the fingers and thumb of the hand are in a pinch configuration. In some aspects, the menu activation gesture may include the fingers and thumb of the hand being positioned in a pinch configuration regardless of the orientation of the hand. The menu sliding gesture may cause a virtual hand that moves in accordance with the user's hand to slide along a dimension of the UI menu 124 while the menu remains stationary in the sliding direction. Movement in a direction other than the menu sliding gesture may result in repositioning the UI menu 124 based on the movement. As an example, the menu sliding gesture may be movement of the user's hand 132 in a horizontal direction while the menu activation gesture is held. The virtual hand 136 may move along a horizontal dimension while the menu remains stationary in the horizontal direction. In some examples, in addition to the virtual hand 136, or as an alternative to the virtual hand 136, the artificial reality application generates a slidably engageable UI element ( Figure 1A ). Movement in the vertical direction may cause the UI menu 124 to be repositioned.

[0055] While maintaining the menu activation gesture, the menu sliding gesture can cause the artificial reality application to render an indication that a particular menu item of the UI menu 124 would be selected if the user were to perform a selection gesture without further performing a menu sliding gesture to slide the virtual hand 132 to, for example, a different position proximate to a different menu item of the UI menu 124. The particular menu is ready for selection by the user. The indication can be positioning of the virtual hand 132 or a slidably engaged UI element proximate to the menu item; highlighting the menu item with a different color, for example; enlargement of the menu item; or some other indication.

[0056] Thus, the technology of the present disclosure provides specific technical improvements to computer-related fields that render and display content through artificial reality systems. For example, the artificial reality system described herein can provide a high-quality artificial reality experience to a user of an artificial reality application (e.g., user 110) by generating and rendering user interface elements overlaid on artificial reality content based on the detection of intuitive but distinctive gestures performed by the user. More specifically, the technology can provide intuitive user input to the user in the form of gestures, through which the user can activate a menu interface and subsequently translate a slidably engaged UI element or other indication of a menu item prepared for selection by the user along a dimension of the menu.

[0057] In addition, the systems described herein can be configured to detect certain gestures based on hand and arm movements that are defined to avoid tracking occlusions. Tracking occlusions may occur when one of a user's hands at least partially overlaps with another hand, making it difficult to accurately track the individual digits (fingers and thumb) of each hand, as well as the position and orientation of each hand. Therefore, the systems described herein can be configured to primarily detect single-hand or single-arm based gestures. The use of single-hand or single-arm based gestures can further provide enhanced accessibility for users with large movements and fine motor skill limitations. In addition, the systems described herein can be configured to detect two-handed or two-arm based gestures where the user's hands do not interact or overlap with each other.

[0058] In addition, the systems described herein can be configured to detect gestures that provide self-tactile feedback to the user. For example, the thumb and one or more fingers on each hand of the user can touch or nearly touch in the physical world as part of a predefined gesture that indicates interaction with a specific user interface element in artificial reality content. The touch between the thumb and one or more fingers of the user's hand can provide the user with a simulation of the sensation that the user feels when interacting directly with a physical user input object (e.g., a button on a physical keyboard or other physical input device).

[0059] Figure 1B is a diagram depicting another example artificial reality system 20 according to techniques of the present disclosure. Figure 1A The artificial reality system 10, in some examples, Figure 1B The artificial reality system 20 can present and control user interface elements designed specifically for user interaction and manipulation in an artificial reality environment. In various examples, the artificial reality system 20 can also generate certain graphical user interface elements and render them to the user in response to detecting one or more specific gestures of the user.

[0060] exist Figure 1B In the example of FIG. 1 , the artificial reality system 20 includes external cameras 102A and 102B (collectively referred to as “external cameras 102”), HMDs 112A-112C (collectively referred to as “HMDs 112”), controllers 114A and 114B (collectively referred to as “controllers 114”), a console 106, and a sensor 90. Figure 1BAs shown, artificial reality system 20 represents a multi-user environment in which an artificial reality application executed on console 106 and / or HMD 112 presents artificial reality content to each of users 110A-110C (collectively, “users 110”) based on the current viewing perspective of the corresponding reference frame of the corresponding user. That is, in this example, the artificial reality application constructs artificial content by tracking and calculating pose information of the reference frame of each HMD 112. Artificial reality system 20 uses data received from camera 102, HMD 112, and controller 114 to capture 3D information in the real-world environment, such as the movement of user 110 and / or tracking information about user 110 and object 108, for calculating updated pose information of the corresponding reference frame of HMD 112. As one example, the artificial reality application may render artificial reality content 122 having virtual objects 128A-128C (collectively, “virtual objects 128”) as spatially overlaid on real-world objects 108A-108C (collectively, “real-world objects 108”) based on the current viewing perspective determined for HMD 112C. Additionally, artificial reality system 20 renders avatars 120A, 120B based on the estimated positions of users 110A, 110B, respectively, from the perspective of HMD 112C.

[0061] Each of the HMDs 112 operates simultaneously in the artificial reality system 20. Figure 1B In the example of FIG. 1 , each user 110 may be a “player” or “participant” in the artificial reality application, and any user 110 may be a “spectator” or “observer” in the artificial reality application. By tracking the hand 132 and / or arm 134 of the user 110C and rendering the portion of the hand 132 within the field of view 130 as a virtual hand 136 within the artificial reality content 122, the HMD 112C may be substantially similar to Figure 1A HMD 112B may receive user input from controller 114A held by user 110B. HMD 112A may also be substantially similar to Figure 1A The HMD 112B may be operated by the user 110A and receive user input in the form of gestures of the hands 132A, 132B of the user 110A. The HMD 112B may receive user input from a controller 114 held by the user 110B. The controller 114 may communicate with the HMD 112B using near field communication such as short-range wireless communication such as Bluetooth, using a wired communication link, or using other types of communication links.

[0062] Similar to the above Figure 1ABy way of example, the console 106 and / or HMD 112C of the artificial reality system 20 generates and renders user interface elements 124, 126 that may be overlaid on the artificial reality content 122 displayed to the user 110C. Furthermore, the console 106 and / or HMD 112C may trigger the generation and dynamic display of the user interface elements 124, 126 based on the detection of intuitive but distinctive gestures performed by the user 110C via gesture tracking. For example, the artificial reality system 20 may dynamically present one or more graphical user interface elements 124, 126 in response to detecting one or more specific gestures of the user 110C, such as a specific movement, configuration, position, and / or orientation of the user's hand, finger, thumb, or arm. As shown in FIG. Figure 1B As shown, in addition to or in lieu of image data captured via the camera 138 of the HMD 112C, input data from the external camera 102 may be used to track and detect specific motions, configurations, positions, and / or orientations of the hands and arms of the user 110 (e.g., the hand 132 of the user 110C), including individual and / or combined movements of the fingers (fingers, thumbs) of the hands.

[0063] In some aspects, an artificial reality application may be running on console 106 and may utilize image capture devices 102A and 102B to analyze the configuration, position, and / or orientation of hand 132B to identify menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, or menu positioning actions, etc., that may be performed by a user of HMD 112A. Similarly, HMD 112C may utilize image capture device 138 to analyze the configuration, position, and / or orientation of hand 132C to identify menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, or menu positioning actions, etc., that may be performed by a user of HMD 112C. The artificial reality application may respond to such gestures in a manner similar to that described above with reference to FIG. Figure 1A The manner in which the UI menu 124 and the virtual hand 136 are rendered is described.

[0064] Figure 2 is a diagram depicting an example HMD 112 configured to operate in accordance with the techniques of this disclosure. Figure 2 The HMD 112 may be Figure 1A and Figure 1B An example of any HMD 112. The HMD 112 may be an artificial reality system such as Figure 1A Artificial reality system 10. Figure 1B The artificial reality system 20 may be part of an artificial reality system 20), or may be operated as an independent, mobile artificial reality system configured to implement the techniques described herein.

[0065] In this example, the HMD 112 includes a front rigid body and a band for securing the HMD 112 to the user. In addition, the HMD 112 includes an inward-facing electronic display 203 that is configured to present artificial reality content to the user. The electronic display 203 can be any suitable display technology, such as a liquid crystal display (LCD), a quantum dot display, a dot matrix display, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a cathode ray tube (CRT) display, electronic ink, or a monochrome, color, or any other type of display capable of generating visual output. In some examples, the electronic display is a stereoscopic display for providing a separate image to each eye of the user. In some examples, when tracking the position and orientation of the HMD 112 for rendering artificial reality content according to the current viewing angle of the HMD 112 and the user, the known orientation and position of the display 203 relative to the front rigid body of the HMD 112 is used as a reference system, also referred to as a local origin. In other examples, the HMD 112 can take the form of other wearable head-mounted displays, such as glasses or goggles.

[0066] like Figure 2 As further shown in , in this example, the HMD 112 also includes one or more motion sensors 206, such as one or more accelerometers (also known as inertial measurement units or "IMUs") that output data representing the current acceleration of the HMD 112, a GPS sensor that outputs data representing the position of the HMD 112, a radar or sonar that outputs data representing the distance of the HMD 112 from various objects, or other sensors that provide an indication of the position or orientation of the HMD 112 or other objects in the physical environment. In addition, the HMD 112 may include integrated image capture devices 138A and 138B (collectively referred to as "image capture devices 138"), such as cameras, laser scanners, Doppler radar scanners, depth scanners, etc., which are configured to output image data representing the physical environment. More specifically, the image capture devices 138 capture image data representing objects in the physical environment that are within the fields of view 130A, 130B of the image capture devices 138, which generally correspond to the viewing angle of the HMD 112. The HMD 112 includes an internal control unit 210, which may include an internal power supply and one or more printed circuit boards having one or more processors, memory, and hardware to provide an operating environment for performing programmable operations to process sensory data and present artificial reality content on the display 203.

[0067] In one example, according to the techniques described herein, the control unit 210 is configured to recognize a particular gesture or combination of gestures performed by a user based on the sensed data, and in response, perform an action. For example, in response to a recognized gesture, the control unit 210 may generate and render a particular user interface element overlaid on the artificial reality content for display on the electronic display 203. As explained herein, according to the techniques of the present disclosure, the control unit 210 may perform object recognition within the image data captured by the image capture device 138 to recognize the hand 132, finger, thumb, arm, or another part of the user, and track the movement, position, configuration, etc. of the recognized part to recognize the predefined gesture performed by the user. In response to recognizing the predefined gesture, the control unit 210 takes some action, such as selecting an option from a set of options associated with the user interface element, translating the gesture into input (e.g., a character), launching an application, or otherwise displaying content, etc. In some examples, the control unit 210 dynamically generates and presents a user interface element, such as a menu, in response to detecting a predefined gesture designated as a "trigger" for presenting a user interface. In other examples, control unit 210 performs these functions in response to instructions from an external device (eg, console 106 ) that may perform object recognition, motion tracking, and gesture detection, or any portion thereof.

[0068] As an example, the control unit 210 may utilize the image capture devices 138A and 138B to analyze the configuration, position, motion, and / or orientation of the hand 132 and / or arm 134 to identify menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, or menu positioning actions, etc., which may be performed by a user of the HMD 112. The control unit 210 may render a UI menu, a slidably engageable UI element, and / or a virtual hand based on the detection of the menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, and menu positioning actions.

[0069] Figure 3 It is shown Figure 1A Artificial reality systems 10 and Figure 1B A block diagram of an example implementation of the console 106 and HMD 112 of the artificial reality system 20 of FIG. Figure 3 In the example of , console 106 performs posture tracking, gesture detection, and user interface generation and rendering for HMD 112 based on sensed data, such as motion data and image data received from HMD 112 and / or external sensors, in accordance with the techniques described herein.

[0070] In this example, the HMD 112 includes one or more processors 302 and memory 304, which in some examples provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. In turn, the operating system 305 provides a multitasking operating environment for executing one or more software components 307 (including an application engine 340). As described with reference to Figure 2 As discussed in the example of , processor 302 is coupled to electronic display 203, motion sensor 206, and image capture device 138. In some examples, processor 302 and memory 304 can be separate, discrete components. In other examples, memory 304 can be on-chip memory placed in parallel with processor 302 within a single integrated circuit.

[0071] Generally speaking, the console 106 is responsible for processing the data from the camera 102 ( Figure 1B ) and / or a computing device that receives images and tracking information received by HMD 112 to perform gesture detection and user interface generation for HMD 112. In some examples, console 106 is a single computing device, such as a workstation, desktop computer, laptop computer, or gaming system. In some examples, at least a portion of console 106 (e.g., processor 312 and / or memory 314) can be distributed on a cloud computing system, a data center, or distributed on a network, such as the Internet, another public or private communication network, such as broadband, cellular, Wi-Fi, and / or other types of communication networks for transmitting data between computing systems, servers, and computing devices.

[0072] exist Figure 3In the example of , the console 106 includes one or more processors 312 and memory 314, which in some examples provide a computer platform for executing an operating system 316, which may be, for example, an embedded real-time multitasking operating system or other types of operating systems. In turn, the operating system 316 provides a multitasking operating environment for executing one or more software components 317. The processor 312 is coupled to one or more I / O interfaces 315, which provide one or more I / O interfaces for communicating with external devices such as keyboards, game controllers, display devices, image capture devices, HMDs, etc. In addition, the one or more I / O interfaces 315 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network such as the network 104. Each of the processors 302, 312 may include any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuits. The memories 304, 314 may include any form of memory for storing data and executable software instructions, such as random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), and flash memory.

[0073] The software application 317 of the console 106 operates to provide an overall artificial reality application. In this example, the software application 317 includes an application engine 320, a rendering engine 322, a gesture detector 324, a posture tracker 326, and a user interface engine 328.

[0074] In general, the application engine 320 includes functionality for providing and presenting artificial reality applications, such as teleconferencing applications, gaming applications, navigation applications, educational applications, training or simulation applications, etc. The application engine 320 may include, for example, one or more software packages, software libraries, hardware drivers, and / or application programming interfaces (APIs) for implementing artificial reality applications on the console 106. In response to control of the application engine 320, the rendering engine 322 generates 3D artificial reality content for display to the user by the application engine 340 of the HMD 112.

[0075] The application engine 320 and the rendering engine 322 construct artificial content for display to the user 110 based on the current pose information of the reference frame (typically the viewing angle of the HMD 112 determined by the pose tracker 326). Based on the current viewing angle, the rendering engine 322 constructs 3D, artificial reality content, which in some cases can be at least partially overlaid on the real world 3D environment of the user 110. During this process, the pose tracker 326 performs sensory data received from the HMD 112 (such as movement information and user commands) and, in some examples, from any external sensors 90 ( Figure 1A , Figure 1B ) (such as an external camera) to capture 3D information within the real-world environment, such as the movement of the user 110 and / or feature tracking information relative to the user 110. Based on the sensed data, the pose tracker 326 determines the current pose of the reference frame of the HMD 112, and based on the current pose, constructs artificial reality content for transmission to the HMD 112 via one or more I / O interfaces 315 for display to the user 110.

[0076] In addition, based on the sensed data, the gesture detector 324 analyzes the tracked motion, configuration, position, and / or orientation of the user's objects (e.g., hands, arms, wrists, fingers, palms, thumbs) to identify one or more gestures performed by the user 110. More specifically, the gesture detector 324 analyzes objects identified in the image data captured by the image capture device 138 and / or the sensors 90 and external cameras 102 of the HMD 112 to identify the hands and / or arms of the user 110, and tracks the movement of the hands and / or arms relative to the HMD 112 to identify gestures performed by the user 110. The gesture detector 324 may track the movement of the hands, fingers, and / or arms based on the captured image data, including changes in position and orientation, and compare the motion vectors of the objects to one or more entries in the gesture library 330 to detect a gesture or combination of gestures performed by the user 110. Some entries in the gesture library 330 may each define a gesture as a series or pattern of motion, such as a relative path or spatial translation and rotation of a user's hand, specific fingers, thumbs, wrists, and / or arms. Some entries in the gesture library 330 can each define a gesture as a configuration, position, and / or orientation of a user's hand and / or arm (or part thereof) at a particular time or over a period of time. Other examples of gesture types are also possible. In addition, each entry in the gesture library 330 can specify the conditions required for the gesture or series of gestures that trigger the action for a defined gesture or series of gestures, such as the spatial relationship with the current field of view of the HMD 112, the spatial relationship with a specific area that the user is currently observing, which can be determined by real-time gaze tracking of the individual, the type of artificial content being displayed, the type of application being executed, etc.

[0077] Each entry in gesture library 330 may also specify, for each defined gesture or combination / series of gestures, a desired response or action to be performed by software application 317. For example, in accordance with the techniques of this disclosure, certain specialized gestures may be predefined such that, in response to detecting one of the predefined gestures, user interface engine 328 dynamically generates a user interface as an overlay of artificial reality content displayed to the user, thereby allowing user 110 to easily invoke a user interface for configuring HMD 112 and / or console 106 even while interacting with artificial reality content. In other examples, certain gestures may be associated with other actions, such as providing input, selecting an object, launching an application, etc.

[0078] As an example, the gesture library 330 may include entries describing menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, and menu positioning actions. The gesture detector 324 may process image data from the image capture device 138 to analyze the configuration, position, motion, and / or orientation of the user's hand, thereby identifying menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, and menu positioning actions that may be performed by the user, etc. The rendering engine 322 may render menus and virtual hands based on the detection of menu prompt gestures, menu activation gestures, menu sliding gestures, and menu positioning actions. The user interface engine 328 may define displayed menus and may control actions performed in response to selections caused by selection gestures.

[0079] Figure 4 It is a description of the technology according to the present disclosure. Figure 1A and Figure 1B 1 is a block diagram of an example of an artificial reality system in which an HMD 112 performs gesture detection and user interface generation.

[0080] In this example, something like Figure 3 , HMD 112 includes one or more processors 302 and memory 304, which in some examples provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system or other types of operating systems. In turn, operating system 305 provides a multitasking operating environment for executing one or more software components 417. In addition, processor 302 is coupled to electronic display 203, motion sensor 206, and image capture device 138.

[0081] exist Figure 4 In the example of , software component 417 operates to provide an overall artificial reality application. In this example, software application 417 includes application engine 440, rendering engine 422, gesture detector 424, posture tracker 426, and user interface engine 428. In various examples, software component 417 is similar to Figure 3 The corresponding components of the console 106 (e.g., the application engine 320, the rendering engine 322, the gesture detector 324, the posture tracker 326, and the user interface engine 328) of the user 110 are operated to construct a user interface element overlaid on or as part of the artificial content based on the detected gestures of the user 110 for display to the user 110. In some examples, the rendering engine 422 constructs 3D artificial reality content, which can be at least partially overlaid on the real world physical environment of the user 110.

[0082] Similar to reference Figure 3 In the described example, based on the sensed data, the gesture detector 424 analyzes the tracked motion, configuration, position, and / or orientation of the user's object (e.g., hand, arm, wrist, finger, palm, thumb) to identify one or more gestures performed by the user 110. In accordance with the techniques of the present disclosure, the user interface engine 428 generates a user interface element as part of the artificial reality content to be displayed to the user 110, such as overlaid thereon, and / or performs an action based on one or more gestures or gesture combinations of the user 110 detected by the gesture detector 424. More specifically, the gesture detector 424 analyzes objects identified in image data captured by the image capture device 138 and / or the sensor 90 or the external camera 102 of the HMD 112 to identify the hand and / or arm of the user 110, and tracks the movement of the hand and / or arm relative to the HMD 112 to identify the gesture performed by the user 110. Gesture detector 424 can track the movement of hands, fingers and / or arms, including changes in position and orientation, based on captured image data and compare the motion vector of the object to one or more entries in gesture library 430 to detect a gesture or combination of gestures performed by user 110.

[0083] Gesture library 430 is similar to Figure 3 The gesture library 330 of FIG. Each entry in the gesture library 430 may specify, for a defined gesture or series of gestures, the conditions required for the gesture to trigger an action, such as a spatial relationship to the current field of view of the HMD 112, a spatial relationship to a particular area that the user is currently viewing, which may be determined by real-time gaze tracking of the individual, the type of artificial content being displayed, the type of application being executed, etc.

[0084] In response to detecting a matching gesture or combination of gestures, HMD 112 performs a response or action assigned to a matching entry in gesture library 430. For example, according to the techniques of this disclosure, certain specialized gestures may be predefined such that in response to gesture detector 424 detecting one of the predefined gestures, user interface engine 428 dynamically generates a user interface as an overlay of artificial reality content displayed to the user, thereby allowing user 110 to easily invoke a user interface for configuring HMD 112 while viewing artificial reality content. In other examples, in response to gesture detector 424 detecting one of the predefined gestures, user interface engine 428 and / or application engine 440 may receive input, select a value or parameter associated with a user interface element, launch an application, modify a configurable setting, send a message, start or stop a process, or perform other actions.

[0085] As an example, the gesture library 430 may include entries describing menu prompt gestures, menu activation gestures, menu sliding gestures, menu positioning actions, and selection gestures. The gesture detector 424 may analyze the configuration, position, and / or orientation of the user's hand using image data from the image capture device 138 to identify menu prompt gestures, menu activation gestures, menu sliding gestures, selection gestures, or menu positioning actions that may be performed by the user, etc. The rendering engine 422 may render UI menus, slidably engaged elements, and / or virtual hands based on the detection of menu activation gestures, menu sliding gestures, selection gestures, and menu positioning actions. The user interface engine 328 may define displayed menus and may control the actions performed by the application engine 440 in response to selections caused by selection gestures.

[0086] Figure 5 and Figure 6 is a flow chart illustrating an example method for activating menu prompts and menus, and for determining the location and user interaction of menus. Figure 5 and Figure 6 The operations shown can be performed by an artificial reality system (e.g. Figure 1A Artificial reality systems 10 and Figure 1B For example, some or all operations may be performed by a gesture detector ( Figure 3 324 and Figure 4 424), user interface engine ( Figure 3 328 and Figure 4 428) and the rendering engine ( Figure 3 322 and Figure 4 422 ) is performed by one or more.

[0087] Figure 5is a flow chart illustrating the operation of an example method for activating a menu prompt or menu interface according to aspects of the present disclosure. As described above, certain configurations of a hand can be detected and used to trigger activation of a menu interface or menu prompt. The artificial reality system can determine a current configuration of a hand (502). The configuration can include the orientation of the hand and the positions of the fingers of the hand relative to each other. In one or more aspects, image data can be captured and analyzed to determine the configuration of the hand. In addition to image data, or in lieu of image data, other sensor data can be used to determine the configuration of the hand.

[0088] The artificial reality system can determine whether the current configuration of the hand indicates that the user is performing a menu prompt gesture (504). In one or more aspects, the artificial reality system can be configurable (e.g., by the user) to determine the configuration of the left hand or the right hand. In one or more aspects, the artificial reality system can utilize data describing the current configuration of the hand and data in one or more entries of a gesture library that specifies a particular gesture to determine whether the current configuration of the hand is a menu prompt gesture. In one or more aspects, the menu prompt gesture can be a configuration of the hand in which the hand is in a substantially upward orientation and the fingers and thumb of the user's hand are positioned so that there is space between the fingers and the thumb. For the menu prompt gesture, the fingers and thumb of the user's hand can form a "C" shape or a clamp shape in which the fingers and thumb do not touch at the ends. Those skilled in the art who benefit from the present disclosure will understand that other configurations of the hand can be used as menu prompt gestures.

[0089] In response to determining that the user has performed a menu prompt gesture, the artificial reality system can render a menu prompt (506). In one or more aspects, the menu prompt is rendered near a virtual hand representing the orientation of the user's hand. The menu prompt can be a UI element located between a virtual finger and a virtual thumb of a virtual hand corresponding to the finger and thumb of the user performing the menu prompt gesture.

[0090] Figure 8 8 is an example HMD display 800 showing a menu prompt 810 according to aspects of the present disclosure. Figure 8In the example shown, the user of the artificial reality system has placed their hand in a substantially upward orientation with space between the index finger and thumb. (Other fingers next to the index finger can be used for menu prompts and menu activation gestures). The artificial reality system can determine the position and orientation of the hand, and can render the virtual hand 136 to match the orientation of the user's hand and the positioning of the fingers. In addition, the artificial reality system can detect that the user has performed a menu prompt gesture with their hand based on the configuration of the hand. In response to detecting the menu prompt gesture, the artificial reality system can render a menu prompt 810 between the index finger and thumb of the virtual hand. The menu prompt 810 can be a user interface element that serves as an indication or reminder (i.e., a prompt) to the user that the user can perform an action (e.g., a pinch action) with the thumb and index finger to place the user's hand in a menu activation gesture, so that the artificial reality system provides a menu to the user. In some aspects, the menu prompt 810 may include a line extending between the index finger and the thumb. In some aspects, the menu prompt 810 may include a virtual object located between the thumb and the index finger. In some aspects, the menu prompt 810 may include highlighting the index finger and / or thumb. Other types of user interface elements may be rendered as menu prompts 810. For example, arrows may be used to indicate the direction in which the user's index finger and thumb should move in order to activate a menu.

[0091] Back to Figure 5 , after rendering the menu prompt, the artificial reality system may determine a new current configuration of the hand (502). Between rendering the menu prompt and determining the new configuration of the hand, there may be many other operations performed by the artificial reality system.

[0092] If the current configuration of the hand does not match the menu prompt gesture, the artificial reality system can determine whether the current configuration of the hand indicates that the user is performing a menu activation gesture. In one or more aspects, the artificial reality system can utilize data describing the current configuration of the hand and data in one or more entries of the gesture library to determine whether the current configuration of the hand is a menu activation gesture. In one or more aspects, the menu activation gesture can be a configuration of the hand in which the hand is in a substantially upward orientation and the fingers and thumb of the user's hand are in a pinch configuration. Those skilled in the art who benefit from the present disclosure will understand that other configurations of the hand can be used as menu activation gestures. For example, the menu activation gesture can include the fingers and thumb positioned in a pinch configuration, regardless of the orientation of the hand.

[0093] In response to determining that the user has performed a menu activation gesture, the artificial reality system can render a UI menu (510). In one or more aspects, the menu is rendered near a virtual hand representing the orientation of the user's hand. In some aspects, the artificial reality system can render the UI menu in response to detecting the menu activation gesture only if the artificial reality system first detects the menu prompt gesture. In some aspects, the menu prompt gesture is not a prerequisite.

[0094] Fig. 7A is an example HMD display 700 depicting a UI menu 124 according to aspects of the present disclosure. Fig. 7A In the example shown, a user of the artificial reality system has positioned their hand in a substantially upward orientation with the index finger and thumb of the hand in a pinch configuration. (Again, the index finger is used as an example of a finger of a hand.) The artificial reality system can determine the position and orientation of the hand, and can render a virtual hand 136 to represent the orientation and finger positioning of the user's hand. In addition, the artificial reality system can detect that the user has performed a menu activation gesture with their hand based on the configuration of the hand. In response to detecting the menu activation gesture, the artificial reality system can render a UI menu 124 near the virtual hand. The UI menu 124 may include one or more UI elements 126 arranged along a dimension of the UI menu 124. In one or more aspects, the one or more UI elements 126 may be menu items arranged along a horizontal dimension of a coordinate space, such as a viewing space or a display space. In Fig. 7A In the example shown, the coordinate axis 704 is shown only to illustrate the coordinate space. The coordinate axis 704 does not need to be presented on the actual display. Figure 7A-7G In the example shown, the horizontal dimension is along the X-axis, the vertical dimension is along the Y-axis, and the depth is along the Z-axis.

[0095] As described above, the menu activation gesture can include the user placing their hand in a substantially upward orientation. For example, the artificial reality system can detect that vector 702 of the palm of the hand, which is perpendicular to the other surfaces of the hand, is also substantially perpendicular to the plane formed by the X-axis and the Z-axis. In one or more aspects, vector 702 can be considered substantially vertical if it is within thirty degrees of the normal to the plane formed by the X-axis and the Z-axis (shown by the dashed line). In one or more aspects, other thresholds other than thirty degrees can be used.

[0096] In one or more aspects, a slidably engageable UI element 706 may be rendered near the virtual hand 136. Fig. 7AIn the example shown, the slidably engageable UI element 706 is a circle. Other graphical elements (such as a sphere, a triangle, a square, etc.) or a separate virtual hand 136 can be used as the slidably engageable UI element 706. In addition, one or more fingers of the virtual hand 136 can be highlighted to indicate that the highlighted portion of the one or more fingers is a slidably engageable UI element.

[0097] Figure 7B 7 is an example HMD display 740 showing a UI menu and slidably engaged UI elements according to aspects of the present disclosure. Figure 7B In the example shown, the user has performed a menu sliding gesture so that the artificial reality system renders a slidably engaged UI element 706 at a position near a menu item 708. In one or more aspects, the menu item 708 near the slidably engaged UI element 706 can be highlighted or otherwise enhanced or modified to indicate that the menu item 708 will be selected when the user performs a selection gesture 708. In addition to highlighting the menu item 708, or in lieu of highlighting the menu item 708, a label 710 can be provided near the menu element 708. Various highlighting mechanisms can be used, including border highlighting, background highlighting, flashing, amplification, etc., which can be used to highlight the menu item 708. Highlighting the menu element 708 can indicate that if the user performs a selection gesture, the menu element 708 will be selected. In one or more aspects, the selection gesture can be the movement of a finger of a hand other than the finger in the pinch configuration, for example, releasing the pinch configuration. In one or more aspects, the selection gesture can be the movement of a hand in a direction substantially perpendicular to the UI menu plane. As used herein, "substantially perpendicular to" a plane may mean within 0-2 degrees of the normal to the plane, within 0-5 degrees of the normal to the plane, within 0-10 degrees of the normal to the plane, within 0-20 degrees of the normal to the plane, or within 0-30 degrees of the normal to the plane. In one or more aspects, the selection gesture may be a reconfiguration of the thumb and fingers of the hand to no longer be in a pinch configuration. In one or more aspects, the selection gesture may be a movement or reconfiguration of a different finger (e.g., a pinky), such as a curl or stretch. Detection of a selection gesture may cause the artificial reality system to perform some action. For example, a selection gesture may cause an application to be instantiated, or may cause a currently running application to be brought to the foreground of the display of the HMD, or in some cases may cause the artificial reality system to perform some action within a specifically executed artificial reality application.

[0098] Thus, a sequence of gestures may be used to trigger display of menu 124, position slidably engaged UI element 706 over or near menu element 708 of menu 124, and select menu element 708. In an example implementation, a user may perform a menu activation gesture (e.g., position the fingers of a hand in a pinch configuration) to cause menu 124 to be displayed by the HMD. A user may perform a menu sliding gesture (e.g., move their hand while maintaining a pinch configuration) to cause slidably engaged UI element 706 to slide along menu 124 in accordance with the motion of the hand. The user may then perform a selection gesture (e.g., release the fingers and thumb from the pinch configuration) to select the menu element 708 indicated by slidably engaged UI element 706.

[0099] Back to Figure 5 , after rendering the UI menu 124, the artificial reality system may determine a new current configuration of the hand (502). Between rendering the UI menu 124 and determining the new configuration of the hand, there may be many other operations performed by the artificial reality system.

[0100] If the artificial reality system detects that the user's hand is no longer performing the menu prompt gesture or the menu activation gesture, the artificial reality system can determine whether the UI menu 124 or menu prompt 810 is already displayed. If so, the artificial reality system can remove the UI menu 124 or menu prompt 810 from the display (514) because the user's hand is no longer in the appropriate configuration to display the UI menu 124 or menu prompt 810.

[0101] After removing the UI menu 124 or menu prompt 810, the process returns to determine the new current configuration of the hand (502). Between removing the UI menu 124 or menu prompt 810 and determining the new configuration of the hand, there may be many other operations performed by the artificial reality system.

[0102] Figure 6 602 is a flow chart illustrating the operation of an example method for locating and interacting with a UI menu in accordance with aspects of the present disclosure. The artificial reality system may determine a position of a hand (602). For example, the position may be determined based on image data captured from an image sensor or from another type of sensor coupled to the artificial reality system.

[0103] The artificial reality system may determine whether the UI menu is currently active (i.e., being rendered and displayed by the HMD) (604). If the UI menu is not currently active, the process may return to determining the updated position of the hand (602). Between determining whether the UI menu is active and determining the updated position of the hand, there may be many other operations performed by the artificial reality system.

[0104] If the UI menu is active, the artificial reality system can determine whether the user has performed a menu sliding gesture (606). In some aspects, the menu sliding gesture can be a substantially horizontal movement of the hand when the menu is active (e.g., when the user's hand is performing a menu activation gesture). For example, the artificial reality system can compare the previous position of the hand with the current position of the hand to determine whether the menu sliding gesture has occurred. If a menu sliding gesture is detected, the artificial reality system can translate the virtual hand and / or the slidably engaged interface elements along the UI menu 124 according to the menu sliding gesture (608). If the menu items are vertically oriented, the menu sliding gesture can be a substantially vertical movement of the hand. If a menu sliding gesture is not detected, the artificial reality system can determine whether other movements of the hand that are not menu sliding gestures are detected (610). "Substantially" vertical and horizontal can be within 5 degrees, 10 degrees, 20 degrees, or 30 degrees of vertical or horizontal.

[0105] Figure 7C 7 is an example HMD display 750 showing a UI menu and a menu sliding gesture according to aspects of the present disclosure. In one or more aspects, the menu sliding gesture can be a movement of a user's hand along the horizontal dimension of the UI menu 124 (e.g., movement parallel to the X-axis). Figure 7C In the example shown, the user has moved their hand along the horizontal dimension of UI menu 124 (while maintaining the menu activation gesture). The artificial reality system can reposition the virtual hand 136 and the slidably engaged UI element 706 according to the movement of the user's hand so that the virtual hand 136 and the slidably engaged UI element 706 are close to the menu item 712. The artificial reality system can remove the highlight from the menu element 708, and can highlight the menu element 712 to indicate that the menu element 712 will be selected if the user performs a selection gesture. In addition to highlighting the menu item 712, or instead of highlighting the menu item 712, a label 714 can be displayed. In some examples, the artificial reality system does not highlight the menu item. In some examples, the artificial reality system does not render the UI element 706. In one or more aspects, the UI menu 124 remains in the same position in the horizontal direction as before the menu sliding gesture. In other words, the UI menu 124 is horizontally stationary, while the virtual hand 136 and the slidably engaged UI element 706 move along the horizontal dimension of the UI menu 124 in response to the user performing the menu sliding gesture.

[0106] Back to Figure 6, after determining whether horizontal motion has been detected, the artificial reality system can determine whether non-horizontal motion of the user's hand has occurred (610). For example, the artificial reality system can determine whether there is motion in the vertical direction (i.e., motion parallel to the Y axis) and / or motion from front to back or from back to front (i.e., motion parallel to the Z axis). If non-horizontal motion of the user's hand is detected, the artificial reality system can translate the position of the virtual hand, slidably engageable UI elements, and UI menus based on the non-horizontal motion. In the example where the UI menu items are arranged vertically, the non-vertical motion of the user's hand constitutes "other movement" of the hand.

[0107] Fig.7D is an example HMD display 760 showing a UI menu after vertical motion has been detected. Fig.7D In the example shown, the user has moved their hand downward. The artificial reality system can detect vertical motion and can translate the positions of UI menu 124, virtual hand 136, and slidably engaged UI element 706 based on the detected vertical motion. In some aspects, if no horizontal motion is detected in addition to the vertical motion, virtual hand 136 and slidably engaged UI element 706 remain in their previous positions relative to UI menu 124. Therefore, when vertical motion occurs, the same menu element 712 remains highlighted.

[0108] Fig. 7E 7 is an example HMD display 770 showing a UI menu after back-to-forward motion has been detected (i.e., the user has moved their hand closer to themselves along the Z axis). The artificial reality system can detect the back-to-forward motion, and can translate the positions of the UI menu 124, the virtual hand 136, and the slidably engaged UI element 706 based on the detected back-to-forward motion. As a result, the positions of the virtual hand, the UI menu 124, and the slidably engaged UI element 706 appear larger and therefore closer to the user. In some aspects, if no horizontal motion is detected in addition to the motion along the Z axis, the virtual hand 136 and the slidably engaged UI element 706 remain in their previous positions relative to the UI menu 124. Thus, the same menu element 712 remains highlighted as motion along the Z axis occurs.

[0109] Back to Figure 6 , after the UI menu, virtual hand, and slidably engaged UI elements have been translated according to the motion of the user's hand (if any), the artificial reality system can render the virtual hand and slidably engaged UI elements near the UI menu 124 based on the current position of the user's hand. The process can then return to determine the updated position of the hand (602). Between rendering the UI menu and determining the updated position of the hand, there can be many other operations performed by the artificial reality system.

[0110] Figure 7F 7 is an example HMD display 780 showing a UI menu 124 and a UI icon array 720 according to aspects of the present disclosure. In some aspects, the menu items of the UI menu 124 may correspond to applications. In one or more aspects, the UI menu 124 may be divided into two portions 716 and 718. The menu elements in portion 716 may represent favorite applications, and the menu elements in portion 718 may represent applications currently running in the artificial reality system. In addition, in some aspects, the artificial reality system may present an icon array 710 representing icons of applications available or running in the artificial reality system. The images on the individual icons in the icon array 720 may represent the current display of the corresponding application, or an image associated with the application.

[0111] Figure 7G is an example HMD display 790 showing a UI menu 124 and a UI icon array 720 according to aspects of the present disclosure. Figure 7G In the example shown, the virtual hand 136 and the slidably engaged UI element 706 are near the menu item 724. In this example, a three-dimensional highlight is used, where the menu items near the slidably engaged UI element 706 can be brought forward, making the image appear larger to the user. In addition, the icon 722 corresponding to the menu item 724 can also be highlighted. In this example, the border of the icon 722 is highlighted.

[0112] The above discussion has presented aspects of artificial reality systems in which the UI menu is configured in the horizontal direction. In other aspects, the UI menu can be configured in the vertical direction. In such aspects, the vertical movement of the hand can cause the slidably engaged UI elements and the virtual hand to move along the vertical dimension of the UI menu while the UI menu remains stationary in the vertical direction. Non-vertical movement (i.e., horizontal movement or front-to-back movement) can cause the position of the UI menu to translate according to the non-vertical movement.

[0113] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the techniques may be implemented in one or more processors, including one or more microprocessors, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, and any combination of such components. The term "processor" or "processing circuit" may generally refer to any of the aforementioned logic circuits (alone or in combination with other logic circuits), or any other equivalent circuit. A control unit including hardware may also perform one or more techniques of the present disclosure.

[0114] Such hardware, software, and firmware may be implemented in the same device or in separate devices to support the various operations and functions described in this disclosure. In addition, any of the units, modules, or components may be implemented together or individually as discrete but interoperable logical devices. Describing different features as modules or units is intended to highlight different functional aspects and does not necessarily mean that such modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with one or more modules or units may be performed by separate hardware or software components, or integrated in common or separate hardware or software components.

[0115] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. The instructions embedded or encoded in the computer-readable storage medium may, for example, cause a programmable processor or other processor to perform the method when the instructions are executed. The computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer-readable media.

[0116] As described herein by various examples, the technology of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. As described, artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., a photo of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., a stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content may be implemented on a variety of platforms, including a head mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

Claims

1. An artificial reality system, comprising: an image capture device configured to capture image data; a head mounted display (HMD) configured to output artificial reality content; a gesture detector configured to identify a menu activation gesture from the image data, the menu activation gesture comprising a configuration of the hand with the hand in a substantially upward orientation and a pinching configuration of a thumb and fingers of the hand; a user interface (UI) engine configured to generate, in response to the menu activation gesture, a menu interface and a slidably engaged UI element in a first position relative to the menu interface; and A rendering engine configured to render the artificial reality content, the menu interface, and the slidably engageable UI element for display at the HMD.

2. The artificial reality system according to claim 1, in, The menu interface includes one or more menu items arranged along a dimension of the menu interface, and wherein the UI engine is configured to highlight one of the menu items based on a position of the slidably engageable UI element relative to the menu interface; and preferably, wherein the one or more menu items correspond to corresponding applications executed on the artificial reality system.

3. The artificial reality system according to claim 1 or claim 2, in, The gesture detector is further configured to identify a menu sliding gesture from the image data, the menu sliding gesture comprising a hand configuration combined with a hand movement, and Wherein, the UI engine is further configured to translate the slidably engaged UI element to a second position relative to the menu interface in response to the menu sliding gesture; and preferably The menu interface includes one or more menu items arranged along a dimension of the menu interface. Wherein, in order to translate the slidably engaged UI element to the second position relative to the menu interface, the UI engine is configured to slide the slidably engaged UI element to the second position relative to the menu interface along the dimension of the menu interface; and / or preferably wherein the menu sliding gesture comprises a movement of a hand in substantially a first direction, wherein the gesture detector is configured to identify from the image data a movement of the hand in a substantially second direction following the menu activation gesture, the substantially second direction being substantially perpendicular to the substantially first direction, Wherein, the UI engine is also configured to translate the slidably engaged UI element and the menu interface in response to movement of the hand in the substantially second direction, while maintaining a position of the slidably engaged UI element relative to the menu interface.

4. The artificial reality system of claim 1, claim 2 or claim 3, further comprising: an application engine for executing one or more artificial reality applications, wherein the gesture detector is configured to identify a selection gesture subsequent to the menu sliding gesture from the image data, and wherein the application engine is configured to perform an action in response to the selection gesture; and preferably, wherein the selection gesture comprises one of: (1) movement of different fingers of a hand, (2) translation of the hand in a direction substantially perpendicular to the menu interface, or (3) reconfiguring the thumb and fingers of the hand to no longer be in the pinch configuration.

5. The artificial reality system according to any one of claims 1 to 4, in, The gesture detector is further configured to identify from the image data a menu prompting gesture preceding the menu activation gesture, and wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture; and preferably, wherein the menu prompt gesture includes the hand being configured in a substantially upward position with a space between the thumb and fingers, and Wherein, the UI engine generates the menu prompt element in the space between the thumb and fingers of the virtual hand; and preferably, wherein the menu prompt element includes a line between the thumb and fingers of the virtual hand.

6. The artificial reality system according to any one of claims 1 to 5, wherein: The image capture device is integrated with the HMD.

7. A method comprising: obtaining image data via an image capture device through an artificial reality system including a head mounted display (HMD); identifying, by the artificial reality system from the image data, a menu activation gesture, the menu activation gesture comprising a configuration of a hand with a hand in a substantially upward orientation and a pinching configuration of a thumb and fingers of the hand; generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engaged UI element at a first position relative to the menu interface; and Artificial reality content, the menu interface, and the slidably engageable UI element are rendered by the artificial reality system for display at the HMD.

8. The method according to claim 7, wherein: The menu interface includes one or more menu items arranged along a dimension of the menu interface, and the method further includes: One of the menu items is highlighted by the artificial reality system based on a position of the slidably engageable UI element relative to the menu interface.

9. The method according to claim 7 or claim 8, further comprising: Recognizing, by the artificial reality system, a menu sliding gesture from the image data, the menu sliding gesture comprising a hand configuration combined with a hand movement; and The artificial reality system translates the slidably engaged UI element to a second position relative to the menu interface in response to the menu slide gesture.

10. The method according to claim 9, wherein: Translating the slidably engaged UI element to the second position relative to the menu interface includes sliding the slidably engaged UI element along a dimension of the menu interface to the second position relative to the menu interface.

Images