Projection projection in virtual environment

Abstract

The embodiment of the invention relates to projection projection in a virtual environment. The present technology relates to an artificial reality system. Such a system provides a projection that can be created by a user to specify object interactions. For example, when a user wishes to interact with objects outside of her direct contact range, her may use the projection to select, move, or otherwise interact with distant objects. The techniques also include object selection techniques for identifying and disambiguating between objects, allowing a user to select objects close to and away from the user. Yet another aspect of the present technology includes techniques for interpreting various two-hand (two-hand operated) gestures interacting with an object. The techniques also include models to distinguish between global and local modes to, for example, provide interpretations of different input modalities or user gestures.

Description

[0001] Division Statement

[0002] This application is a divisional application of a PCT patent application with an application date of September 21, 2020, a Chinese national phase application number of 202080054646.1, and an invention name of “Projection in a Virtual Environment”.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application is related to U.S. Patent Application No. 16 / 583,069, filed on September 25, 2019, with Attorney Docket No. 134875-8002.US02, entitled “PROJECTION CASTING IN VIRTUAL ENVIRONMENTS”; U.S. Patent Application No. 16 / 578,236, filed on September 20, 2019, with Attorney Docket No. 134875-8003-US01, entitled “GLOBAL AND LOCAL MODEVIRTUAL OBJECT INTERACTIONS”; and U.S. Patent Application No. 16 / 578,236, filed on September 20, 2019, with Attorney Docket No. 134875-8004-US01, entitled “THREE-STATE GESTURE VIRTUAL CONTROLS”; U.S. Patent Application No. 16 / 578,251, filed on September 20, 2019, attorney docket number 134875-8005-US01, entitled “PROGRESSIVE DISPLAY OF VIRTUAL OBJECTS”; and U.S. Patent Application No. 16 / 578,260, filed on September 20, 2019, attorney docket number 134875-8006-US01, entitled “VIRTUAL INTERACTIONS AT A DISTANCE”; all of which are incorporated herein by reference in their entirety. Technical Field

[0005] The present disclosure is directed to object interaction in an artificial reality environment. Background Art

[0006] In an artificial reality environment, some or all objects that a user sees and interacts with are "virtual objects", that is, representations of objects that appear in the environment generated by a computing system. Virtual objects in an artificial reality environment can be presented to a user through a head-mounted display, a mobile device, a projection system, or other computing systems. Typically, a user can interact with a virtual object using a controller and / or gestures. For example, a user's "interaction" with a virtual object can include selecting, moving, rotating, resizing, activating controls, changing color or skin, defining interactions between real or virtual objects, setting virtual forces to act on virtual objects, or making changes or any other actions to objects that the user can imagine. In some systems, a user can also interact with a "real object" that exists independently of a computer system that controls the artificial reality environment. For example, a user can select a real object and add a virtual overlay to change the color of the object or present the object to the user in some other way so that it affects other virtual objects, etc. As used herein, unless otherwise specified, an "object" can be a real object or a virtual object. Summary of the invention

[0007] According to the present invention, a method for interacting with an object in an artificial reality environment is provided, the method comprising: tracking a portion of a user's hand and setting a control point based on the tracked portion of the user's hand; tracking points on the user's body and setting an origin based on the tracked points on the user's body; determining a projection orientation centered on a line passing through the control point and the origin; projecting a ray projection from the control point outward from the user along the projection orientation; and performing an action related to one or more objects based on one or more positions of the ray projection.

[0008] Preferably, the point on the user's body is a first point on the user's body, and wherein the method further comprises: identifying one or more origin conversion factors; determining, based on the one or more origin conversion factors, that the origin should be moved to a second tracking point on the user's body; and moving the origin from the first point on the user's body to the second point on the user's body.

[0009] Conveniently, moving the origin comprises moving the origin at a set speed relative to a path connecting a first point on the user's body to a second point on the user's body.

[0010] Preferably, the first point on the user's body is the user's eyes or shoulders; the second point on the user's body is the user's hips; and the path connects A) the user's eyes or the user's shoulders to B) the user's hips via a straight line.

[0011] Conveniently, the one or more origin conversion factors include the tracked user gaze direction.

[0012] Preferably, determining that the origin should move comprises: determining an angle associated with a user's gaze direction; determining that the angle is above a threshold; and in response, selecting a high interaction mode in which a second point on the user's body is lower than a first point on the user's body.

[0013] Conveniently, the one or more origin conversion factors include a measurement of the angle of a line passing through the control point and the origin relative to a specified plane; and determining that the origin should be moved includes: determining that the angle is below a threshold; and in response, selecting a low interaction mode in which a second point on the user's body is higher than a first point on the user's body.

[0014] Preferably, the part of the user's hand that is tracked relative to the control point is the user's fingertip.

[0015] Conveniently, the part of the user's hand that is tracked relative to the control point is the user's wrist.

[0016] Preferably, setting the control point based on the tracked portion of the user's hand comprises selecting a control point at a location that is a specified offset from the portion of the user's hand.

[0017] Conveniently, the point on the user's body that is tracked relative to the origin is the user's dominant eye.

[0018] Preferably, the point on the user's body that is tracked relative to the origin is the user's shoulder.

[0019] Conveniently, the point on the user's body that is tracked relative to the origin is the user's hip.

[0020] Preferably, performing an action related to the one or more objects comprises causing the one or more objects to be selected.

[0021] According to another aspect of the present invention, a computer-readable storage medium storing instructions is provided, which, when executed by a computing system, causes the computing system to perform operations for interacting with objects in an artificial reality environment, the operations including: setting a control point based on the position of a portion of a user's hand; setting an origin based on a point on the user's body; determining a projection orientation centered on a straight line passing through the control point and the origin; and performing actions related to one or more objects based on one or more positions of the projection.

[0022] Preferably, the point on the user's body is a first point on the user's body, and wherein the operation further comprises: identifying one or more origin conversion factors; determining, based on the one or more origin conversion factors, that the origin should be moved to a second point on the user's body along a path connecting the first point on the user's body and the second point on the user's body; and moving the origin along the path at a set speed, in which case optionally, the one or more origin conversion factors include a tracked user gaze direction; and determining that the origin should be moved comprises: measuring an angle associated with the user gaze direction; and based on the determined angle, selecting an interaction mode in which the second point on the user's body is higher than the first point on the user's body.

[0023] Conveniently, the part of the hand is one of the user's fingertips; the user's palm; the user's wrist; or the user's fist; and the point on the body is one of the user's dominant eye, the user's shoulder or the user's hip.

[0024] According to another aspect of the present invention, a computing system is provided, comprising: one or more processors; and a memory storing instructions, which, when executed by the one or more processors, cause the computing system to perform the following operations: set a control point based on the position of a portion of a user's hand; set an origin based on a point on the user's body; and project a projection centered on a straight line passing through the control point and the origin.

[0025] Preferably, the point on the user's body is a first point on the user's body, and wherein the operation further comprises: identifying one or more origin conversion factors; determining, based on the one or more origin conversion factors, that the origin should be moved to a second point on the user's body along a path connecting the first point on the user's body and the second point on the user's body; and moving the origin along the path at a set speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram illustrating an overview of a device upon which some implementations of the present technology may operate.

[0027] Figure 2A is a wiring diagram illustrating a virtual reality headset that may be used in some implementations of the present technology.

[0028] Figure 2B is a wiring diagram illustrating a mixed reality headset that may be used in some implementations of the present technology.

[0029] Figure 3 is a block diagram illustrating an overview of an environment in which some implementations of the present technology may operate.

[0030] Figure 4 is a block diagram illustrating components that, in some implementations, may be used in a system employing the disclosed techniques.

[0031] Figure 5 is a flow chart illustrating a projection process using an origin and control points for projection used in some implementations of the present technology.

[0032] Figure 6 is a conceptual diagram illustrating an example ray projection with a shoulder origin and a palm control point.

[0033] Figure 7 is a conceptual diagram illustrating an example ray projection with a dominant eye origin and a fingertip control point.

[0034] Figure 8 is a conceptual diagram illustrating an example ray projection with a hip origin and a fist control point.

[0035] Fig. 9 is a conceptual diagram illustrating an example ray projection with a shoulder origin and a wrist control point.

[0036] Fig.10 is a flow chart illustrating a process for adjusting the projection origin used in some implementations of the present technology.

[0037] Fig.11A , Fig. 11B and Fig. 11C is a conceptual diagram illustrating an example of adjusting the projection origin between the hip and the shoulder.

[0038] Fig.12 is a flow chart illustrating a process for object interaction with ray casting used in some implementations of the present technology.

[0039] Fig.13 is a conceptual diagram illustrating an example ray projection.

[0040] Fig.14 is a flow chart illustrating a process for object interaction with a cylindrical projection used in some implementations of the present technology.

[0041] Fig.15 is a conceptual diagram illustrating an example cylindrical projection.

[0042] Fig.16 is a flow chart illustrating a process for object interaction with conic projections used in some implementations of the present technology.

[0043] Fig.17 is a conceptual diagram illustrating an example conic projection.

[0044] Fig.18 is a flow diagram illustrating a process for context-based selection of a projection type used in some implementations of the present technology.

[0045] Fig.19 is a flow diagram illustrating a process used in some implementations of the present technology for selecting an object using a cut gesture.

[0046] Fig. 20 is a conceptual diagram illustrating an example of selecting an object using a cut gesture.

[0047] Fig.21 is a flow diagram illustrating a process used in some implementations of the present technology for actuating an object using a shear click gesture.

[0048] Fig. 22 is a conceptual diagram illustrating an example of actuating an object using a shear click gesture.

[0049] Fig.23 is a flow diagram illustrating a process used in some implementations of the present technology for selecting an object using a lasso gesture.

[0050] Fig.24A and Fig. 24B is a conceptual diagram illustrating an example of selecting an object using a lasso gesture.

[0051] Fig.25 is a flow diagram illustrating a process used in some implementations of the present technology for selecting an object using a frame gesture.

[0052] Fig.26A and Fig.26B is a conceptual diagram illustrating an example of selecting an object using a frame gesture.

[0053] Fig. 27 is a flow chart illustrating a process used in some implementations of the present technology for disambiguating between target and non-target objects intersected by a ray projection.

[0054] Fig.28 and Fig.29 is a conceptual diagram illustrating example hook disambiguation between selected and non-selected objects intersected by a ray projection.

[0055] Fig.30 is a flow chart illustrating a process used in some implementations of the present technology for disambiguating between target and non-target objects intersecting a conical or cylindrical projection using a "trombone" gesture.

[0056] Fig.31 and Fig.32 is a conceptual diagram illustrating an example trombone gesture for disambiguating between selected and non-selected objects that intersect a conic projection.

[0057] Fig.33 and Fig.34 is a conceptual diagram illustrating a second example trombone gesture for disambiguating between selected objects and unselected objects that intersect a cylindrical projection.

[0058] Fig.35 is a flow diagram illustrating a process used in some implementations of the present technology for two-handed interaction with an object using a menu attached to a dominant hand.

[0059] Fig.36 , Fig.37 and Fig.38 is a conceptual diagram illustrating an example of two-handed interaction with an object using a menu attached to the dominant hand.

[0060] Fig.39 is a flow chart illustrating a process for switching input modalities for global or local mode interactions used in some implementations of the present technology.

[0061] Fig.40A and Fig.40B is a conceptual diagram illustrating an example of switching input modalities for global or local mode interaction.

[0062] Fig.41 is a flow diagram illustrating a process used in some implementations of the present technology for interpreting gestures as interactions in global or local modes.

[0063] Fig.42A and Fig.42B is a conceptual diagram illustrating an example of interpreting gestures as interactions in a global or local mode.

[0064] The technology introduced herein may be better understood by referring to the following detailed description in conjunction with the drawings, wherein like reference numbers indicate identical or functionally similar elements. DETAILED DESCRIPTION

[0065] Embodiments for interacting with objects in an artificial reality environment are described herein. In an artificial reality environment, a user may wish to perform indirect interactions, where the cause and effect of the interaction are spatially separated. The present technology provides projections that a user can create to perform object interactions. For example, when a user wishes to interact with an object outside of her direct contact range, she can use a projection to select, move, or otherwise interact with a distant object. The present technology also includes object selection techniques for identifying and disambiguating between target objects, allowing the user to select objects that are close to and far from the user. Another aspect of the present technology includes techniques for interpreting various two-handed (two-handed) gestures to increase functionality and control when interacting with objects. The present technology also includes models for distinguishing global and local modes, for example, to provide different input modalities or interpretations of user gestures.

[0066] A projection is an indication of an area in two dimensions (e.g., a line or "ray") or three dimensions (e.g., a cylinder, sphere, cone, pyramid, etc.) that can specify an object that a user wishes to interact with. For example, a hand interaction system can identify one or more objects that intersect the projection as target objects. Due to the inaccuracy of tracking hand and body positions and postures, interpreting hand and body postures to correctly interact with objects outside the user's direct contact range can be difficult. These difficulties are compounded by the exponential effect of small hand movements on the projection position as the projection extends outward from the user. To increase the stability of the projection, the hand interaction system can control the projection position based on an origin and a control point. The hand interaction system can set a line extending from the origin through the control point as the center of the projection. In various implementations, the origin can be a tracked part of the user's body, such as a dominant eye, hip, or shoulder, and the control point can be a part of the user's hand, such as a fingertip, palm, wrist, or fist. In some implementations, the origin can be changed based on context, such as where the user directs her gaze or the angle of the projection centerline.

[0067] The hand interaction system can provide various projection types. A ray projection can be a single line extending from the user's hand to the artificial reality environment, for example. Ray projections can provide very precise interactions, but are less accurate, especially at a distance. A sphere projection can be a ray projection with a sphere at its end, and a cylindrical projection can be a cylinder extending from the user's hand into the artificial reality environment, for example. A cone or pyramid projection can be a cone or pyramid with a tip extending from the user's hand or eye, for example, and the projection increases exponentially in coverage area as the projection extends into the artificial reality environment. Because they cover a larger area than ray projections, projections such as cylinders, spheres, cones, and pyramids can be more easily used to correctly indicate objects that the user wishes to interact with, but are generally less accurate.

[0068] In some cases, the projection may intersect with more objects than the user wants to interact with. For example, the user may want to select only some tight clusters of objects. In other cases, the projection may not intersect with the object the user wants to interact with. For example, the projection can be configured not to pass through the object and the user may want to select an object that is blocked by another object. The hand interaction system may include various two-handed (two-handed) techniques for disambiguating or extending the projection coverage between selected objects. In some implementations, the hand interaction system may specify a "hook" at a point on the ray projection. The ray projection distance between the user's dominant hand and the hook can be based on another gesture, such as the distance between the user's dominant hand and the non-dominant hand or the distance between the user's two fingers. In other implementations, the hand interaction system can control the shape of the projection based on another gesture. For example, the hand interaction system can change the diameter of the bottom surface of the projection of a cylinder, sphere, or cone based on another gesture (such as the distance between the user's dominant hand and the non-dominant hand or the distance between the user's two hands).

[0069] In various implementations, the hand interaction system may also detect additional object interaction gestures, such as shear, shear click, lasso, or frame gestures. In some implementations, these gestures may use a ray projection configured with an origin at the user's dominant eye and a control point on her hand, so that it appears to the user as if she is "drawing" into the environment. A shear gesture is a situation where the user moves the projection to intersect with one or more objects. The hand interaction system may select each object that intersects with the projection. A shear click gesture is a situation where the user moves the projection to intersect with one or more operable controls (e.g., buttons). The hand interaction system may actuate each control that intersects with the projection. A lasso gesture is a situation where the hand interaction system identifies a lasso movement, such as by comparing the user's movement with a previously observed gesture (e.g., using a machine learning model) or by identifying that the user has made a closed shape gesture. The hand interaction system may identify a projection (e.g., a distorted cone) having a point extending from the user's eye and a distorted cone shape defined by a lasso gesture. The hand interaction system may select an object that intersects with the distorted cone. A framing gesture is when the hand interaction system identifies a framing input, such as by identifying that the user has defined the edges of a rectangle using the thumbs and index fingers of both hands, or by pulling apart opposing corners of a rectangle. The hand interaction system may identify a projection (e.g., a pyramid) using points extending from the user's eyes and the pyramid walls defined by the rectangle. The hand interaction system may select objects that intersect the pyramid.

[0070] The hand interaction system can also identify whether the user is trying to interact with the object in various operation modes (e.g., global mode or local mode). The global mode can be used to operate objects within the environment, such as selecting objects, controlling object placement in the virtual environment, etc. The local mode can be used to interact with the interface of one or more selected objects, such as pressing buttons on objects, interacting with menus or other controls associated with objects, etc. In various implementations, the user can explicitly select between global and local modes (e.g., using gestures or button presses) or the global / local mode selection can be implicit (e.g., based on the user's gaze). In some implementations, the input mode can be automatically changed based on which operation mode is activated. For example, in global mode, the input can be switched to a projection projection, such as a ray, cylinder, sphere, or cone projection. In local mode, the input can be switched to other modes such as by mapping the controls associated with the selected object to fingers on the user's hand, where the specific detected movement of the mapped fingers controls the actuation of the corresponding controls. In some implementations, the actions that can be taken for the object can be divided into executable in a specific operation mode. The hand interaction system can match gestures to actions in the action set defined for the currently activated mode. For example, in global mode, a gesture of dragging downward from the upper right corner of the virtual chessboard can be interpreted as an intention to rotate the chessboard. However, in local mode, the same gesture can be interpreted as moving the rook forward from the upper right square by an amount corresponding to the length of the gesture.

[0071] Embodiments of the disclosed technology may include or be implemented in conjunction with an artificial reality system. Artificial reality or hyperreality (XR) is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in an artificial reality and / or for use in an artificial reality (e.g., to perform activities therein). Artificial reality systems that provide artificial reality content may be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0072] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views an image of the real world after passing through a computing system. For example, a tablet with a camera on the back can capture an image of the real world, and then display the image on a screen on the opposite side of the tablet from the camera. The tablet can process and adjust or "enhance" the image as it passes through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially constitutes light reflected from objects in the real world. For example, an MR headset can be shaped as glasses with a pass-through display, which allows light from the real world to pass through a waveguide, which simultaneously emits light from a projector in the MR headset, thereby allowing the MR headset to present virtual objects mixed with real objects that the user can see. As used herein, "artificial reality", "hyperreality" or "XR" refers to any one of VR, AR, MR, or any combination or mixture thereof.

[0073] There are existing XR systems for selecting and interacting with objects in artificial reality environments. However, these XR systems are inaccurate, imprecise, and provide limited functionality. Existing XR systems typically require a separate controller device for input, and often fail to correctly interpret user gestures associated with objects in artificial reality environments. Existing XR systems are time-consuming and laborious for users to operate. The hand interaction system and process described in this article overcome these problems associated with conventional XR interaction technologies, and are expected to provide users with greater control over object interactions, provide more functionality, and be more natural and intuitive than interactions in existing XR systems. Although natural and intuitive, the XR systems and processes described in this article are rooted in computerized artificial reality systems rather than simulations of traditional object interactions. For example, existing object interaction technologies cannot describe projection positioning based on multiple body parts or gestures, let alone provide users with the ability to resize and customize such projections. In addition, existing XR systems do not provide effective selection and disambiguation techniques, nor do they provide options for switching input modalities or interpreting user input based on user intent patterns (e.g., global and local patterns).

[0074] Several implementations are discussed in more detail below with reference to the accompanying drawings. Figure 1is a block diagram illustrating an overview of a device on which some implementations of the disclosed technology may operate. The device may include hardware components of a computing system 100 that interpret gestures for interacting with objects in an artificial reality environment. In various implementations, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over a wired or wireless channel to distribute processing and share input data. In some implementations, the computing system 100 may include a standalone headset capable of providing a computer-created or enhanced experience to a user without the need for external processing or sensors. In other implementations, the computing system 100 may include multiple computing devices, such as a headset, and a core processing component (such as a console, mobile device, or server system), where some processing operations are performed on the headset and other processing operations are offloaded to the core processing component. The following is combined with Figure 2A and Figure 2B An example headset is described. In some implementations, location and environmental data may be collected solely by sensors incorporated into the headset device, while in other implementations, one or more non-headset computing devices may include sensor components that can track environmental or location data.

[0075] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be one or more processing units in a single process or distributed across multiple devices (e.g., distributed on two or more computing devices 101-103).

[0076] The computing system 100 may include one or more input devices 120 that provide input to the processor 110, informing them of actions. The actions may be mediated by a hardware controller that interprets the signals received from the input devices and communicates the information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, keyboard, touch screen, touchpad, wearable input device (e.g., tactile gloves, bracelets, rings, earrings, necklaces, watches, etc.), camera (or other light-based input device, such as an infrared sensor), microphone, or other user input device.

[0077] The processor 110 can be coupled to other hardware devices, for example, using an internal or external bus such as a PCI bus, a SCSI bus, or a wireless connection. The processor 110 can communicate with a hardware controller of the device, such as a display 130. The display 130 can be used to display text and graphics. In some implementations, such as when the input device is a touch screen or is equipped with an eye direction monitoring system, the display 130 includes an input device that is part of the display. In some implementations, the display is separated from the input device. Examples of display devices include: LCD display screens, LED display screens, projection displays, holographic displays, or augmented reality displays (such as heads-up display devices or head-mounted devices), etc. Other I / O devices 140 can also be coupled to the processor, such as a network chip or card, a video chip or card, an audio chip or card, a USB, FireWire or other external device, a camera, a printer, a speaker, a CD-ROM drive, a DVD drive, a disk drive, etc.

[0078] The computing system 100 may include a communication device capable of wireless or wired communication with other local computing devices or network nodes. The communication device may communicate with another device or server via a network using, for example, the TCP / IP protocol. The computing system 100 may utilize the communication device to distribute operations across multiple network devices.

[0079] Processor 110 may access memory 150, which may be contained on one of the computing devices of computing system 100 or may be distributed across multiple computing devices or other external devices of computing system 100. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include read-only memory and writable memory. For example, memory may include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory (such as flash memory, hard drive, floppy disk, CD, DVD, magnetic storage device, tape drive, etc.). Memory is not a propagating signal that is separated from the underlying hardware; memory is therefore non-temporary. Memory 150 may include program memory 160 that stores programs and software (such as operating system 162, hand interaction system 164, and other applications 166). Memory 150 may also include data storage 170, which may include, for example, projection descriptors, gesture identifiers (e.g., trained machine learning models) and / or gesture paths, mappings between gestures and actions, mappings between actions and user intent patterns, configuration data, settings, user options or preferences, etc., which may be provided to program storage 160 or any element of computing system 100.

[0080] Some implementations may operate with numerous other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above, and the like.

[0081] Figure 2A 2 is a wiring diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a strap 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. Position sensor 220, IMU 215, and computing unit 230 may be internal to HMD 200 and may not be visible to the user. In various implementations, IMU 215, position sensor 220, and locator 225 may track the movement and position of HMD 200 in three degrees of freedom (3DoF) or six degrees of freedom (6DoF) in the real world and in a virtual environment. For example, locator 225 may emit an infrared beam that creates a light spot on a real object around HMD 200. One or more cameras (not shown) integrated with HMD 200 may detect the light spot. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200 , and to identify the shapes and positions of real objects around the HMD 200 .

[0082] The electronic display 245 may be integrated with the front rigid body 205 and may provide image light to the user according to the instructions of the computing unit 230. In various embodiments, the electronic display 245 may be a single electronic display or multiple electronic displays (e.g., a display for each user's eye). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a display including one or more quantum dot light emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.

[0083] In some implementations, the HMD 200 may be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC may use the HMD 200 in conjunction with outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0084] In some implementations, the HMD 200 can communicate with one or more other external devices, such as a controller (not shown) that the user can hold with one or two hands. The controller can have its own IMU unit, position sensor, and / or can emit more light points. The HMD 200 or external sensors can track these controller light points. The computing unit 230 in the HMD 200 or the core processing component can use this tracking to monitor the user's hand position and movement in combination with the IMU and position output. The controller can also include various buttons that the user can actuate to provide input and interact with virtual objects. In various implementations, the HMD 200 can also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc. In some implementations, instead of or in addition to the controller, one or more cameras 200 included in or external to the HMD can monitor the position and posture of the user's hands to determine gestures and other hand and body movements.

[0085] Figure 2B 25 is a wiring diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other implementations, the mixed reality system 250 includes only a headset without an external computing device or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a pass-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., LASER, LED, etc.), a camera, an eye tracking sensor, a MEMS element, a networking component, etc.

[0086] The projector can be coupled to a pass-through display 258, for example, via optical elements, to display media to the user. The optical elements can include one or more waveguide components, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projector to the user's eyes. Image data can be transmitted from the core processing component 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes as output light via optical elements. The output light can be mixed with the light passing through the display 258, allowing the output light to present virtual objects that appear as if they exist in the real world.

[0087] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, in 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear stationary while HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0088] Figure 3 3 is a block diagram illustrating an overview of an environment 300 in which some implementations of the disclosed technology may operate. The environment 300 may include one or more client computing devices 305A-D, examples of which may include the computing system 100. In some implementations, some of the client computing devices (e.g., client computing device 305B) may be the HMD 200 or the HMD system 250. The client computing device 305 may operate in a networked environment using logical connections to one or more remote computers (such as server computing devices) through a network 330.

[0089] In some implementations, server 310 may be an edge server that receives client requests and coordinates fulfillment of these requests with other servers, such as servers 320A-C. Server computing devices 310 and 320 may include computing systems, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, the server computing devices may each be a distributed computing environment encompassing multiple computing devices located at the same or geographically different physical locations.

[0090] Client computing device 305 and server computing devices 310 and 320 can each act as a server or client of other server / client devices. Server 310 can be connected to database 315. Servers 320A-C can each be connected to corresponding databases 325A-C. As described above, each server 310 or 320 can correspond to a server group, and each of these servers can share a database or can have their own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can each be a distributed computing environment covering multiple computing devices, can be located in their corresponding servers, or can be located at the same or geographically different physical locations.

[0091] Network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 may be the Internet or some other public or private network. Client computing device 305 may be connected to network 330 via a network interface, such as via wired or wireless communication. Although the connections between server 310 and server 320 are shown as separate connections, these connections may be any kind of local, wide area, wired or wireless network, including network 330 or a separate public or private network.

[0092] Figure 4 4 is a block diagram illustrating a component 400, which, in some implementations, can be used in a system employing the disclosed technology. The component 400 can be included in one device of the computing system 100 or can be distributed across multiple devices of the computing system 100. The component 400 includes hardware 410, a mediator 420, and a dedicated component 430. As described above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.), and a storage memory 418. In various implementations, the storage memory 418 can be one or more of a local device, an interface with a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard disk drives or flash drives accessible by means of a system bus, or can be a cloud storage provider (such as in a database 315 or 325) or other network storage devices accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device, such as client computing device 305 , or in a server computing device, such as server computing devices 310 or 320 .

[0093] Mediator 420 may include components that mediate resources between hardware 410 and specialized components 430. For example, mediator 420 may include an operating system, a service, a driver, a basic input output system (BIOS), a controller circuit, or other hardware or software system.

[0094] The dedicated component 430 may include software and / or hardware configured to perform operations for recognizing gestures and interpreting their interactions with objects in an artificial reality environment. The dedicated component 430 may include, for example, a projection projection engine 434, an object selection engine 436, a two-handed interaction engine 438, a global and local mode engine 440, and components and APIs that can be used to provide a user interface, transmit data, and control dedicated components (such as interface 432). In some implementations, the component 400 can be implemented in a computing system distributed across multiple computing devices or can be an interface to a server-based application that executes one or more dedicated components 430.

[0095] The projection engine 434 can create various types of projections, such as ray projections (as described below with respect to Fig.12 and Fig.13 more details), spherical or cylindrical projections (as described below for Fig.14 and Fig.15 more details), or a cone projection (as described below for Fig.16 and Fig.17 In some implementations, the projection engine 434 can select a projection type based on the current context (as described below with respect to Fig.18 described in more detail).

[0096] The projection projection engine 434 can also control the projection position based on multiple tracked body parts, such as a portion of the hand as a control point and another point on the user's body as an origin, such as the eye, shoulder, or hip (as described below with respect to Figure 5-Figure 9 For example, the projection engine 434 may set the origin of the projection at or offset from the control point, and may determine the line connecting the origin and the control point as the center line of the projection. In some implementations, the projection engine 434 may move the origin of the projection according to an origin conversion factor such as the user's gaze direction, the angle of the projection, or the location of the target object (as described below with respect to Fig.10 and Figure 11A-11C described in more detail).

[0097] The object selection engine 436 can identify interactions that select multiple objects. In some implementations, the object selection engine 436 can identify a cut gesture in which a user draws a line that appears to the user to intersect multiple objects (as described below with respect to Fig.19 and Fig. 20 In some implementations, the object selection engine 436 can identify a shear click gesture in which a user draws a line that appears to the user to intersect multiple objects, and the system actuates those objects (as described below with respect to Fig.21 and Fig. 22 In some implementations, the object selection engine 436 can identify a lasso gesture that forms a shape around multiple objects selected by the system from the user's perspective (as described below with respect to Fig.23 and Figure 24A-24B In some implementations, the object selection engine 436 can identify a framing gesture from the user specifying two opposing corners of a rectangle, and the system can then select objects that are within the rectangle from the user's perspective (as described below with respect to Fig.25 and Figure 26A-26B described in more detail).

[0098] The two-hand interaction engine 438 can identify gestures performed by two hands (or different parts of the same hand) or gestures in which a second hand (or a different part of a first hand) modifies the result of a gesture performed by a first hand (or a first part of a first hand). In some implementations, the two-hand interaction engine 438 can identify a hook gesture, where a point on a ray projection is identified as a hook. The hook can define the object that the ray selects or otherwise interacts with. The position of the hook along the ray can be specified based on the distance between two user hands or the distance between two fingers on the same hand. The following combination Figure 27-Figure 29 In some implementations, the two-handed interaction engine 438 can identify a trombone gesture where the diameter of the cone, cylinder, or sphere projection is based on the distance between two users' hands or the distance between two fingers on the same hand. Figure 30-Figure 34 The trombone gesture is described in more detail. In some implementations, the two-handed interaction engine 438 can facilitate two-handed interaction, where a user selects an object using a gesture with a first hand, a menu is presented in association with the first hand for interacting with the selected object and the user can interact with the object using her second hand. Figure 35-Figure 38 To describe the menu attached to the selection hand in more detail.

[0099] The global and local mode engine 440 can identify whether the hand interaction system is in global mode or local mode, for example, based on explicit user commands or context, such as how long the user has been looking at the pointing object continuously. In some implementations, the global and local mode engine 440 can switch input modalities for global and local modes (as described below with respect to Fig.39 and Figure 40A-40BIn some implementations, the global and local mode engine 440 can interpret ambiguous inputs based on whether the hand interaction system is in global mode or local mode (as described below with respect to Fig.41 and Figure 42A-42B described in more detail).

[0100] Those skilled in the art will understand that the above Figure 1-Figure 4 The components shown in and in each of the flowcharts discussed below can be changed in a variety of ways. For example, the order of the logic can be rearranged, sub-steps can be performed in parallel, the illustrated logic can be omitted, other logic can be included, etc. In some implementations, one or more of the above components can perform one or more of the processes described below.

[0101] Projection using origin and control points

[0102] When using a controller to point and select objects in an artificial reality environment, it is easy to set and track a single point on the controller that can define a straight line. Existing controller-based selection techniques cannot be directly transferred to bare hands because there are inherent differences between the way people use tools and their own bodies. It is much more difficult to track a single point as the projection source when the user's hand is the input method. In addition, unlike a rigid controller surface, the user's hand tends to make many small movements, which causes the projection originating from the user's hand to jump around, and these movements are amplified as the projection extends away from the user. These natural hand tremors and tracking issues can interfere with the projection projection and cause inaccurate object selection.

[0103] In order to increase the accuracy of the projection projection based on hand tracking, the projection position can be limited using a primary control point (e.g., a part of the user's hand) and a secondary origin outside the user's bare hand to anchor the projection. The prediction based on the combination of these two positions can produce a more stable and accurate projection experience. Various origins can be used and can provide different typical input speeds, precision and ergonomics according to the size, distance, vertical position and density of the target object. In various implementations, the origin can include the user's dominant eye, a point between the user's eyes, another point on the user's head, the user's hips, the user's shoulders, or contextual variable points (e.g., between the user's hips and shoulders). Each of the various origins can be paired with various control points, such as the user's fingertips, the user's palms, the user's wrists or the user's fist center when making a specific gesture. When projecting a projection, the projection can be projected outward from the user, starting from the control point and directed to be centered on a straight line passing through the origin and the control point.

[0104] Figure 5is a flow chart illustrating a process 500 for projecting a projection using an origin and a control point used in some implementations of the present technology. At block 502, the process 500 may track a portion of a hand as a control point. For example, the control point may be a fingertip of a user, a palm of a user, a wrist of a user, or the center of a fist of a user. In some implementations, the control point may be identified in response to a user making a particular gesture, such as forming her fingers into a "pinch". In some cases, the control point may be offset from the tracked portion of the user. For example, the control point may be an offset from the palm or wrist of the user. That is, when the control point moves when forming a gesture, this helps eliminate the jitter that occurs when using the fingertip. In contrast, when such a gesture is formed using a finger, the point 6 inches in front of the wrist moves less. In some implementations, the control point may be based on other body parts, such as points on a portion of the user's head or foot.

[0105] At box 504, process 500 can track a second body part as an origin. In various implementations, the origin can be based on the location of the user's eyes, shoulders, hips, etc. In some implementations, the eye, shoulder, or hip used can be the eye, shoulder, or hip on the user's dominant side. For example, the user's dominant eye can be determined based on which hand the user tends to use most, the user's description of which hand or eye is dominant (e.g., during an administration process), observations of how the user positions herself relative to past stimuli, visual tests provided to the user, etc. In some implementations, the origin can move depending on the environment (such as where the user's gaze is directed, the angle of the projection the user is creating, or the choice of a machine learning model). The following is combined with Fig.10 and FIG. 11 discuss adjusting the origin in more detail.

[0106] At blocks 502 and 504, the origin and control points may be tracked by analyzing images from, for example, a camera integrated with an XR headset or external sensors. For example, a machine learning model may be used to analyze images from such a camera and generate 3D position data for a model of a user's hands or other various body parts. In some implementations, additional or alternative methods may be applied, such as by using position or IMU tracking sensors in a headset, bracelet, glove, or other wearable device.

[0107] At block 506, process 500 may determine a projection orientation centered on a line through A) the origin determined at block 502 and B) the control point determined at block 504. As described below, the projection may be one of various types, such as a ray, a sphere, a cylinder, a cone, a pyramid, etc. The projection may extend outward from the user from the control point or be offset from the control point.

[0108] At block 508, process 500 may perform an action related to a real or virtual object based on one or more positions of the projection. For example, the action may include any interaction, such as selecting an object, moving or rotating an object, interacting with a control (e.g., pressing a button, pulling or pushing a control), etc. In some cases, the action may be based on multiple positions of the projection, such as corresponding movement of the projection when the user moves a control point and / or an origin. For example, when the user moves her palm as a control point with her shoulder as the corresponding origin, a ray projection cast outward from the user's palm may intersect multiple objects, each of which may be selected.

[0109] Figure 6 is a conceptual diagram illustrating an example 600 of a ray projection 606 using a shoulder origin 602 and a palm control point 604 . Figure 7 is a conceptual diagram illustrating an example 700 of ray projection 706 using a dominant eye origin 702 and a fingertip control point 704 . Figure 8 is a conceptual diagram illustrating an example 800 of ray projection 806 using a hip origin 802 and a fist control point 804 . Fig. 9 is a conceptual diagram illustrating an example 900 of a ray projection 906 utilizing a shoulder origin 802 and a wrist control point 804. In each example 600, 700, 800, or 900, as the user moves her tracked origin (shoulder, eye, or hip) and / or tracked control point (wrist, palm, or fingertip), a corresponding movement of the ray projection 606, 706, 806, or 906 will occur.

[0110] Different origins may work better in different contexts. For example, a higher origin may more easily access lower targets, while a lower origin may more easily access higher targets. In some implementations, the hand interaction system may identify the condition and move the origin accordingly. In various implementations, the origin may snap between different origins or may move at a given speed on a path between origins.

[0111] Fig.10 1 is a flow chart illustrating a process 1000 for adjusting a projection origin used in some implementations of the present technology. At block 1002, the process 1000 may identify a gaze direction and / or other origin conversion factors. In various implementations, the origin conversion factor may be one or more of: a user's gaze, a horizontal angle of the current projection, a position of a target object relative to the current origin, etc. For example, the process 1000 may determine an angle of a centerline of the current projection. As another example, the process 1000 may determine that the current projection selects objects and those objects are above or below a threshold plane.

[0112] At box 1004, process 1000 can compare the origin conversion factor determined at box 1002 with a threshold to determine whether the origin should be set higher or lower. For example, when the user's gaze is above a threshold angle from their body, a high interaction mode can be selected, and when the user's gaze is below a threshold angle from their body, a low interaction mode can be selected. As another example, process 1000 can determine that the angle of the centerline of the current projection is greater than a threshold angle (e.g., pointing upward) and a high interaction mode can be selected or can determine that the angle of the centerline of the current projection is less than a threshold angle (e.g., pointing downward) and a low interaction mode can be selected. As yet another example, process 1000 can determine that the current projection selects an object above a threshold plane and a high interaction mode can be selected, or can determine that the current projection selects an object below a threshold plane and a low interaction mode can be selected.

[0113] At box 1006, process 1000 can select a high interaction mode, where the origin is transformed toward a low position such as the user's hips. In some implementations, the high interaction mode can make it easier to interact with objects above a specific plane. In some implementations, the transformation can be a snap of the origin to a low position. In some implementations, there is a specified path between two points (a high point and a low point) on the user's body (such as the user's shoulders and hips). Process 1000 can start at box 1006 and move the origin toward the low end of the path at a set speed unless the origin is already at the lowest point on the path.

[0114] At box 1008, process 1000 may select a low interaction mode, where the origin is transformed toward a high position (such as the user's eyes or shoulders). In some implementations, the low interaction mode can make it easier to interact with objects below a certain plane. In some implementations, the transformation can be a snap of the origin to a high position. In some implementations, there is a specified path between two points (a high point and a low point) on the user's body (such as the user's eyes and hips). At box 1008, unless the origin is already at the highest point on the path, process 1000 can begin moving the origin toward the high end of the path at a set speed.

[0115] Process 1000 may repeat, returning to block 1002, as a new origin conversion factor is determined.

[0116] Fig.11A , Fig. 11B and Fig. 11C1 is a conceptual diagram illustrating an example of adjusting an origin 1004 of a ray projection also controlled by a control point 1102, where the origin 1104 moves on a path 1106 between the hips and shoulders. In the example 1100, the movement of the origin 1104 is based on a user's eye gaze 1108. The point along the path 1106 for which the origin has been set is based on a linear relationship of a portion of the path to an angle formed by the user's eye gaze and a plane 1110 perpendicular to the ground.

[0117] exist Fig.11A In , the user's eye gaze 1108 is perpendicular to the plane 1110, and the origin 1104 is set so that the ray between the origin 1104 and the control point 1102 is perpendicular to the path 1106. Fig. 11B , the user's eye gaze 1108 has traveled upward beyond the maximum upward angle, so the origin 1104 is set at the lowest point on the path 1106. Fig. 11C , the user's eye gaze 1108 has traveled downward below the minimum downward angle, so the origin 1104 is set at the highest point on the path 1106. In example 1100, the transitions between these points occur at a set speed, moving the origin 1104 smoothly along the path 1106.

[0118] Interacting with distant virtual objects of the projected type

[0119] Existing XR systems have many difficulties when interacting with objects, especially when those objects are outside the user's direct reach. For example, many existing hand tracking systems are inaccurate, causing the virtual model of the user's hand to either shake or lag behind the actual position of the user's hand. In addition, the user's hand is prone to tremors and other small movements, which translate into large changes in the projection position when the projection is projected over a large distance. Another difficulty, known as the Heisenberg effect, is the movement that often occurs in the pointing device (hand, mouse, stylus, etc.) when the user uses the pointing device to perform input (make gestures, click buttons, etc.). In many cases involving long-distance interaction, these difficulties can be overcome by using the appropriate projection projection type. A non-exhaustive list of projection types includes ray projection, spherical projection, cylindrical projection, conic projection, and pyramid projection.

[0120] For purposes of this disclosure, a ray projection is a single line extending from a control point or a line offset from a control point. A ray projection provides little ambiguity as to what the user is targeting. However, small target sizes can make the interaction brittle, and users can find it difficult to point to a single object, especially a small object that is outside the user's reach.

[0121] For purposes of this disclosure, a spherical projection is a ray projection utilizing a sphere attached to an end. In some implementations, the user can control the length of the ray and / or the diameter of the sphere. In other implementations, the diameter of the sphere and / or the length of the ray can be fixed. In other implementations, the length of the ray can be set so that the sphere intersects the object(s) closest to the user along the ray. A cylindrical projection is similar to a spherical projection, except that the sphere is translated along the entire length of the ray, forming a cylinder extending from the projection control point. Compared to ray projection, sphere and cylindrical projections tend to improve selection accuracy for interacting with objects that are just outside the user's reach (e.g., 1-3 feet).

[0122] For purposes of this disclosure, a conic projection is a cone that extends from a projection control point and increases in diameter as the cone extends away from the user. In some implementations, the degree to which the diameter changes as the conic projection extends away from the user can be fixed. In other implementations, as described below with respect to Figure 30-Figure 34 As discussed, the user can control the diameter of the cone. A pyramidal projection can be similar to a conic projection, but can use a rectangular base instead of a circular base.

[0123] In some cases, when at least a portion of the projection is three-dimensional, such as a sphere, cylinder, cone, or pyramid, the (multiple) target objects of the projection can be objects that the projection partially intersects, objects that the projection completely covers, or objects that the projection covers at least a threshold amount (e.g., 30%, 50%, or 80%). In some implementations where the projection target can be multiple objects, secondary selection techniques can be used to distinguish which objects the user intends to target. For example, a single object that intersects with the projection can be mapped to a finger on the user's hand, and movement of the mapped finger (e.g., tapping the thumb with the finger) can cause the corresponding mapped object to be added or removed from a set of target objects. As another example, another input modality such as user gaze can distinguish which intersecting objects become target objects.

[0124] Fig.12 1 is a flow chart illustrating a process 1200 for object interaction using ray casting used in some implementations of the present technology. At block 1202, process 1200 may determine control points and projection directions for ray casting based on tracked positions of one or more body parts. In some implementations, as described above with respect to Figures 5 to 11CAs discussed, the control points may be palms, fingertips, fists, wrists, etc., and the projection direction may be along a line passing through the control points (such as shoulders, eyes, or hips) and the origin. In other implementations, the control points may be based on other tracked body parts, such as the user's eyes, head, or chest. For example, the control points may be the estimated location of the center of the user's pupil, and the origin may be the estimated location of the center of the user's retina. In some cases, the system may define a plane corresponding to the control points, and may determine a projection direction perpendicular to the plane, rather than using the origin to define the direction. For example, where the control points are located on the head or torso, the system may identify a coronal plane corresponding to the head or torso, and may identify the projection direction as perpendicular to the coronal plane.

[0125] At block 1204, process 1200 may generate a ray projection (i.e., a straight line) from the control point along the projection direction. In some cases, a graphical representation of the ray projection (the entire line or just the point where the ray hits the object) may be displayed in the artificial reality environment, while in other cases the ray projection is tracked by the XR system without displaying the ray projection. In various implementations, the ray projection may extend from the control point until it intersects a first (real or virtual) object, or the ray projection may extend through multiple objects. In some implementations, the direction of the ray projection may be adjusted to "snap" to an object that it is close to intersecting, or the ray projection may be bent to a threshold amount to maintain intersection with such an object.

[0126] At block 1206, process 1200 may determine an intersection between the ray projection and one or more target objects. In some implementations, process 1200 may determine the intersection when the user makes a particular gesture, such as tapping the thumb with the index finger. In some implementations, the target object may be the first object that the ray projection intersects, excluding other objects that are occluded by the first object. In other implementations, the ray may extend through multiple objects, and the target object may be all objects that the ray projection intersects. In other implementations, the objects that the ray intersects may be identified as target objects based on other body part positions or postures, such as the user's gaze or a gesture involving a finger that maps to the intersected object.

[0127] At block 1208, process 1200 may enable interaction with one or more target objects determined at block 1206. For example, process 1200 may select target objects, cause them to move or rotate, or perform other interactions.

[0128] Fig.13is a conceptual diagram illustrating an example 1300 of a ray projection 1302. In example 1300, a user's hand 1306 forms a gesture by connecting her thumb and middle finger, indicating the start of the projection. In example 1300, a control point 1304 is an offset from the user's middle finger. Ray projection 1302 extends from control point 1304 to a target object 1308. In example 1300, the user closes her index finger and her thumb (not shown) and reopens it, making a "click" gesture, causing target object 1308 to be selected.

[0129] Fig.14 1 is a flow chart illustrating a process 1400 for object interaction using cylindrical projection used in some implementations of the present technology. At block 1402, process 1400 may determine control points and a projection direction of a cylindrical projection based on the tracked positions of one or more body parts. In some implementations, as described above with respect to Figures 5 to 11C As discussed, the control points may be palms, fingertips, fists, wrists, etc., and the projection direction may be along a line passing through the control points and an origin, such as a shoulder, eye, or hip. In other implementations, the control points may be based on other tracked body parts, such as the user's eyes, head, or chest. For example, the control points may be an estimated location of the center of the user's pupil, and the origin may be an estimated location of the center of the user's retina. In some cases, the system may define a plane corresponding to the control points, and the projection direction may be determined to be perpendicular to the plane, rather than using the origin to define the direction. For example, where the control points are located on the head or torso, the system may identify a coronal plane corresponding to the head or torso, and the projection direction may be identified as perpendicular to the coronal plane.

[0130] At block 1404, process 1400 may generate a cylindrical projection from the control point along the projection direction. In some implementations, instead of a cylinder, the projection may be a spherical projection that includes a sphere located at the end of the ray projection. In some cases, a graphical representation of the cylindrical (or spherical) projection may be displayed in an artificial reality environment (e.g., showing the entire projection or the location where it intersects with an object), while in other cases, the cylindrical (or spherical) projection is tracked by the XR system without displaying the projection. In various implementations, the cylindrical projection may extend from the control point until it intersects the first (real or virtual) object, or the cylindrical projection may extend to at least partially intersect with multiple objects. In some implementations, the ray of the spherical projection may extend from the control point until the sphere intersects the first (real or virtual) object, or the ray of the spherical projection may extend a fixed distance.

[0131] At box 1406, process 1400 can determine the intersection between the cylinder (or sphere) projection and one or more target objects. In some implementations, process 1400 can determine the intersection when the user makes a specific gesture (such as tapping the thumb with the index finger). In some implementations, the target object can be the first object that the cylinder (or sphere) projection at least partially intersects or completely encompasses. In other implementations, the target objects can be all objects that the cylinder (or sphere) projection at least partially intersects or completely encompasses. In other implementations, objects that the cylinder (or sphere) intersects can be identified as target objects based on other body part positions or postures, such as the user's gaze or gestures involving fingers mapped to the intersecting objects. In some implementations, as described below with respect to Figure 30-Figure 34 As discussed, the diameter of the cylinder (or sphere) may be controlled by other gestures, such as the distance between the user's hands or the distance between the tip of the user's index finger and thumb.

[0132] At block 1408, process 1400 may cause interaction with one or more target objects determined at block 1406. For example, process 1400 may select target objects, cause them to move or rotate, or perform other interactions.

[0133] Fig.15 1500 is a conceptual diagram illustrating an example 1500 of a cylindrical projection 1502. In example 1500, a user's hand 1506 has formed a gesture by connecting her thumb and middle finger, indicating that the projection begins. In example 1500, a control point 1504 is offset from the user's middle finger. Cylindrical projection 1502 extends from control point 1504 to target object 1508, at least partially intersecting each of target objects 1508A, 1508B, 1508C, 1508D, and 1508E. In example 1500, the user has made a gesture by rotating her hand 1506 clockwise (not shown) so that target objects 1508A-E will be selected.

[0134] Fig.16 1 is a flow chart illustrating a process 1600 for object interaction using conic projection used in some implementations of the present technology. At block 1602, process 1600 may determine control points and a projection direction for projection based on the tracked positions of one or more body parts. In some implementations, as described above with respect to Figures 5 to 11CAs discussed, the control points may be palms, fingertips, fists, wrists, etc., and the projection direction may be along a line passing through the control points (such as shoulders, eyes, or hips) and the origin. In other implementations, the control points may be based on other tracked body parts, such as the user's eyes, head, or chest. For example, the control points may be the estimated location of the center of the user's pupil, and the origin may be the estimated location of the center of the user's retina. In some cases, the system may define a plane corresponding to the control points, and the projection direction may be determined to be perpendicular to the plane, rather than using the origin to define the direction. For example, where the control points are located on the head or torso, the system may identify a coronal plane corresponding to the head or torso, and the projection direction may be identified as perpendicular to the coronal plane.

[0135] At box 1604, process 1600 can generate a conic projection, the conic projection having a cone tip at the control point and the cone extending away from the user along the projection direction. In some implementations, the projection can be a pyramidal projection with a pyramid tip at the control point and the pyramid extending along the projection direction, rather than a cone. In some cases, a graphical representation of the conic (or pyramidal) projection (e.g., a representation of the entire cone or pyramid or a representation of where the projection intersects the object) can be displayed in the artificial reality environment, while in other cases, the conic (or pyramidal) projection is tracked by the XR system, but the projection is not displayed. In various implementations, the conic (or pyramidal) projection can extend from the control point until it intersects the first (real or virtual) object, or the conic (or pyramidal) projection can extend to at least partially intersect multiple objects.

[0136] At box 1606, process 1600 can determine the intersection between the cone (or pyramid) projection and one or more target objects. In some implementations, process 1600 can determine the intersection when the user makes a specific gesture (such as tapping the thumb with the index finger). In some implementations, the target object can be the first object that the cone (or pyramid) projection at least partially intersects or completely covers. In other implementations, the target objects can be all objects that the cone (or pyramid) projection at least partially intersects or completely covers. In other implementations, the objects that the cone (or pyramid) intersects can be identified as target objects based on other body part positions or postures (such as the user's gaze or a gesture involving a finger mapped to the intersecting object). In some implementations, as described below with respect to Figure 30-Figure 34 As discussed, the diameter of the cone may be controlled by other gestures, such as the distance between the user's hands or the distance between the tip of the user's index finger and thumb.

[0137] At block 1608, process 1600 may enable interaction with one or more target objects determined at block 1606. For example, process 1600 may select target objects, cause them to move or rotate, or perform other interactions.

[0138] Fig.17 1700 is a conceptual diagram illustrating an example 1700 of two conic projections 1702 and 1752. A user's hands 1706 and 1756 have formed a gesture by connecting the thumb and middle finger, indicating the start of the projection. In example 1700, control points 1704 and 1754 are offset from the user's middle finger. Conic projection 1702 extends from control point 1704, increasing in diameter until it intersects target object 1708, at least partially intersecting each of target objects 1708A, 1708B, 1708C, 1708D, and 1708E. In example 1700, an object is selected once the projection intersects the object, so target objects 1708A-E are selected by conic projection 1702. Conic projection 1752 extends from control point 1754, also increasing in diameter. However, target object 1758 is closer to hand 1756 than target object 1708 is to hand 1706, so conic projection 1752 intersects only target object 1758. Since in example 1700 an object is selected whenever a projection intersects the object, target object 1758 is selected by conic projection 1752.

[0139] In some implementations, the hand interaction system can automatically select the type of projection to cast based on the current environment. For example, the system can use ray projection when obvious target objects are tightly grouped (e.g., when another type of projection would encompass more than a threshold number of objects, or when objects within a threshold distance from the projection center are within a threshold distance of each other). As another example, the hand interaction system can switch to a sphere or cone projection when the nearest object that a ray projection would intersect is greater than a threshold distance. Fig.18 is a flow diagram illustrating a process 1800 for context-based selection of a projection type used in some implementations of the present technology.

[0140] At block 1802, process 1800 may identify an interaction plane for the user's gaze. The interaction plane is the plane in the artificial reality environment of the object (or center between multiple objects) with which the user is currently interacting. For example, the interaction plane may be determined based on the plane to which the user's tracked gaze is directed. As another example, if the user selects a virtual object, the interaction plane may be centered on the object. At block 1804, process 1800 may calculate the distance between the user and the interaction plane identified at block 1802 in the virtual space of the artificial reality environment.

[0141] At block 1806, process 1800 may identify a surface type of the current target object. In various implementations, the surface type may be classified based on whether the surface of the target object is flat, the size of the target object, whether the target object includes a control, what type of control the target object includes, what type of interaction the user or other users have previously performed with respect to the target object, and the like. In other implementations, the type may be a setting assigned to the target object, for example, by the creator of the object. In some implementations, process 1800 may begin at block 1806 instead of block 1802. In various implementations, process 1800 may include or skip block 1806.

[0142] At box 1808, process 1800 can select a projection type based on the distance calculated at box 1804 and / or the surface type identified at box 1806. In some implementations, the selection can be based on a mapping of distance ranges to projection types. For example, when the interaction plane is within three feet of the user, a ray projection can be used; when the interaction plane is between three and seven feet from the user, a cylindrical projection can be used; and when the interaction plane is more than 7 feet away from the user, a conical projection can be used. In other implementations, the mapping can be surface type to projection type. For example, where the surface type indicates a flat surface with multiple buttons and controls, a ray projection can be used, otherwise a cylindrical projection can be used. In further implementations, mapping can be used for pairs of distance ranges and surface types to projection types. For example, when the interaction plane is within three feet of the user and the surface type indicates a plane with multiple buttons and controls, a ray projection may be used; when the interaction plane is within three feet of the user and the surface type indicates no controls, a cylindrical projection may be used; when the interaction plane is between 3 and 7 feet away from the user and the surface type indicates that the object is larger than the 7-inch threshold, a cylindrical projection may be used, but if the surface type indicates that the object is smaller than the 7-inch threshold, a ray projection may be used; and if the interaction plane is more than 7 feet away from the user, a conic projection may be used regardless of the surface type.

[0143] Multiple object selection

[0144] As the user operates in the artificial reality environment, she can select various objects for further interaction or specify that the XR system perform operations on the objects. Although some existing XR systems have techniques for selecting objects (such as by performing finger gestures on each object), these techniques have many disadvantages. For example, when a user wants to select a group of objects, selecting each object individually is time-consuming, frustrating, and requires additional processing power. When selecting distant or small objects, the precision and accuracy of existing techniques are low, making it difficult for the XR system to correctly interpret which objects the user intends to select. In addition, when the user intends to select objects that are occluded from each other from the perspective of the XR system, the XR system does not know how to disambiguate the objects that the user intends to select. In addition, for existing XR systems, it is impossible to focus on refining the user's intentions within a group of objects when the object is located in a dense area of ​​other objects.

[0145] Techniques for overcoming these shortcomings in existing systems include systems and processes for identifying and interpreting "cut," "lasso," and "frame" gestures for selecting multiple objects, "cut click" gestures for activating controls, and various two-handed "two-handed" gestures to disambiguate between potential target objects.

[0146] Fig.19 1 is a flow diagram illustrating a process 1900 for selecting an object using a cut gesture used in some implementations of the present technology. At block 1902, the process 1900 may identify an action corresponding to initiating object selection. In some implementations, the action may be a specific gesture, such as a "pinch" between a user's thumb and index or middle finger. In some implementations, the action may also include identifying a gating condition, such as a hand tracking system providing one or more of the following indications: the user's hand is within the field of view, the confidence value of the current hand tracking position is above a threshold, or the hand is not at rest.

[0147] At box 1904, process 1900 can continuously determine control points and projection directions based on the tracked positions of one or more body parts. The body part(s) can be, for example, a portion of a hand (e.g., fingertips, wrists, palms, fists, etc.), an eye, a head, chest or torso, a leg or foot, etc. The control point can be at one of the body parts or offset from it. In some implementations, the projection direction can be along a line connecting one of the user's eyes to the control point. For example, process 1900 can record the intersection of an object with a line extending outward from a control point and intersecting both the control point and the user's dominant eye. The effect of this example appears to the user as if they are "drawing" in space by moving the control point. In other implementations, the control point can be moved using methods described above with respect to the control point. Figures 5 to 11CIn these implementations, the projection direction can be along a line passing through a control point (such as a shoulder, eye, or hip) and an origin. In some implementations, the drawing can be done by the user's gaze, where the control point can be an estimated location of the center of the user's pupil and the origin can be an estimated location of the center of the user's retina.

[0148] At block 1906, process 1900 may display a path based on the projection direction. The path may be a record of the locations indicated by the control points and the projection direction. In some implementations, the path may appear to the user as if she is drawing the path in the artificial reality environment by moving the control points. In other implementations, the path may appear to the user as if she is creating the path at the end of the projection. In some implementations, process 1900 does not display the path and skips block 1906.

[0149] At block 1908, as process 1900 continues to monitor the control points and the projection direction, process 1900 can record any objects that the control points intersect from the user's perspective. In some implementations, process 1900 can record an indication of any objects that the projection projection intersects.

[0150] At block 1910, in response to one or more triggers, such as the user releasing the gesture or the speed of the user's hand movement falling below a threshold, process 1900 may select to record the indicated object at block 1908. For example, process 1900 may begin at block 1902 by recognizing that the user has formed a gesture of putting all five fingertips together. At blocks 1904, 1906, and 1908, process 1900 may monitor the point directly in front of the user's fingertips as a control point, determine the direction based on the tracked position of the user's dominant eye, and record each object that the control point intersects from the user's perspective. Process 1900 may continue until the user releases the five-finger gesture, at which time process 1900 may cause any recorded object to be selected.

[0151] Fig. 202000 . The hand interaction system then begins tracking a control point 2004 (a point offset in front of the gesture 2002) and a projection direction 2006. In example 2000, the projection direction 2006 is formed by determining a line connecting the user's eyes (not shown) and the control point 2004. Therefore, from the user's perspective, the control point 2004 occludes the drawing point 2008. When the user first forms the gesture 2002, the drawing point 2008 is located at the point 2010. The user then moves her hand, causing the control point 2004 to move, which in turn causes the drawing point 2008 to advance from the point 2010 along the path 2012. As a result, path 2012 intersects objects 2014 and 2016, but does not intersect object 2018, causing the hand interaction system to record indications of objects 2014 and 2016. In example 2000, the hand interaction system displays path 2012 in the artificial reality environment for the user to view. When the user releases gesture 2002, the hand interaction system selects objects 2014 and 2016 based on the recorded indications. As shown in the figures herein, an object is illustrated as "selected" when its wire frame has a thicker line than other objects in the figure.

[0152] Fig.21 21 is a flow diagram illustrating a process 2100 for actuating an object using a cut click gesture used in some implementations of the present technology. At block 2102, the process 2100 may identify an action corresponding to initiating object selection. In some implementations, the action may be a specific gesture, such as a "pinch" between a user's thumb and index or middle finger. In some implementations, the action may also include identifying a gating condition, such as a hand tracking system providing one or more indications that the user's hand is in view, a confidence value for the current hand tracking position is above a threshold, or the hand is not at rest.

[0153] At box 2104, process 2100 can continuously determine control points and projection directions based on the tracked positions of one or more body parts. The (multiple) body parts can be, for example, a part of a hand (e.g., a fingertip, wrist, palm, fist, etc.), an eye, a head, chest or torso, a leg or foot, etc. The control point can be at or offset from one of the body parts. In some implementations, the projection direction can be along a line connecting one of the user's eyes to the control point. For example, process 2100 can record the intersection between an object and a line extending outward from the control point, which falls along a line that intersects both the control point and the user's dominant eye. The effect of this example appears to the user as if they are "drawing" in space by moving the control point. In other implementations, the control point can be moved using the method described above with respect to the control point. Figures 5 to 11CIn these implementations, the projection direction can be along a line passing through a control point and an origin (such as a shoulder, an eye, or a hip). In some implementations, the drawing can be done by the user's gaze, where the control point can be the estimated location of the user's pupil center and the origin can be the estimated location of the user's retina center.

[0154] At block 2106, process 2100 may display a path based on the projection direction. The path may be a record of the locations indicated by the control points and the projection direction. In some implementations, the path may appear to the user as if she is drawing the path in the artificial reality environment by moving the control points. In other implementations, the path may appear to the user as if she is creating the path at the end of the projection. In some implementations, process 2100 does not display the path and skips block 2106.

[0155] At block 2108, as the process 2100 continuously monitors the control point and the projection direction, the process 2100 can record any objects that the control point intersects from the user's perspective based on the projection direction. In some implementations that use projections, the process 2100 can record an indication of any objects that the projection intersects. In some implementations, instead of or in addition to recording intersections, each time the process 2100 identifies an intersection with an actionable object (such as a button or other control), the process 2100 can actuate the object.

[0156] At box 2110, in response to one or more triggers, such as the user releasing the gesture or the speed of the user's hand movement falling below a threshold, process 2100 can actuate the object indicated at box 2108. For example, process 2100 can start at box 2102 by recognizing that the user forms a gesture of putting all five fingertips together. At boxes 2104, 2106 and 2108, process 2100 can monitor the point directly in front of the user's fingertips as a control point, determine the direction based on the tracking position of the user's dominant eye, and record each operable object that the control point intersects from the user's perspective. Process 2100 can continue until the user releases the five-finger gesture, at which time process 2100 can cause any recorded object to be actuated. In some implementations, processes 1900 and 2100 can be performed together, wherein the movement of the control point can cause the operable intersecting object to be actuated and other intersecting objects to be selected. In some implementations, the operable object is actuated, and both the operable object and the inoperable object are selected.

[0157] Fig. 222200 . The hand interaction system then begins tracking a control point 2204 (a point offset in front of the gesture 2202) and a projection direction 2206. In the example 2200 , the projection direction 2206 is formed by determining a line connecting the user's eyes (not shown) and the control point 2204. Therefore, from the user's perspective, the control point 2204 occludes the drawing point 2208. When the user first forms the gesture 2202, the drawing point 2208 is located at the point 2210. The user then moves her hand, causing the control point 2204 to move, which in turn causes the drawing point 2208 to advance along the path 2212 from the point 2210. As a result, the path 2212 intersects the button object 2214, causing the hand interaction system to actuate the button object 2214. In example 2200 , the hand interaction system displays a path 2212 in an artificial reality environment.

[0158] Fig.23 23 is a flow diagram illustrating a process 2300 for selecting an object using a lasso gesture used in some implementations of the present technology. At block 2302, the process 2300 may identify an action corresponding to initiating object selection. In some implementations, the action may be a particular gesture, such as a "pinch" between a user's thumb and index or middle finger. In some implementations, the action may also include identifying a gating condition, such as a hand tracking system providing one or more indications that the user's hand is in view, a confidence value for the current hand tracking position is above a threshold, or the hand is not at rest.

[0159] At box 2304, process 2300 can continuously determine control points and projection directions based on the tracked positions of one or more body parts. (Multiple) body parts can be, for example, a part of a hand (e.g., a fingertip, a wrist, a palm, a fist, etc.), a part of an eye, a head, a chest or torso, a leg or a foot, etc. The control point can be at or offset from one of the body parts. In some implementations, the projection direction can be along a line connecting one of the user's eyes to the control point. For example, process 2300 can determine whether an object falls into a distorted cone formed by the shape of the tip of the distorted cone at the user's eye and the side of the distorted cone based on the user moving the control point. As used herein, a "distorted cone" is a cone with a bottom that is not necessarily circular or elliptical. In some implementations, the distorted cone shape can be created by the user moving her gaze, where the gaze is determined by a line formed from an estimated position of the center of the user's pupil and an estimated position of the center of the user's retina.

[0160] At block 2306, process 2300 may display a path based on the projection direction. The path may be a record of the locations indicated by the control points and the projection direction. In some implementations, the path may appear to the user as if she were drawing the path in the artificial reality environment by moving the control points. In some implementations, process 2300 does not display the path and skips block 2306.

[0161] At box 2308, as process 2300 continuously monitors the control points and the projection direction, process 2300 can determine that the user's movement of the control points is intended as a lasso input. In some implementations, process 2300 makes this determination by providing a path indication to a machine learning model trained to identify lasso input. In some implementations, process 2300 makes this determination by flattening the path into a 2D plane from the user's perspective and determining that the path intersects itself. The lasso input can define a closed outline of the shape formed by the lasso input. If the lasso input does not form a closed shape, process 2300 can close it with a straight line or curve that matches the lasso input.

[0162] At block 2310, in response to one or more triggers, such as identifying a lasso input, a user releasing a gesture, or a speed of the user's hand movement falling below a threshold, process 2300 can identify a distortion cone having a tip of the distortion cone formed at one of the user's eyes and a distortion cone shape formed by the closed shape defined by the lasso input. For example, the distortion cone can be formed by every possible line extending from the user's eyes and passing through a point on the closed shape.

[0163] At block 2312, process 2300 may select an object that falls within the distortion cone identified at block 2310. In some implementations, process 2300 may also select an object that at least partially intersects the distortion cone or has a threshold amount of its volume within the distortion cone (e.g., at least 30%, 50%, or 75%). For example, process 2300 may begin at block 2302 by recognizing that a user forms a gesture of putting all five fingertips together. At blocks 2304-2310, process 2300 may monitor points directly in front of the user's fingertips as control points, determine directions based on the position of the tracked user's dominant eye, track a path defined by the control points from the user's perspective, and use a machine learning model to determine that the path forms a lasso input. Process 2300 may then identify a distortion cone defined by a path of lines starting from the user's eyes and moved to also extend through each point on the closed portion of the lasso input. Process 2300 may then cause any object that at least partially falls within the distortion cone to be selected.

[0164] In some implementations, multiple selection processes described herein may be combined into a single gesture. For example, processes 1900 and 2300 may be combined. In one example, process 1900 may start at box 1902, while continuously performing analysis from box 2308 to determine whether the input is a lasso gesture. If a lasso gesture is identified, process 1900 may be converted to box 2308 of process 2300. If no lasso gesture is identified, process 1900 may continue. In another example, a lasso gesture drawn by a user may have a closed shape with a start and / or end "tail". This example combination of processes 1900 and 2300 may select all objects (including tails) that intersect the path of the lasso gesture from the user's perspective, and may select all objects that fall within the distorted cone formed by the closed shape portion of the lasso gesture. In another example, process 2100 may be combined with process 2300 so that any operable object selected by process 2300 may also be actuated.

[0165] Fig.24A is a conceptual diagram illustrating an example 2400 of selecting an object using a lasso gesture. In example 2400, a user begins by making gesture 2402, placing the tips of her thumb, index finger, and middle finger together. The hand interaction system then begins tracking a control point 2404 (a point offset in front of gesture 2402) and a projection direction formed by determining a line connecting the user's eyes (not shown) and control point 2404. Thus, from the user's perspective, moving control point 2404 can be used to "draw" a path in an artificial reality environment. When the user first makes gesture 2402, she begins drawing at point 2406. The user then moves her hand, causing control point 2404 to move, creating path 2408. When path 2408 intersects itself at point 2410 (from the user's perspective), the hand interaction system determines, based on the machine learning model, that the user has drawn a lasso selection. The hand interaction system then forms the closed segment of path 2408 into a warp cone (not shown) and determines that the warp cone at least partially intersects objects 2412, 2414, and 2416. The hand interaction system therefore selects intersecting objects 2412, 2414, and 2416.

[0166] Fig. 24Bis another conceptual diagram illustrating example 2450 of selecting an object using a lasso gesture. In example 2450, a user forms a path having a closed shape 2452 by moving control points based on a gesture in a manner similar to example 2400. The hand interaction system forms a distorted cone 2454, the tip of which is at the user's dominant eye 2456 and extends so that the edge of the distorted cone coincides with the closed shape 2452. The hand interaction system then determines any objects that extend beyond shape 2452 (i.e., on the opposite side of closed shape 2452 from the user) and intersect with that portion of distorted cone 2454. In example 2450, these are objects 2456 and 2458. The hand interaction system therefore selects intersecting objects 2456 and 2458, rather than non-intersecting objects 2460 and 2462.

[0167] Fig.25 is a flow diagram illustrating a process 2500 for selecting an object using a frame gesture used in some implementations of the present technology. At block 2502, the process 2500 may identify an action corresponding to initiating object selection. In some implementations, the action may be a two-handed "pinch" gesture formed by a user touching from the thumb of each hand to the index or middle finger of that hand, the two pinches touching at the thumb / finger intersection point. In some implementations, the action may be a two-handed "L" gesture formed by the thumb of a first hand being raised perpendicular to the ground, the index finger being parallel to the ground, and the thumb of a second hand being extended perpendicularly downward to the ground, the index finger being parallel to the ground, defining two opposite corners of a rectangle. In some implementations, the gesture may be the fingers of a user on one or both hands, defining another shape, such as a triangle or an oval. In some implementations, the gesture may be formed using one hand. For example, the gesture may be a pinch that, when opened, defines opposite points of a rectangle corresponding to the points at the tips of the user's thumb and fingers. As another example, the gesture may be a pinch that begins the system tracking a control point. The control point can be moved to draw a line, thereby defining opposite points of the rectangle as the start and end points of the line. In some implementations, the action can also include identifying a gating condition, such as the hand tracking system providing one or more indications that the user's hand is in view, the confidence value of the current hand tracking position is above a threshold, or the hand is not at rest.

[0168] At block 2504, process 2500 may continuously determine a shape defined by a first tracked portion of a hand of a first user and a second tracked portion of a hand of a second user. The shape may be a rectangle and the tracked portion of the hand may be the points corresponding to the opposite corners of the rectangle identified at block 2502. As the user moves her hand, process 2500 may determine the new size and position of the rectangle. In other implementations, a shape may be defined by attributing lines or curves to portions of a user's hand, connecting lines, and / or snapping lines to conform to a given shape. For example, process 2500 may be configured to identify a rectangle and a user may make an L-shape using each hand. Process 2500 may attribute lines to each finger to form a portion of the L-shape, extend them to intersect, and snap the four lines to form a right angle to form a rectangle. As another example, process 2500 may be configured to recognize an oval and a user may make a "C" shape using the thumb and index finger of one hand and another reverse "C" shape using the thumb and index finger of the other hand. Process 2500 can be for attributing curves to each C-shape, extending them to intersect and smoothing the curves to form ellipses. As described above, in some implementations, the shapes can be defined using a single hand gesture, such as a rectangle formed by opposite corners defined at points on the user's index finger and thumb or defining opposite corners of a rectangle as the start and end points of a line drawn by the user (in a manner similar to the paths traced by blocks 1902 and 1904).

[0169] At block 2506, process 2500 can display a representation (such as an outline) of the shape determined at block 2504. For example, if the user is gesturing to pull apart opposing corners of a rectangle using a pinch gesture, the rectangle can be drawn continuously, thereby showing the outline of the rectangle to the user. In some implementations, process 2500 does not display a shape representation and block 2506 is skipped.

[0170] At block 2508, process 2500 can identify a pyramid formed at one eye of the user having a tip of the pyramid and pyramid walls formed based on the rectangle determined at block 2504. For example, the pyramid can be formed by four triangles extending from the user's eye and passing through one side of the rectangle. In some implementations where the shape determined at block 2504 is not a rectangle, other 3D shapes can be determined, such as a tetrahedron if the shape is a triangle or a cone if the shape is an ellipse (formed similarly to block 2310).

[0171] At block 2510, in response to one or more triggers, such as after continuously identifying the action at block 2502, when the user releases the gesture identified at block 2502, or when the speed of the user's hand movement drops below a threshold, the process 2500 can select an object that falls within the pyramid (or other shape) identified at block 2508. In some implementations, the process 2500 can also select an object that at least partially intersects the pyramid (or other shape) or has a threshold amount (e.g., at least 20%, 50%, or 85%) of its volume within it. For example, the process 2500 can begin at block 2502 by recognizing that the user has formed a two-touch pinch gesture. At blocks 2504-2510, the process 2500 can continuously determine the size and position of the rectangle formed by the diagonal corners at the user's two pinch gestures as the user pulls them apart. Process 2500 may then draw the rectangle for viewing by the user, identify a pyramid having a tip at the user's dominant eye and having a pyramidal shape defined by the rectangle, and cause any object that at least partially falls within the pyramid to be selected.

[0172] In some implementations, multiple selection processes described herein may be combined into a single gesture. For example, processes 2100 and 2500 may be combined so that any operable object selected by process 2500 may also be actuated.

[0173] Fig.26A is a conceptual diagram illustrating an example 2600 of selecting an object using a framing gesture. In example 2600, a user begins by making gestures 2602A and 2602B, placing the tip of her thumb and the tip of her index finger together and touching these gestures together at point 2612. The hand interaction system then begins tracking the opposite corners of a rectangle 2604 from the user's perspective based on the positions of gestures 2602A and 2602B. Thus, from the user's perspective, moving the points of gestures 2602A and 2602B forms rectangle 2604. As the user spreads her hands apart, the corners of rectangle 2604 spread apart, increasing the size of rectangle 2604. As the user forms the rectangle, the hand interaction system determines a pyramid formed by the tip of the pyramid at the user's dominant eye and the sides extending through the edges of rectangle 2604. In example 2600, the hand interaction system continuously selects (or deselects) objects that at least partially intersect the pyramid until the user releases one of gestures 2602A or 2602B. The hand interaction system therefore selects intersecting objects 2606 and 2608 , but not 2610 .

[0174] Fig.26Bis another conceptual diagram illustrating example 2650 of selecting an object using a framing gesture. In example 2650, a user forms a rectangle 2652 by forming two pinch gestures and pulling them apart in a manner similar to example 2600. The hand interaction system forms a pyramid 2654 with the tip of the pyramid located at the user's dominant eye 2656 and extending so that the four triangles forming the edges of the pyramid coincide with rectangle 2652. The hand interaction system then determines any objects that extend beyond rectangle 2652 (i.e., on the side of rectangle 2652 opposite the user) and fall completely within pyramid 2654. In example 2650, this is object 2658. The hand interaction system therefore selects enveloping object 2658, rather than non-enveloping objects 2660 and 2662.

[0175] Two-handed interaction

[0176] One-handed gestures limit how a user can behave when interacting with an XR system. A hand interaction system can track the poses of a user's two hands, allowing a second hand (e.g., a non-dominant hand) to provide nuances to gestures controlled by a first hand (e.g., a dominant hand). These two-handed (i.e., two-handed) interactions allow users to transfer their everyday experience and skills to interactions in an artificial reality environment. For example, when performing interactions using the projection projection techniques discussed above, such as when multiple objects occlude each other, when objects are tightly grouped and the user wants to select a subset of them, or when objects are scattered and the user wants to select all objects, the projection may intersect with more or fewer objects than the user expected. In these cases, a second gesture performed by a second hand (and / or using a different part of the same hand) can control aspects of the projection, such as the length of a ray projection (e.g., a "hook" interaction) or the diameter of a cone projection (e.g., a "trombone" interaction). In some implementations, a user may want to perform more complex interactions with one or more target objects, and may specify the details of these interactions by using her second hand to interact with controls.

[0177] Fig. 27 2700 is a flow chart illustrating a process 2700 used in some implementations of the present technology for disambiguating between target and non-target objects intersected by a ray projection. For example, this can allow a user to disambiguate objects that occlude each other when using ray projection.

[0178] At block 2702, process 2700 may determine a control point and a projection direction based on the tracked positions of one or more body parts. Process 2700 may be performed in a manner similar to block 1202. At block 2704, process 2700 may generate a ray projection from the control point along the projection direction. Process 2700 may be performed in a manner similar to block 1204.

[0179] At block 2706, process 2700 may continuously determine a distance relationship between a first hand (e.g., a dominant hand) and a second hand (e.g., a non-dominant hand) controlling a control point. In some implementations, the distance relationship may be based on a distance between two fingers on one hand (e.g., the tip of an index finger and the tip of a thumb), rather than a distance between two hands. In other implementations, the distance relationship may be based on a distance between a hand and another body part (such as a torso of a user). In some implementations, the ray projection may be along the user's gaze, and the user may specify a distance by moving her hand closer to or farther from her eyes along the ray. In various implementations, the distance may be in any direction or may be in a specific direction (e.g., perpendicular to the ray or along the ray).

[0180] In various implementations, the distance relationship can be linearly or exponentially proportional to the actual distance between the user's hands. In some implementations, the relationship can be based on the speed at which the user changes the distance between her hands. For example, the amount determined by process 2700 can be increased or decreased logarithmically so that faster movements increase more amounts than slow movements. This allows the user to use fast movements to control amounts with greater granularity and slower movements for more precise adjustments. In some implementations, the distance relationship can be scaled according to the size of the determined artificial reality environment. For example, in a smaller space, the movement can be more precise, while in a larger space, the movement can further extend the ray to interact with objects at a greater distance.

[0181] At block 2708, process 2700 can continuously set the length or "hook" location of the ray projection based on the distance relationship determined at block 2706. In some implementations, instead of setting the ray length, process 2700 can set an interaction point along the ray based on the distance relationship. Whether at the end of the ray or at some point along the ray, the interaction point is referred to herein as a "hook." In some implementations, process 2700 can provide a visual representation of the ray and can include a graphical representation of the hook (such as a sphere).

[0182] At block 2710, process 2700 may identify one or more objects based on the intersection with the hook. In some implementations, this may occur when the hook intersects the object. In other implementations, process 2700 may identify a third gesture performed by the user, such as making a click gesture with one hand, nodding, tapping her foot, etc., as a trigger for selecting the object that the hook intersects when the gesture is made. At block 2712, process 2700 may toggle selection of or otherwise interact with the object identified at block 2710.

[0183] Fig.28 and Fig.292800 and 2900 are conceptual diagrams illustrating disambiguation between selected and unselected objects that intersect a ray projection 2802 using a hook. Example 2800 begins with a user creating a ray projection 2802 by performing a pinch gesture 2804 between the thumb and middle finger on her dominant hand 2806. The user can position the ray projection 2802 so that it intersects objects 2808 and 2810. The user can then control the length of the ray 2802 using a hook 2814 at the end of the ray 2802 based on the distance 2816 between her dominant hand 2806 and her non-dominant hand 2818. When the hook 2814 intersects the object 2810 that the user intends to aim at, she can make a second pinch gesture (not shown), this time using the index finger and thumb on her dominant hand 2806. When the hand interaction system identifies this gesture, it selects the object 2810 with which the hook 2814 now intersects. In example 2900, the user then changes the length of ray 2802 by extending the distance 2816 between her dominant hand 2806 and her non-dominant hand 2818, and changes the position of hook 2814 accordingly, while still maintaining the first thumb / middle finger pinch, so that hook 2814 intersects object 2812. The user can also select object 2812 by again making gesture 2902 using the thumb / index finger pinch, and object 2812 now intersects hook 2814.

[0184] Fig.30 3000 is a diagram illustrating a process used in some implementations of the present technology for disambiguating between target and non-target objects that intersect a cone or cylinder projection using a "trombone" gesture. For example, this can allow a user to expand or contract the diameter of a cone or cylinder projection to easily control which objects the cone or cylinder intersects.

[0185] At block 3002, process 3000 may determine control points and a projection direction based on the tracked positions of one or more body parts. Process 3000 may be completed in a manner similar to block 1402 or 1602. At block 3004, process 3000 may generate a conic or cylindrical projection from the control points along the projection direction. Process 3000 may be completed in a manner similar to block 1404 or 1604.

[0186] At block 3006, process 3000 may continuously determine a distance relationship between a first hand (e.g., a dominant hand) and a second hand (e.g., a non-dominant hand) controlling a control point. In some implementations, instead of a distance between two hands, the distance relationship may be based on a distance between two fingers on one hand (e.g., the tip of an index finger and the tip of a thumb). In other implementations, the distance relationship may be based on a distance between one hand and another body part (such as a torso of a user). In some implementations, a cone or cylinder projection may be along the user's gaze and the user may specify a distance by moving her hand closer to or further away from her eyes along the center of the cone or cylinder. In various implementations, the distance may be in any direction or may be in a specific direction (e.g., perpendicular or parallel to the center of the cone or cylinder).

[0187] In various implementations, the distance relationship can be linearly or exponentially proportional to the actual distance between the user's hands. In some implementations, the relationship can be based on the speed at which the user changes the distance between her hands. For example, the amount determined by process 3000 can be increased or decreased logarithmically so that faster movements increase the amount more than slower movements. This allows the user to use fast movements to control the amount with greater granularity and slower movements to make more precise adjustments.

[0188] At block 3008, process 3000 can continuously set the diameter of the cylinder or the diameter of the base of the cone based on the distance relationship determined at block 3006. In some implementations, process 3000 can provide a visual representation of the cone or cylinder showing it changing size as the user changes the distance relationship.

[0189] At block 3010, process 3000 may identify one or more objects based on at least partial intersection or complete coverage by the cone or cylinder. In some implementations, this may occur when the cone or cylinder intersects the object. In other implementations, process 3000 may identify a third gesture performed by the user, such as making a click gesture with one hand, nodding, tapping her foot, etc., as a trigger to identify any objects that intersect the cone or cylinder at that moment. At block 3012, process 3000 may toggle selection of or otherwise interact with the object identified at block 3010.

[0190] In an alternative implementation that combines processes 2700 and 3000, a user may control the length of a cylinder or cone using one gesture (e.g., the distance between the tip of the thumb and the index finger on the dominant hand) and may control the diameter of the base of the cylinder or cone using another gesture (e.g., the distance between her two hands). As a modification of this alternative, a spherical projection may be used instead of a cylindrical or conical projection, wherein the user controls the position of a sphere along a ray based on one gesture (e.g., the distance between her two hands) and controls the diameter of the sphere using another gesture (e.g., the distance between the tip of the thumb and the index finger on the dominant hand).

[0191] Fig.31 and Fig.32 3100 and 3200 are conceptual diagrams illustrating examples of a trombone gesture for disambiguating between selected and non-selected objects that intersect a conic projection. Example 3100 begins with a user creating a conic projection 3102 by performing a pinch gesture 3104 between her thumb and middle finger on her dominant hand 3106. The user can position the conic projection 3102 so that it intersects objects 3108, 3110, 3112, and 3120. The user can then control the diameter of the base 3114 of the conic projection 3102 based on the distance 3116 between her dominant hand 3106 and her non-dominant hand 3118. The hand interaction system selects objects 3108, 3110, 3112, and 3120 because they intersect the conic projection 3102. In example 3200, the user has changed the diameter of the base 3114 of the conic projection 3102 by changing the distance 3116 between her dominant hand 3106 and non-dominant hand 3118, while still maintaining the first thumb / middle finger pinch gesture, resulting in the conic projection 3102 intersecting only with objects 3112 and 3108. In response, the hand interaction system can deselect objects 3120 and 3110, but keep objects 3108 and 3112 selected. The user can continue until her intended target object is selected, at which point she releases gesture 3104.

[0192] Fig.33 and Fig.343300 and 3400 are conceptual diagrams illustrating examples of a trombone gesture for disambiguating between selected objects and non-selected objects that intersect with a cylindrical projection. Example 3300 begins with a user creating a cylindrical projection 3302 by performing a pinch gesture 3304 between her thumb and middle finger on her dominant hand 3310. The user can position the cylindrical projection 3302 so that it intersects objects 3308, 3306, and 3330. The user can then control the diameter of the base 3314 of the cylinder 3302 by changing the distance 3316 between her dominant hand 3310 and her non-dominant hand 3318. The hand interaction system selects objects 3308, 3306, and 3330 because these are objects that intersect with the cylindrical projection 3302, but does not select object 3332 because it does not intersect with the cylindrical projection 3302. In example 3400, the user then changes the diameter of the base 3314 of the cylindrical projection 3302 while still maintaining the first thumb / middle finger pinch by changing the distance 3316 between her dominant hand 3310 and non-dominant hand 3318. This change causes the cylindrical projection 3302 to intersect objects 3306, 3308, 3330, and 3332. The user can continue until her intended target object is selected, at which point she releases gesture 3304.

[0193] While gesture combinations as described above are powerful interaction techniques, in some cases mapping a different gesture to each action can confuse the user or limit which actions the user can perform. However, when interacting with distant objects, menus attached to those objects can be difficult for the user to interact with. World-locked menus can limit where the user wants to move themselves or the target object when interacting with the menu. It may also be difficult for the user to match the world-locked menu to the target object. These issues can be overcome by having a menu associated with the object selected using projection, but with the menu attached to the projection hand instead of the object. This allows a second hand to interact with the menu and thus easily interact with the object. Fig.35 is a flow diagram illustrating a process 3500 used in some implementations of the present technology for two-handed interaction with an object using a menu attached to a dominant hand.

[0194] At block 3502, process 3500 may receive identification of one or more objects selected using a dominant hand. In some implementations, the selection may be performed using one of the projection or multiple selection processes described above. In some implementations, the selection may be associated with a particular gesture (such as a gesture for initiating projection or selection), and process 3500 may continue only while the user maintains the gesture. For example, a user may cast a raycast to select an object, and process 3500 may continue to provide a menu corresponding to the selected object until the user moves the projection (deselecting the object) or releases the projection gesture.

[0195] At box 3504, process 3500 may obtain one or more menus associated with one or more objects, and process 3500 received an indication of one or more objects at box 3502. At box 3506, process 3500 may display one or more menus associated with the dominant hand. In other implementations, process 3500 may display one or more menus associated with a non-dominant hand or a hand that performs selection of one or more objects, rather than always displaying one or more menus associated with the dominant hand. In various implementations, each of the one or more menus may be offset from a point on the hand (such as a wrist, a center of a thumb, a fingertip, a middle of the back of the hand, a center of mass of the hand, etc.), covered as a "skin" at such a point, wrapped around a portion of the hand (e.g., displayed as a bracelet, a ring, etc.), or a combination thereof. In some implementations, one or more menus may be triggered to appear and disappear based on gestures such as movement of a particular finger or rotation of the hand.

[0196] At box 3508, process 3500 can receive an interaction with one or more menus displayed. In some implementations, the interaction can occur using the same hand associated with the menu (e.g., a "ring" menu that appears on the middle finger can be tapped by the thumb on the same hand) or can be an interaction from the opposite hand, whether or not it is the non-dominant hand. One or more menus can have controls to perform any type of interaction with the selected object, and interaction with the menu can trigger that interaction. In some implementations, a menu selection can cause the hand interaction system to interpret further gestures in a particular way. For example, if the menu selection is "move", movement of the dominant hand can cause the hand interaction system to move the selected one or more objects accordingly. As another example, if the menu selection is "rotate", rotation of the dominant hand can cause the hand interaction system to rotate the selected one or more objects proportionally.

[0197] Fig.36 , Fig.37 and Fig.383600, 3700, and 3800 are conceptual diagrams illustrating examples 3600, 3700, and 3800 of two-handed interaction with an object using a menu attached to a dominant hand. In example 3600, an indication of an object 3602 is received as a result of selecting the object 3602 using a ray projection 3604 cast by a user's dominant hand 3606. In example 3700, in response to the selection of the object 3602, the menu 3702 is associated with the dominant hand 3606, offset from the center of mass of the dominant hand 3606. The menu 3702 includes controls for the object 3602, which can be actuated by the user's non-dominant hand 3710. In example 3700, the controls include a control 3704 for performing a 90 degree rotation of the selected object 3602, a control 3706 for allowing the selected object 3602 to rotate freely according to the rotation of the dominant hand 3606, and a control 3708 for resizing the selected object 3602. In example 3800, the user has actuated control 3706 by performing a pinch gesture 3802 next to control 3706 using her non-dominant hand 3710. Actuation of control 3706 causes the hand interaction system to map the rotation of dominant hand 3606 to a rotation of selection object 3602.

[0198] Operation Mode

[0199] There are different types of interactions that users can perform with respect to objects, such as moving objects and actuating controls on objects in an artificial reality environment. However, gestures for performing such interactions may be ambiguous, causing existing XR systems to be unable to correctly distinguish them. In particular, accuracy decreases with increasing distance, and users of existing XR systems may find the interaction annoying and stressful because these systems use the same input modality in different situations and cannot distinguish the type of interaction. To address these issues, a hand interaction system can switch operating modes (e.g., switching between global and local interaction modes). The global mode can be used to manipulate objects within the environment (e.g., select objects in a virtual environment, control object placement, orientation, size, etc.) or switch to an input modality useful for such global interactions (e.g., ray, sphere, or cone projection). The local mode can be used to interact with the interface of an object (e.g., press a button on an object or interact with a menu associated with the object) or switch to an input modality useful for such local interactions (e.g., mapping a button to a user's finger). Therefore, ambiguous input can be interpreted according to any mode currently set for the hand interaction system. In various implementations, alternative or additional operating modes can be used. For example, more than two modes may be defined, modes may be defined for different purposes, modes may be triggered in different ways, and so on.

[0200] In various implementations, a user may explicitly select between operating modes (e.g., via a gesture or button press) or the mode selection may be implicit (e.g., based on user gaze). In response to an explicit or implicit trigger corresponding to a switch between operating modes, the hand interaction system may then interpret the input gesture based on a mapping of the gesture to actions defined for the selected mode. For example, in response to identifying a switch to a first mode (e.g., a global mode), the hand interaction system may interpret the first gesture by mapping the first gesture to a first action in a first set of actions defined for the first (e.g., global) mode. Then, in response to a second trigger resulting in a switch to a second operating mode (e.g., a local mode), the hand interaction system may interpret the second gesture by mapping the second gesture to a second action in a second set of actions defined for the second (e.g., local) mode.

[0201] In some implementations, the input modality can be altered based on which operating mode is activated. For example, in global mode, the input can be switched to an input modality based on determining the direction of the gesture (e.g., a projection projection such as a ray, cylinder, sphere, or cone projection). In local mode, the input can be switched to other modalities (such as by mapping a control associated with a selected object to a finger on the user's hand), where a particular detected movement of the mapped finger controls the actuation of the corresponding control.

[0202] Fig.393900 is a flowchart illustrating a process 3900 for switching input modalities (e.g., for global or local mode interactions) used in some implementations of the present technology. At box 3902, the process 3900 may identify a trigger for switching between operating modes, and at box 3904, the process 3900 may identify which mode the trigger corresponds to (e.g., whether the trigger is to enter a global input mode or to enter a local input mode). In some implementations, the trigger may be an explicit user action. As an example, a particular gesture may cause the hand interaction system to switch between modes or to switch to a first (e.g., local) mode when the gesture is held, and to switch to a second (e.g., global) mode when the gesture is not identified; the hand interaction system may switch to a first mode when the input hand is rotated vertically, and to a second mode when it is not rotated vertically; the hand interaction system may switch between modes in response to a real or virtual button press or voice command, etc. In some implementations, the trigger to switch modes may be implicit. As an example, the hand interaction system may enter a local mode when the user's tracking gaze stays on an object for more than a threshold amount of time (e.g., one, two, or three seconds), and may enter a global mode when the user's gaze on the object ends for another threshold amount of time. In another example, the hand interaction system may start in a global mode, enter a local mode when the user first selects an object, and switch back to a global mode when the user taps the object or otherwise performs a defined "return" gesture. In some implementations where the trigger is a related gesture such as rotating the user's hand, hysteresis may be applied to prevent jitter between modes when the user's hand is on the boundary between modes. If the trigger switches to a first (e.g., global) mode, process 3900 may continue to box 3906, and if the trigger switches to a second (e.g., local) mode, process 3900 may continue to box 3908.

[0203] At block 3906, process 3900 can switch to an input modality established for the current mode (e.g., global mode). In some implementations, the input modality for the global mode can be based on determining a gesture direction (e.g., a projected projection (e.g., ray, sphere, cylinder, or cone) as described above). In other implementations, the input modality can present a group of controls for global interaction (e.g., rotating, resizing, or moving an object) when an object is selected.

[0204] At box 3908, process 3900 can switch to an input modality established for a second mode (e.g., local mode). In various implementations, the input modality can be a mapping of available interactions to specific finger movements or other gestures; a switch between projection types, such as switching from cone projection to ray projection; or displaying controls configured to perform positional interactions. In some implementations, local interactions can be directed to specific selected objects, such as one or more objects selected in global mode. This allows the user to perform interactions related to the selected one or more objects without having to point or otherwise direct her interactions to the target object. In some cases, the selected object can be changed, for example, based on the user's gaze, without exiting the current (e.g., local) mode. In other implementations, local interactions can be based on objects indicated by the user.

[0205] In some implementations, process 3900 can be performed separately for each hand. For example, the user's dominant hand can be in a first (e.g., global) mode, controlling the projection for selecting an object, while the user's non-dominant hand can be in a second (e.g., local) mode, where controls are mapped to fingers on the non-dominant hand to interact with any object selected by the dominant hand.

[0206] After executing block 3906 or 3908, process 3900 may return to block 3902 when another trigger for switching to an alternate global or local mode is identified.

[0207] Fig.40A and Fig.40B is a conceptual diagram illustrating an example 4000 of switching input modalities for global or local mode interactions. In example 4000, a user switches between local and global modes based on the rotation direction of her hand 4002. In example 4000, the user has caused the hand interaction system to enter local mode by rotating her hand 4002 vertically as indicated by arrow 4004 (e.g., representing that the flat plane of her hand is perpendicular to the established floor plane by at least a threshold number of degrees, such as 30 degrees).

[0208] Upon entering local mode, local mode control group 4008 appears along with previously selected object 4006. Controls 4008 include buttons 4010-4016. In this example, control 4010 is mapped to finger 4018, control 4012 is mapped to finger 4020, control 4014 is mapped to finger 4022, and control 4016 is mapped to finger 4024. Because the user has extended finger 4018, the corresponding mapped control 4010 is activated.

[0209] Example 4000 in Fig.40B4052, where the user has caused the hand interaction system to switch to global mode by rotating her hand 4002 to horizontal as indicated by arrow 4052 (e.g., representing that the flat plane of her hand is at least a threshold degree, such as 30 degrees, from the established ground plane, rather than perpendicular). In response, the hand interaction system switches the input modality from Fig.40A The hand mapping control switches to ray projection 4054, allowing the user to further perform global interaction with object 4006 or other objects using ray projection 4054.

[0210] Fig.41 4100 is a flow diagram illustrating a process 4100 used in some implementations of the present technology for interpreting a gesture as a specific interaction in a current (e.g., global or local) mode. At block 4102, process 4100 may identify a trigger for transitioning between two modes (e.g., a global mode and a local mode), and at block 4104, process 4100 may identify which mode the trigger corresponds to (e.g., whether the trigger is to enter a global mode or to enter a local mode). Blocks 4102 and 4104 may be performed similar to blocks 3902 and 3904. If the trigger is for a first (e.g., global) mode, process 4100 may continue to block 4106, and if the trigger is for a second (e.g., local) mode, process 4100 may continue to block 4108.

[0211] At box 4106, process 4100 can cause the hand interaction system to interpret further user input according to the current (e.g., global) mode. And at box 4108, process 4100 can cause the hand interaction system to interpret further user input according to the second (e.g., local) mode. The actions that can be taken for the object can be divided into one of the executable available modes (e.g., in global or local mode). The hand interaction system can match the input gesture with the action in the action group defined for the current activation mode. For example, in global mode, the gesture of dragging downward from the upper right corner of the virtual chessboard can be interpreted as the intention of rotating the chessboard. However, in local mode, the same gesture can be interpreted as moving the car forward from the chess grid in the upper right corner by an amount corresponding to the length of the gesture. As another example, in the case where the target object is a virtual drawing application, the hand interaction system can interpret the click and drag gesture in the local mode as drawing a line within the application, and the hand interaction system can interpret the same gesture as a command to rotate the virtual panel of the display drawing surface in global mode. In some implementations, process 4100 can be performed separately for each hand. For example, a user's dominant hand may be in a first (eg, global) mode, controlling the orientation of a 3D model, while the user's non-dominant hand may be in a second (eg, local) mode, actuating controls on the face of the 3D model.

[0212] After executing block 4106 or 4108, process 4100 may return to block 4102 when another trigger for switching to the alternate mode is identified.

[0213] Fig.42A and Fig.42B is a conceptual diagram illustrating an example 4200 of interpreting a gesture as an interaction in global or local mode. In example 4200, a user transitions to local mode based on her gaze remaining on an object for at least two seconds, and transitions to global mode when her gaze leaves the object for at least 1.5 seconds. In example 4200, a user causes the hand interaction system to enter local mode associated with object 4204 by directing her gaze 4202 toward object 4204 for two seconds.

[0214] Example 4200 in Fig.42B 4254, where the user uses her hand 4252 to control the ray casting 4254. Because the user enters local mode based on her gaze (e.g. Fig.42A ), so the ray projection 4254 intersects the object 4204 to cause local interaction by actuating the control 4256 of the object 4204. The user's gaze ( Fig.42B 4204 for more than 1.5 seconds, so the hand interaction system remains in local mode. In some implementations, processes 3900 and 4100 can be performed together so that switching between global mode and local mode can change the input modality and cause ambiguous input to be interpreted according to the current global or local mode.

[0215] References in this specification to "implementations" (e.g., "some implementations," "various implementations," "an implementation," "implementation," etc.) indicate that a particular feature, structure, or characteristic described in conjunction with the implementation is included in at least one implementation of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same implementation, nor are independent or alternative implementations mutually exclusive of other implementations. In addition, various features are described that may be exhibited by some implementations but not by other implementations. Similarly, various requirements are described that may be requirements for some implementations but not for other implementations.

[0216] As used herein, a “finger” may be any of the five fingers of a hand, and a fingertip may be the tip of any of the five fingers of a hand.

[0217] As used herein, above a threshold value means that the value of the item being compared is higher than other specified values, the item being compared is in a certain specified number of items with a maximum value, or the value of the item being compared is within a specified maximum percentage value. As used herein, below a threshold value means that the value of the item being compared is lower than other specified values, the item being compared is in a certain specified number of items with a minimum value, or the item being compared has a value within a specified bottom percentage value. As used herein, within a threshold value refers to that the value of the item being compared is between two other specified values, the item being compared is between the items of a specified number in the middle, or the value of the item being compared is within a specified percentage range in the middle. Relative terms, such as high or unimportant, when not otherwise defined, can be understood as assigning a value and determining how the value is compared with a given threshold value. For example, the phrase "selecting a fast connection" can be understood to mean selecting a connection with a value assigned corresponding to its connection speed above a threshold value.

[0218] As used herein, the word "or" refers to any possible permutation of a set of terms. For example, the phrase "A, B, or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiples of any term, such as A and A; B, B, and C; A, A, B, C, and C, etc.

[0219] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. For illustrative purposes, specific embodiments and implementations have been described herein, but various modifications may be made without departing from the scope of the embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited by the appended claims.

[0220] Any of the above patents, patent applications and other references are incorporated herein by reference. As needed, various aspects may be modified to adopt the systems, functions and concepts of the above various references to provide further implementations. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for selecting a plurality of objects in an artificial reality environment, the method comprising: Determining a control point and a projection direction based on the position of one or more bodies or body parts of the user; identifying a distance between the first hand of the user and the second hand of the user based on one or more images from one or more cameras capturing the first hand of the user and the second hand of the user; generating a conical projection starting at the control point and extending along the projection direction, wherein a diameter of at least a portion of the conical projection is based on the identified distance; identifying a plurality of objects that at least partially intersect at least a portion of the conical projection; as well as The plurality of objects are selected.

2. The method according to claim 1, wherein: The tip of the cone projection is at the control point; The conical projection extends outwardly from the user from a tip of the conical projection; and The conical projection is centered in the projection direction from the tip of the conical projection to the center of the circular base of the conical projection.

3. The method of claim 1 or 2, wherein the projection direction is along a line connecting one eye of the user to the control point.

4. A method according to any one of claims 1 to 3, wherein the method is performed in response to identifying that the user has performed a pinch gesture between the user's thumb and middle finger, and wherein the control point is defined relative to the point where the user's thumb and middle finger touch.

5. A method for selecting a plurality of objects in an artificial reality environment, the method comprising: identifying an action corresponding to initiating object selection; continuously determining the control point and the projection direction based on the tracked positions of one or more body parts of the user as the user moves the control point; determining that the user movement of the control point is a lasso input; identifying a distortion cone, wherein a shape of the distortion cone is formed by a closed shape defined by the lasso input; and Select objects that fall within the distortion cone. 6 . The method of claim 5 , wherein the action corresponding to initiating object selection comprises a pinch between a thumb and an index or middle finger of the user.

7. The method according to claim 5 or 6 also includes displaying a path based on the projection direction to the user in the artificial reality environment.

8. A method according to any one of claims 5 to 7, wherein continuously determining control points and projection directions includes providing an indication of the path to a machine learning model, the machine learning model being trained to identify lasso inputs.

9. A method for selecting a plurality of objects in an artificial reality environment, the method comprising: identifying an action corresponding to initiating object selection with a framing gesture; continuously determining a shape defined by a first tracked portion of a first user's hand and a second tracked portion of a second user's hand; identifying a formed pyramid, wherein a wall of the pyramid is formed based on the determined shape; and Objects that fall within the pyramid are selected.

10. The method of claim 9, wherein the action associated with initiating object selection with a frame gesture comprises one of: a two-handed pinch gesture formed by the user's thumb from each hand touching the index or middle finger of said hand, wherein the two said pinches touch at the intersection of said thumb / fingers; a two-handed L-shaped gesture formed by the thumb of a first hand extending upward perpendicular to the ground and the index finger of the first hand extending parallel to the ground, and the thumb of the second hand extending downward perpendicular to the ground and the index finger of the second hand extending parallel to the ground, thereby defining two opposite corners of a rectangle; fingers on one or both hands of the user define a shape; a pinch that, when opened, defines opposing points of a rectangle corresponding to points at the tips of the user's thumb and fingers; as well as A pinch defines a control point, wherein movement of the control point defines a line, and a start point and an end point of the line define opposite points of a rectangle.

11. The method of claim 9 or 10, further comprising displaying a representation of the determined shape to the user in the artificial reality environment.

12. The method of any one of claims 9 to 11, wherein continuously determining a shape defined by a first tracked portion of a first user's hand and a second tracked portion of a second user's hand comprises: a curved line formed into a C shape using the index finger and thumb of each hand of the user; extending the curves to intersect; as well as The curve is smoothed to form an ellipse.

13. A computing system comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of the preceding claims.

14. A computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 12.

15. A computer program product comprising instructions which, when the program is executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 12.

Images