Multi-mode cursor movement
Through the multi-mode hand cursor system, the problem of insufficient interaction functions of the existing XR system in the 3D environment is solved, and the rich interaction in 3D space and on the 2D plane is achieved, which improves the user experience.
Patent Information
- Application Number
- CN202410687242.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-16
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
The existing extended reality (XR) systems provide insufficient interaction functions in the 3D environment, especially in the interaction of the environment in the 3D space and the interaction of the object surface, which is difficult to meet the diverse needs of users.
Using a multi-mode hand cursor system, dynamically switches two modes by tracking the user's hand movement: one mode converts 3D hand movement into 2D hand cursor movement to move on the 2D plane, the other mode allows the hand The cursor and virtual elements move 6DOF in the 3D space to realize multi-dimensional interaction of virtual elements.
It enhances users' interaction capabilities in the 3D environment and provides richer interaction methods, including movement on the 2D plane and movement in the 3D space, meeting the multi-dimensional manipulation needs of virtual elements and improving user experience.
Smart Images

Figure CN119065543A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to systems, methods, and devices that enable interaction with elements in a three-dimensional (3D) environment based on hand tracking. Background Art
[0002] Extended reality (XR) systems provide content within a 3D environment, with which a user can interact via various input mechanisms. Such systems may not provide adequate interaction functionality, for example, in terms of using input to provide both interaction in the 3D space of the environment (e.g., moving objects around in 3D), and interaction on the surface of objects within the 3D space (e.g., controlling selection and manipulation of objects on a 2D user interface displayed within the 3D space). Summary of the invention
[0003] Various embodiments disclosed herein include devices, systems, and methods that enable interaction with virtual elements within a (3D) space of an extended reality (XR) environment. Interaction may be enabled based on a cursor (visible or invisible) that moves relative to user hand motion tracking. Some embodiments enable the use of two modes, such as two hand cursor modes. The first mode allows a user's 3D hand motion to move a hand cursor (and / or a virtual cursor) on a two-dimensional (2D) plane (e.g., the surface of a UI or other object) in response to converting the 3D hand cursor motion into a 2D hand cursor motion. For example, a hand cursor (and / or a virtual cursor) may move along a flat surface of a 2D user interface displayed in a 3D space. The second mode allows a user's 3D hand motion to move a hand cursor in a 3D space without converting the 3D cursor motion into a 2D cursor motion. For example, a hand cursor and / or an associated object (e.g., an entire user interface) may move in a 3D space. In some implementations, the hand cursor can be linked to a virtual element (such as a user interface in a 3D space) so that the virtual element moves with the cursor relative to six degrees of freedom (6DOF). Movement relative to 6DOF allows the hand cursor and the virtual element to move in a forward direction, a backward direction, a left direction, a right direction, an upward direction, a downward direction, a twisting direction / motion, a tilting direction / motion, etc. based on the 3D movement of the hand.
[0004] In some implementations, the first mode may be automatically enabled until the second mode is triggered (or vice versa). For example, the second mode may be triggered in response to a user positioning a hand cursor over a specified object type (e.g., an object that can be moved in 3D space, a user interface (UI) position control icon (which allows object (e.g., UI) drag-and-drop operations, drag operations), etc.) and providing a user selection input (e.g., a user performing a finger pinch operation). In some implementations, the first mode may be reactivated when the user releases the selection input (e.g., the user terminates the finger pinch operation). In some implementations, the first mode may apply unless the second mode is triggered based on any type of pinch and hold (or any hand type) gesture that moves in the third z dimension relative to the xy dimension of the 2D plane.
[0005] In some implementations, the first mode and the second mode may be associated with an application configured to enable Z movement of the hand to dynamically adjust the size of the virtual structure. Likewise, the first mode and the second mode may be triggered in response to movement of the hand (e.g., rotating the hand or translating the hand in the Z dimension) to modify properties of the application.
[0006] In some implementations, interaction with virtual elements within the (3D) space of an extended reality (XR) environment can be enabled based on a cursor that moves relative to input from a virtual reality (VR) controller.
[0007] In some implementations, an electronic device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some implementations, an XR environment including virtual elements and a cursor is presented. In some implementations, hand data corresponding to 3D movement of a hand in a 3D environment is obtained. In some implementations, a first mode of operation is operated so that the 3D movement of the hand is converted into 2D movement. In some implementations, a 3D user input criterion is detected, and in response, the operating mode is modified to a second mode in which the 3D movement of the hand is maintained without being converted into 2D movement.
[0008] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] So that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be had with reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.
[0010] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.
[0011] FIG. 2A to FIG. 2I A diagram of an XR environment in which multimodal hand cursor movements are recognized as input is shown, according to some implementations.
[0012] Figure 3 is a flow chart illustrating an exemplary method that enables interaction with elements in a three-dimensional (3D) environment relative to multimodal hand cursor movement based on user hand motion tracking, according to some implementations.
[0013] Figure 4 is a flow chart illustrating an alternative exemplary method that enables interaction with elements in a three-dimensional (3D) environment relative to multimodal hand cursor movement based on user hand motion tracking according to some specific implementations.
[0014] Figure 5 is a block diagram of an electronic device according to some specific implementations.
[0015] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION
[0016] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.
[0017] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1B In the example of , the physical environment 100 is a room including a table 121. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein, as well as information about the user 102 of the electronic devices 105 and 110 and evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.
[0018] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 100 and a representation of the user 102 based on the camera image and / or the depth camera image of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system associated with the XR environment, which may correspond to a 3D coordinate system of the physical environment 100.
[0019] In some embodiments, the HMD (e.g., device 105) is configured to present an extended reality (XR) environment to the user. The XR environment can be a virtual reality (VR) environment or a mixed reality (MR) environment. The XR environment may include virtual elements / structures (e.g., a flat user interface (UI) located at a specified distance in front of the user and presented via the HMD) and a cursor (e.g., a hand cursor, a virtual cursor, etc.). The HMD may be further configured to obtain data corresponding to three-dimensional (3D) movement of (the user's) hand within the 3D environment. The data may be obtained via a sensor, which may include, in particular, an outward-facing image, a depth sensor, etc. included in the HMD. The sensor may be configured to capture an image of the user's hand and use the image to determine the 3D position of the hand and / or the configuration of the hand (e.g., a posture represented by a skeleton of the hand).
[0020] In some embodiments, the HMD (or any intermediate device) may be configured to determine whether to use the first mode or the second mode to provide cursor-based interaction (e.g., interaction with a virtual element such as a UI) in response to determining whether a specified user input criterion is met. The specified user input criterion may be a currently selected UI element of a specified type. For example, the specified user input criterion may be a user who is currently performing a finger pinch operation (e.g., thumb to index finger, thumb to middle finger, thumb to ring finger, thumb to pinky finger, any combination of thumb to fingers, etc.) when the cursor is currently located on a UI position control icon. Alternatively, the specified user input criterion may be a gesture type (a pinch and hold gesture) that moves in a third z dimension relative to the xy dimensions of a 2D plane presented within the XR environment via the HMD.
[0021] In some implementations, the first mode can enable detected 3D hand movements of a user to move a hand cursor and / or a virtual cursor on a surface of an object presented within the XR environment (e.g., on a 2D plane) in response to converting the 3D hand cursor movements into 2D hand cursor movements. For example, a hand cursor can move on or over a flat surface of a 2D user interface displayed within the 3D space of the XR environment.
[0022] In some implementations, the second mode enables a user's 3D hand motion to move a hand cursor in 3D space without converting the 3D hand cursor motion to a 2D hand cursor motion (e.g., moving the hand cursor and associated objects, such as the entire UI, in 3D space). The first mode may apply unless the second mode is triggered (e.g., based on the user positioning the hand cursor over a particular object type (e.g., a "3D drag and drop" or drag type object and / or UI 3D position control icon) and providing a selection input (e.g., a hold pinch gesture). When the user releases the selection input (e.g., stops pinching), the first mode may be reactivated. Alternatively, the first mode may apply unless the second mode is triggered based on any type of pinch and hold or any hand type gesture that moves in the third z dimension relative to the xy dimension of the 2D plane.
[0023] In some implementations, the aforementioned first mode and second mode may be triggered in response to an application (e.g., a drawing application) configured to enable the Z movement of the hand to dynamically adjust the size of a virtual drawing structure (e.g., a brush on a canvas). Similarly, the aforementioned first mode and second mode may be triggered in response to the movement of the hand (e.g., rotating the hand or translating the hand in the Z dimension) to modify the properties of the application (e.g., modifying the color of paint in a drawing application).
[0024] In some implementations, the aforementioned interactions with virtual elements and associated cursor modes can be enabled based on a cursor that moves relative to input from a virtual reality (VR) controller rather than a hand cursor.
[0025] FIG. 2A to FIG. 2I Shown are views 210A-210I of an XR environment provided by the devices 105 and / or 110 of Figure 1. The views 210A-210I include depictions 226 of a user's hands performing various hand motion-based user inputs (eg, gestures). FIG. 2A to FIG. 2I Each of the views 210A-210I includes an exemplary user interface 250 and a depiction 221 of the table 121. The depiction 221 of the table 121 and other aspects of the physical environment 100 may be viewed as a pass-through video, or may be a direct view of a physical object through a transparent or translucent display. The user interface 250 may be any type of virtual element positioned at a 3D location within the XR environment, and may be capable of being moved to another 3D location and / or redirected within the XR environment. The user interface 250 may be approximately two-dimensional / flat, or may be a 3D object having a surface (e.g., flat, mostly flat, curved, etc.) on which 2D interactions are enabled. Providing views 210A-210I may involve determining 3D properties of the physical environment 100 (e.g., of FIG. 1 ) and positioning virtual content (e.g., the user interface 250) in a 3D coordinate system corresponding to the physical environment 100.
[0026] in addition, FIG. 2A to FIG. 2I The hand movement gestures 225a-225i (eg, as shown in FIG. 1 ) of the hand (including fingers) 226 of the user 102 of FIG. 1 Figure 2A and Figure 2E The pointer shown in FIG. 2B to FIG. 2D , FIG. 2F to FIG. 2I The fingers shown in FIG. 1 are brought together and touched to form various combinations of representations of a pinch and hold gesture (such as fingers, etc.).
[0027] exist FIG. 2A to FIG. 2I In the example of FIG. 1 , user interface 250 includes various user interface (visual) elements 252 (e.g., applications or icons representing software applications and / or user interfaces), elements 253 (such as, in particular, photos (i.e., FIG. 2F to FIG. 2I )), and optional user interface 3D position control icons 235 (which allow hand movements and gestures to select the user interface to move in 3D (e.g., in X, Y, and Z degrees of motion 240) to different locations within the XR environment, as described below).
[0028] exist FIG. 2A to FIG. 2I In the example of , an indicator 228 (e.g., a pointer, cursor, etc.) may be provided to indicate a point of interaction with any of the user interface elements 252, elements 253 (e.g., a document or photo), or user interface 250. For example, the indicator 228 may indicate interaction with the user interface 3D position control icon 235 to allow hand movement gestures 225a-225i to move the user interface 250 itself to different 3D positions and orientations within the XR environment (e.g., to perform a “3D drag and drop operation” with respect to moving the user interface 250 to different positions within the XR environment).
[0029] The following procedures represent examples associated with the following operations (via FIG. 2A to FIG. 2I210A-210I of ): a drag or drop operation (of user interface 250) is initiated using hand cursor mode with respect to X, Y, and Z motions of hand 226 (represented by hand movement gestures 225a-225i) for a duration of the drag or drop operation, thereby allowing user interface 250 to travel in full 3D space (of the XR environment) during the drag or drop operation. The process begins when the user enables hand cursor mode. In response, 3D hand cursor motion is converted to 2D hand cursor motion. Subsequently, the user moves a cursor (e.g., indicator 228) over a draggable object (such as user interface 250 and / or element 253) and performs a pinch gesture to select the draggable object located under the cursor. When the user begins a drag operation, the 2D hand cursor motion is converted to a full 3D hand cursor motion, thereby allowing the user interface 250 to be moved in full 3D (according to the full 3D (6DOF) hand motion) within the XR environment (i.e., providing a corresponding 6DOF change to the position and orientation of the UI 250). When the user releases the pinch gesture, the user interface 250 is dropped at the designated location in the XR environment, and the 3D hand motion is converted back to a 2D hand motion. During the drag or drop operation, the indicator 228 may not be visible to the user, but may be visible after the drag and drop operation is completed.
[0030] exist FIG. 2A to FIG. 2I In the example of , user interface 250 is simplified for illustrative purposes, and the user interface may actually include any level of complexity, any number of different applications and / or computer-based interactive items, and / or a combination of 2D and / or 3D content. User interface 250 may be provided by various types of operating systems and / or applications (user interface elements), including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, photos, or any other application that can display, present, or otherwise use visual and / or audio content.
[0031] FIG. 2A to FIG. 2I Each of shows the user's hand movement (movement of hand 226, represented by hand movement gestures 225a-225i) at different times.
[0032] exist Figure 2A210A, at a first moment corresponding to view 210A, hand cursor mode is enabled (via a first mode), and the motion of hand move gesture 225a causes corresponding motion (i.e., visible or invisible) of indicator 228. The motion of hand move gesture 225a is converted from 3D motion (e.g., X, Y, and Z degrees of motion 240) to 2D motion (X and Y degrees of motion 241) for planar motion relative to X, Y motion (on the surface of a virtual object such as UI 250) without allowing Z motion (e.g., motion that moves above or below the plane of UI 250). As a result of the conversion, 3D hand move gestures 225a-225a' produce an associated motion of indicator 228 on the 2D surface of user interface 250 in a direction from position 228a to position 228b.
[0033] For example, Figure 2A 2D motion on user interface 250 (after 3D to 2D conversion) is shown such that hand movement gesture 225a causes an associated movement of indicator 228 (shown as 228a, which represents indicator 228 at a first moment in time) from a first position on UI 250 to a second position on UI 250 (i.e., indicator 228 is shown as 228b, which represents indicator 228 at a second moment in time at a position located over one of user interface elements 252). Subsequently, selection of user interface element 225 is enabled in response to the user providing a selection input (i.e., a pinch gesture via hand movement gesture 252a') while indicator 228 (shown as 228b, which represents indicator 228 at a second moment in time) is located over one of user interface elements 252, thereby launching an application associated with one of user interface elements 252. Before the indicator 228 is moved onto the UI 250, the hand cursor motion (i.e., the motion of the hand relative to the motion of the indicator 228) is converted from 3D motion (e.g., X, Y, and Z degrees of motion 240) to 2D (e.g., X and Y degrees of motion 241) motion, thereby allowing planar movement (over the UI 250 and subsequently over one of the user interface elements 252) relative to the X, Y motion (on the surface of the UI 250 and the user interface elements 252) without allowing Z motion (e.g., motion that moves above or below the plane of the UI 250). The 2D motion that allows planar movement allows the user to control the movement of the indicator 228 on the surface of the UI 250.
[0034] exist Figure 2B210a, during a period corresponding to view 210B, hand movement gesture 225b causes an associated movement of indicator 228 such that indicator 228 is located above user interface 3D position control icon 235 (within or associated with user interface 250). Hand movement gesture 225b may include any type of hand gesture, including, among other things, a pinch gesture, movement of a hand in 3D space, and the like. User interface 3D position control icon 235 may be a UI element that can be used in another mode of hand-based input (e.g., 3D input mode). Such a mode may enable hand-based input (e.g., hand movement gesture 225b) to switch modes and select user interface 250 for moving to a different 3D position within the XR environment depicted in view 210a. Alternatively, indicator 228 may be positioned directly above another portion of the user interface and other inputs that are recognized as enabling 3D input mode, for example, for selecting user interface 250 to move user interface 250 to a different position within the XR environment.
[0035] In some implementations, a transition to another mode (e.g., 3D mode) based on hand-based input is initiated based on a selection operation on the user interface 3D position control icon 235. The selection operation may be enabled via a hand movement gesture 225b that performs a pinch and hold operation. Alternatively, the selection operation may be enabled via a hand movement gesture 225b that performs any type of gesture (e.g., open palm facing up, fist, thumb or any finger pointing up, waving, etc.). Similarly, the selection operation may be enabled via a combined hand movement gesture 225b and / or an additional hand movement gesture performed by another hand. Similarly, the selection operation may be enabled via a combined hand movement gesture 225b and / or a gaze-based gesture (implemented via an eye tracking process). In response to the selection operation, another mode based on hand input (e.g., 3D mode) is initiated, and the 3D movement of the hand movement gesture 225b causes the indicator 228 to move and produce corresponding 3D movement of the indicator 228 and the entire user interface 250 in 3D, for example, changes relative to 6DOF based on the 6DOF movement of the hand.
[0036] exist Figure 2C 2, at a third moment in time corresponding to view 210C, the 3D movement (e.g., relative to X, Y, and Z degrees of motion 240) of hand move gesture 225c, together with indicator 228, has moved user interface 250 in all three dimensions (e.g., relative to X, Y, and Z degrees of motion 240) to a different location (relative to XR environment) in the XR environment. Figure 2B2D). The hand movement gesture 225 c (during the pinch and hold operation), the indicator 228, and the movement of the user interface 250 have been performed in all directions (such as X, Y, and Z) relative to full 3D motion (within the XR environment 210) (i.e., via the 3D hand cursor mode).
[0037] Alternatively, hand movement gestures, indicators, and movement of the user interface may be performed in all directions (such as X, Y, and Z) (e.g., 6DOF) relative to full 3D motion (within the XR environment) in response to a twisting motion (made by the hand movement gesture).
[0038] exist Figure 2D , at a fourth moment corresponding to view 210D, hand movement gesture 225d together with indicator 228 has moved user interface 250 to a different position and orientation (relative to Figure 2C 225d (during the pinch operation), movement of indicator 228, and user interface 250 have been performed (i.e., via 3D hand cursor mode) in all directions (such as X, Y, and Z) (e.g., 6DOF) relative to full 3D movement (within XR environment 210) in response to the twisting motion (made by hand move gesture 225c). User interface 250 is shown tilted inward in the Z direction.
[0039] exist Figure 2E , at a fifth moment corresponding to view 210E, hand move gesture 225e has disabled the selection operation by releasing the pinch operation, thereby transitioning the hand cursor (i.e., hand move gesture 225b and / or indicator 228) back to 2D input mode for interacting with user interface 250 (e.g., performing a drop operation). In response, the 3D motion of hand move gesture 225e is converted to 2D motion for planar movement relative to X, Y motion without allowing any type of Z motion. Subsequently, hand move gesture 225e moves together with indicator 228 in a direction away from user interface 250 to optionally perform operations with respect to additional objects within the XR environment.
[0040] exist Figure 2F , at a sixth moment corresponding to view 210F, hand movement gesture 225f (relative to 2D input mode (e.g., relative to X and Y degrees of motion 241) to interact with user interface 250) causes associated movement of indicator 228 such that indicator 228 is located over user interface (visual) element 252a (e.g., messaging application), while hand movement gesture 225f performs a pinch and hold gesture. The pinch and hold gesture causes user interface (visual) element 252a to open as application UI 252b (e.g., messaging application UI).
[0041] exist Figure 2G , at a seventh moment corresponding to view 210G, hand movement gesture 225g (relative to 2D input mode (e.g., relative to X and Y degrees of motion 241) to interact with user interface 250) causes an associated movement of indicator 228 such that indicator 228 is located over element 253 (e.g., a photo), while hand movement gesture 225g performs a pinch and hold gesture. The pinch and hold gesture causes element 253 to be selected for the drag and drop operation.
[0042] exist Figure 2H , at a seventh moment corresponding to view 210h, hand movement gesture 225h (relative to a 3D input mode (e.g., relative to X, Y, and Z degrees of motion 240) for interacting within an XR environment provided within view 210h) causes an associated movement of indicator 228 (during a pinch and hold gesture) such that element 253 has been moved (e.g., dragged) from user interface 250 to the XR environment provided within view 210H (i.e., toward application UI 252b (e.g., a messaging application UI)).
[0043] exist Fig.2I , at an eighth time instant corresponding to view 210I, hand move gesture 225i (relative to 2D input mode (e.g., relative to X and Y degrees of motion 241) to interact with application UI 252b) causes associated movement of indicator 228 such that element 253 has been moved (e.g., dragged) from user interface 250 (in 2D) across the XR environment (in 3D) to application UI 252b (e.g., messaging application UI). When the pinch and hold operation terminates, element 253 is dropped within application UI 252b such that hand move gesture 252i, along with indicator 228, can be moved in a direction away from application UI 252b to optionally perform operations with respect to additional objects within the XR environment.
[0044] Figure 3300 is a flowchart representation of an exemplary method 300 that enables interaction with elements in a three-dimensional (3D) environment relative to multimodal hand cursor movement based on user hand motion tracking. In some implementations, method 300 is performed by a device (such as a mobile device, a desktop computer, a laptop computer, an HMD, or a server device). In some implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as, for example, device 105 of FIG. 1). In some implementations, method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the boxes in method 300 may be enabled and executed in any order.
[0045] At block 302, method 300 presents (via an electronic device, such as, among other things, an HMD) an extended reality (XR) environment including a virtual element and a cursor. For example, the virtual element may be a flat user interface positioned one or two meters in front of the user.
[0046] At block 304, method 300 obtains hand data corresponding to three-dimensional (3D) movement of a hand in a 3D environment. For example, an externally facing image and / or depth sensor on the HMD may capture an image of the hand and determine a 3D position of the hand and / or a configuration of the hand (e.g., a pose of a skeletal representation of the hand).
[0047] At block 306, method 300 operates the electronic device in a first mode such that 3D motion of the cursor is converted to 2D motion. For example, in the first mode, cursor movement of the cursor is limited to 2D motion.
[0048] At block 308, method 300 detects 3D user input criteria. The 3D user input criteria may enable selection of a particular type of UI element. In some implementations, the 3D user input criteria may determine that a cursor is over an object of the object type when providing a selection input. (e.g., a cursor over a UI position control icon). In some implementations, the 3D user input criteria includes performing a user gesture and moving a hand in the z direction. For example, the user gesture may include a pinch gesture.
[0049] At block 310 , the method 300 modifies the operating mode to a second mode (in response to the 3D user input criteria) such that the 3D movement of the cursor is maintained and not converted to 2D movement.
[0050] In some implementations, the second mode is provided while the user gesture is maintained. For example, the second mode can be provided while the pinch gesture continues.
[0051] In some implementations, both the cursor and the virtual object are moved in 3D space based on the 3D movement of the hand while maintaining the user gesture.
[0052] In some implementations, the first mode is provided based on determining that the user gesture has ceased.
[0053] In some implementations, 3D movement of virtual elements is enabled via the second mode.
[0054] In some implementations, the first mode enables a cursor to be moved on the surface of the virtual element, and the second mode enables the virtual element to be moved within the XR environment while maintaining the position of the cursor on the surface of the virtual element.
[0055] In some implementations, the virtual object is moved based on the 3D movement of the hand while maintaining the user gesture.
[0056] Figure 4 4 is a flowchart representation of an exemplary method 400 according to some specific implementations, which enables interaction with elements in a three-dimensional (3D) environment relative to multimodal hand cursor movement based on user hand motion tracking. In some specific implementations, method 400 is performed by a device such as a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some specific implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as, for example, device 105 of Figure 1). In some specific implementations, method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the boxes in method 400 can be enabled and executed in any order.
[0057] At block 402, method 400 presents (via an electronic device, such as, in particular, an HMD) an extended reality (XR) environment including a virtual element and a cursor. For example, the virtual element may be a flat user interface positioned one or two meters in front of the user.
[0058] At block 404, method 400 obtains hand data corresponding to three-dimensional (3D) movement of a hand in a 3D environment. For example, an externally facing image and / or depth sensor on an HMD may capture an image of the hand and determine a 3D position of the hand and / or a configuration of the hand (e.g., a pose of a skeletal representation of the hand).
[0059] At block 406, method 400 determines whether to use the first mode or the second mode to provide cursor-based interaction based on determining whether a criterion is met. For example, the criterion may include whether a particular type of UI element is currently selected, such as, for example, when the cursor is over a UI position control icon, the user is currently pinching.
[0060] In some implementations, based on determining to use the first mode to provide cursor-based interaction, the cursor is moved on a surface of a virtual element in the XR environment based on determining two-dimensional (2D) movements corresponding to 3D movements of the hand.
[0061] In some implementations, based on determining to use the second mode to provide cursor-based interaction, the cursor is moved in 3D in the XR environment based on the 3D movement of the hand. For example, the cursor can be linked to a virtual element so that the virtual element moves with the cursor in 6DOF, so that the UI moves forward, backward, left, right, up, down, twist, tilt, etc. based on the 3D movement of the hand.
[0062] In some implementations, based on the determination, a first mode is used to provide cursor-based interaction, and cursor movement is limited to 2D movement.
[0063] In some implementations, determining whether the criteria are met includes determining whether a particular type of UI element is currently selected.
[0064] In some implementations, determining whether a particular type of UI element is currently selected includes determining whether a cursor is over an object of the object type when the selection input is provided.
[0065] In some implementations, method 400 determines to use the second mode to provide cursor-based interaction in response to a user gesture (eg, pinching) to select an object.
[0066] In some implementations, method 400 determines to use a second mode to provide cursor-based interaction while the user gesture is maintained (eg, while the pinch continues).
[0067] In some implementations, both the cursor and the virtual object are moved based on the 3D movement of the hand while maintaining the user gesture.
[0068] In some implementations, based on determining that the user gesture has ceased, a determination is made to use the first mode to provide cursor-based interaction.
[0069] In some implementations, 3D movement of the virtual element is enabled based on determining to use the second mode to provide cursor-based interaction.
[0070] In some implementations, the first mode causes the cursor to move on the surface of the virtual element, and the second mode causes the virtual element to move within the XR environment while maintaining the position of the cursor on the surface of the virtual element.
[0071] Figure 5 1 is a block diagram of an exemplary device 500. Device 500 illustrates an exemplary device configuration for electronic devices 105 and 110 of FIG1. Although some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 500 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 506, one or more communication interfaces 508 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, output devices (e.g., one or more displays) 512, one or more internal-facing and / or external-facing image sensor systems 514, memory 520, and one or more communication buses 504 for interconnecting these and various other components.
[0072] In some implementations, the one or more communication buses 504 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), one or more cameras (e.g., an internally facing camera and an externally facing camera of the HMD), one or more infrared sensors, one or more thermal map sensors, etc.
[0073] In some implementations, one or more displays 512 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc. to the user. In some implementations, one or more displays 512 are configured to present content (determined based on the determined user / object position of the user within the physical environment) to the user. In some implementations, one or more displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more displays 512 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 500 includes a single display. As another example, the device 500 includes a display for each eye of the user.
[0074] In some implementations, the one or more image sensor systems 514 are configured to obtain image data corresponding to at least a portion of the physical environment 100. For example, the one or more image sensor systems 514 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 514 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 514 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.
[0075] In some implementations, the device 500 includes an eye tracking system for detecting eye position and eye movement (e.g., eye gaze detection). For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near infrared (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) to the user's eyes. In addition, the illumination source of the device 500 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on the near-eye display of the device 500.
[0076] The memory 520 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 520 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 520 optionally includes one or more storage devices remotely located from the one or more processing units 502. The memory 520 includes a non-transitory computer-readable storage medium.
[0077] In some implementations, the memory 520 or a non-transitory computer-readable storage medium of the memory 520 stores an optional operating system 530 and one or more instruction sets 540. The operating system 530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 540 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 540 is software that can be executed by one or more processing units 502 to implement one or more of the techniques described herein.
[0078] The instruction set 540 includes a first mode operation instruction set 542, a standard detection instruction set 544, and a second instruction set 546. The instruction set 540 may be embodied as a single software executable file or multiple software executable files.
[0079] The first mode operation instruction set 542 is configured with instructions executable by the processor to enable a first hand-cursor mode for converting 3D motion of the hand and cursor into 2D motion.
[0080] Criteria detection instruction set 544 is configured with instructions executable by the processor to detect 3D user input criteria, such as a pinch gesture, when moving the hand in the z-direction.
[0081] The second mode operating instruction set 546 is configured with instructions executable by the processor to modify the operating mode to a second mode (in response to the 3D user input criteria) such that 3D movement of the hand is maintained and not converted to 2D movement.
[0082] Although the instruction set 540 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 5Rather, it serves as a functional description of various features present in a particular implementation that are different from the structural schematics of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.
[0083] Returning to Figure 1, the physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).
[0084] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.
[0085] Those of ordinary skill in the art will appreciate that well-known systems, methods, components, devices, and circuits are not described in detail to avoid obscuring more relevant aspects of the example implementations described herein. In addition, other effective aspects and / or variations do not include all of the specific details described herein. Therefore, several details are described in order to provide a thorough understanding of the example aspects shown in the accompanying drawings. In addition, the accompanying drawings only illustrate some exemplary embodiments of the present disclosure and should not be considered as limiting.
[0086] Although this specification includes many specific implementation details, these specific implementation details should not be understood as limitations on the scope of any invention or possible content to be protected, but should be understood as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in the form of any suitable sub-combination. In addition, although certain features may be described above as acting in certain combinations and even initially claimed in this way, one or more features of the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a variation of the sub-combination.
[0087] Similarly, although the operations are shown in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in a sequential order or in the specific order shown, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned embodiment should not be understood as requiring such division in all embodiments, and it should be understood that the program components and the system can be generally integrated together in a single software product or encapsulated in multiple software products.
[0088] Thus, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired results. In addition, the processes shown in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some specific implementations, multitasking and parallel processing may be advantageous.
[0089] The embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium, for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).
[0090] The term "data processing device" covers all kinds of devices, equipment and machines for processing data, such as programmable processors, computers, systems on chip, or multiple or combinations of the foregoing. The device may include a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). In addition to hardware, the device may also include a code for creating an execution environment for the computer program under consideration, such as a code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more of them. The device and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout the specification, discussions using terms such as "processing", "calculating", "calculating", "determining", and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data represented as physical electronic quantities or magnetic quantities in the memory, registers, or other information storage devices, transmission devices, or display devices of the computing platform.
[0091] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages may be used to implement the teachings contained herein in software for programming or configuring a computing device.
[0092] The specific implementation of the method disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.
[0093] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values may in practice be based on additional conditions or values beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.
[0094] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0095] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.
[0096] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
Claims
1. A method, comprising: At an electronic device having a processor and a display: Rendering an extended reality (XR) environment including virtual elements and cursors; obtaining hand data corresponding to 3D motion of the hand in a three-dimensional (3D) environment; operating in a first mode in which 3D movement of the cursor is converted into 2D movement; Detect 3D user input criteria; as well as In response to the 3D user input criteria, the operating mode is modified to a second mode in which the 3D movement of the cursor is maintained without conversion to the 2D movement.
2. The method according to claim 1, wherein: Based on the operation in the first mode, cursor movement of the cursor is restricted to 2D motion. The method of claim 1 , wherein the 3D user input criteria enables selection of a specific type of UI element. 4 . The method of claim 3 , wherein the 3D user input criteria determines that the cursor is over an object having an object type when a selection input is provided. The method of claim 1 , wherein the 3D user input criteria comprises performing a user gesture and moving the hand in a z-direction. The method of claim 5 , wherein the method determines to provide the second mode while maintaining the user gesture. The method of claim 5 , wherein the user gesture comprises a pinch gesture.
8. The method of claim 5, further comprising moving both the cursor and the virtual object based on the 3D motion of the hand while maintaining the user gesture. 9 . The method of claim 5 , further comprising determining to provide the first mode based on determining that the user gesture has ceased. 10 . The method of claim 1 , further comprising using the second mode to implement 3D movement of the virtual element.
11. The method according to claim 1, wherein: The first mode causes the cursor to move on the surface of the virtual element; and The second mode moves the virtual element within the XR environment while maintaining the position of the cursor on the surface of the virtual element. 12 . The method of claim 1 , further comprising moving the virtual object based on the 3D motion of the hand while maintaining the user gesture.
13. An electronic device, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Rendering an extended reality (XR) environment including virtual elements and cursors; Obtaining hand data corresponding to 3D movement of the hand in a three-dimensional (3D) environment; and operating in a first mode in which the 3D movement of the cursor is converted into 2D movement; detecting 3D user input criteria; and In response to the 3D user input criteria, the operating mode is modified to a second mode in which the 3D movement of the cursor is maintained without conversion to the 2D movement.
14. The electronic device according to claim 13, wherein: Based on the operation in the first mode, cursor movement of the cursor is restricted to 2D motion.
15. The electronic device of claim 13, wherein the 3D user input criteria enables selection of a particular type of UI element. 16 . The electronic device of claim 15 , wherein the 3D user input criteria determines that the cursor is over an object having an object type when a selection input is provided. 17 . The electronic device of claim 13 , wherein the 3D user input criteria comprises performing a user gesture and moving the hand in a z-direction.
18. The electronic device of claim 17, wherein the method determines to provide the second mode while maintaining the user gesture.
19. The electronic device of claim 17, wherein the user gesture comprises a pinch gesture.
20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors of an electronic device to perform operations including: Rendering an extended reality (XR) environment including virtual elements and cursors; Obtaining hand data corresponding to 3D movement of the hand in a three-dimensional (3D) environment; and operating in a first mode in which the 3D movement of the cursor is converted into 2D movement; Detect 3D user input criteria; as well as In response to the 3D user input criteria, the operating mode is modified to a second mode in which the 3D movement of the cursor is maintained without conversion to the 2D movement.
