Input recognition based on distinguishing direct and indirect user interactions

Through the combination of sensor data and gaze direction, distinguishing direct and indirect interaction modes is solved, and the problem of user interaction systems in the prior art is difficult to distinguish input modes, achieving a more flexible and accurate user interaction experience.

CN119065576APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410699595.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-09-29
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

There is room for improvement in existing user interaction systems in terms of user activities-based interactions, especially when evaluating and interpreting user interactions with virtual elements, it is difficult to effectively distinguish direct and indirect input modes, resulting in inflexible and accurate interaction experience.

Method used

By using sensor data (such as image, depth sensor data, motion data) to determine the position of the user's hand and the position of the virtual object, combining the gaze direction and head posture, selecting the applicable interaction mode, and using direct or indirect interaction recognition process to explain User activity, providing a combined solution of direct and indirect input modalities.

Benefits of technology

It realizes a more flexible and accurate user interaction experience, and can automatically switch interaction modes according to user's intentions, thereby improving the functionality, accuracy and efficiency of interaction and providing a more intuitive user interface.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065576A_ABST
    Figure CN119065576A_ABST
Patent Text Reader

Abstract

An input recognition based on distinguishing between direct and indirect user interactions is provided. Various implementations disclosed herein include devices, systems, and methods for interpreting user activity when a user interacts with a user interface (UI) element positioned within a three-dimensional (3D) space, such as an augmented reality (XR) environment. Some implementations enable a user to interact with virtual elements displayed in a 3D environment utilizing alternative input modalities, such as an XR environment that interprets user activity as direct or indirect interaction with virtual elements.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 470,565, filed on June 2, 2023, and U.S. Provisional Application Serial No. 63 / 521,807, filed on June 19, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to evaluating user interactions with electronic devices involving hand and body gestures, gaze, voice, and / or other user activities. Background Art

[0004] Existing user interaction systems can be improved in facilitating interactions based on user activities. Summary of the invention

[0005] Various embodiments disclosed herein include devices, systems, and methods for interpreting user activities when the user interacts with virtual elements (e.g., user interface elements) positioned in a three-dimensional (3D) space, such as an extended reality (XR) environment. Some embodiments enable a user to interact with virtual elements displayed in a 3D environment that utilizes an alternative input modality, such as an XR environment that interprets user activities as direct interactions or indirect interactions with virtual elements. Whether user activities should be interpreted using one of multiple input modality schemes (e.g., using a direct interpretation process or an indirect interpretation process) may be determined based on sensor data (e.g., images, depth sensor data, motion data of a device held or worn by a user, etc.) and / or using one or more modality distinction criteria.

[0006] In some embodiments, the XR environment may enable a user to provide input using both (a) a direct input modality, in which the user interacts with virtual content by virtually touching the virtual content (and / or getting close to the virtual content as determined via one or more proximity criteria), and (b) an indirect input modality, in which the user interacts with the virtual content using gaze, pointing gestures, an input device, etc. to interact with the content without having to virtually touch or get close to the virtual content.

[0007] In some implementations, for direct interaction, a virtual element (e.g., a user interface (UI) element) is identified based on the intersection of a user's hand position with a 3D region associated with the virtual element (or proximity to the 3D region as determined via one or more proximity criteria). Based on determining that a direct input modality is applicable, a direct interaction recognition process is applied to interpret the user activity, e.g., determining whether the gesture / hand motion is a tap, swipe, etc. During such user activity, a hand (or other body part) virtually interacts with the virtual element by virtually contacting the UI element. For example, a user may move their fingers to directly tap, pinch, swipe, or otherwise interact with a UI element in 3D space.

[0008] In some embodiments, for indirect interaction, the virtual element that is the subject of the user interaction is identified in another manner (e.g., based on the user's gaze direction and associated gestures). Based on determining that the indirect input modality is applicable, an indirect interaction recognition process is applied to interpret the user activity (e.g., gaze direction and associated hand pinching) as initiating or performing an action on the virtual element identified by the gaze. For example, the user activity can be indirect in the sense that the hand does not need to be in contact with or close to the UI element in 3D space. For example, when the UI is displayed at a certain distance in 3D space relative to the user's viewpoint (e.g., the user's position in 3D space), the gesture can be performed by the user's hand when the user's hand is resting on the side of the user or otherwise close to the user.

[0009] In some embodiments, the processor performs the method by executing instructions stored on a computer-readable medium. The method obtains the position of a virtual object in an XR environment corresponding to a 3D space. The virtual object can be an element of a UI. For example, a user interface may include one or more user interface elements, such as buttons, icons, text input fields, sliders, or menu items, which are presented as part of a virtual user interface displayed at a fixed position or otherwise within the XR environment, for example, a virtual two-dimensional (2D) menu with buttons displayed a few feet in front of the user in XR relative to the user's viewpoint.

[0010] The method includes obtaining a user hand position in the 3D space based on sensor data, wherein the hand position is associated with a gesture. In some specific implementations, the user hand position is tracked using sensor data (e.g., image sensor data, depth sensor data, motion sensor data, etc.), which is used to determine a hand configuration and / or a path of movement of the user's hand over a period of time. In one example, the user position data can be a skeletal representation of the user generated periodically (e.g., 30fps) based on an outward-facing image / depth sensor on a head-mounted device (HMD). Such user position data can provide a path of a portion of the skeletal representation of the user (e.g., a finger skeletal portion of the skeletal user representation). In other words, for example, by tracking the orientation of the skeleton at multiple moments, the path of the finger portion of the skeleton relative to the 3D space over time can be determined and / or a configuration can be determined. The configuration can be defined by or based on the relative spatial positioning and / or orientation of the parts of the hand (e.g., fingers, palm, etc.) relative to each other, which can define a gesture such as pinching the thumb and index finger together.

[0011] The method includes determining an interaction mode based on a user hand position and a position of a virtual object in 3D space. Determining the interaction mode may involve selecting whether the interaction mode is direct or indirect based on the user hand position and the position of the virtual object in 3D space. For example, a direct interaction mode may be selected based on the intersection of a hand position and a 3D area associated with a virtual object in 3D space (e.g., a bounding box or other volume defined around the virtual object) or using a threshold distance relative to the position of the virtual object. Conversely, an indirect interaction mode may be selected based on the hand position not intersecting such a 3D area. Other criteria may be applied in addition or alternatively. For example, whether the interaction mode is direct or indirect may be based on: determining a focus area based on a user's gaze direction or head posture, determining an intersection of a hand position and a 3D area associated with a virtual object in 3D space, and / or determining whether the intersection is within or outside the focus area. Whether the interaction mode is direct or indirect may be determined based on whether the intersection of the hand position and the 3D area associated with the virtual object in the 3D space is within or outside the region of interest indicated by the user's gaze direction or head pose.

[0012] The method includes interpreting the gesture using an interaction recognition process associated with the determined interaction mode (eg, using an indirect interaction recognition process based on the determined indirect interaction mode or using a direct interaction recognition process based on the determined direct interaction mode).

[0013] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0015] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.

[0016] Figure 2 According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs direct interaction.

[0017] Figure 3 According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs indirect interactions.

[0018] Figure 4 An exemplary determination of a direct interaction mode based on a 3D area surrounding a virtual object is shown according to some implementations.

[0019] Figure 5 An exemplary determination of a direct interaction mode based on a 3D area around a virtual object and a region of interest is shown according to some implementations.

[0020] Figure 6 Another exemplary determination of a direct interaction mode based on a 3D area around a virtual object and a region of interest is shown according to some implementations.

[0021] Figure 7 An exemplary determination of a direct interaction mode based on a time threshold according to some implementations is shown.

[0022] FIG. 8A to FIG. 8B An exemplary determination of an indirect interaction mode according to some implementations is shown.

[0023] Fig. 9 Indirect interaction identification according to some implementations is shown.

[0024] Fig.10 A transition from a direct interaction mode to an indirect interaction mode is shown according to some implementations.

[0025] FIG. 11A to FIG. 11E Responses to direct interactions and indirect interactions are shown according to some implementations.

[0026] FIG. 12A to FIG. 12D Responses to direct interactions and indirect interactions are shown according to some implementations.

[0027] Fig.13 is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data according to some implementations.

[0028] Fig.14 is a flow chart illustrating a method for interpreting user activities as user interactions with virtual elements positioned within a 3D space, according to some implementations.

[0029] Fig.15 is a block diagram of an electronic device according to some specific implementations.

[0030] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0031] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0032] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1BIn the example of , the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein (e.g., images, sounds, lighting, characteristics, etc.) and information about the user 102 of the electronic devices 105 and 110 and evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 (e.g., including the location of objects such as the table 120 in the physical environment 100) and / or the location of the user within the physical environment 100.

[0033] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 100 and a representation of the user 102 based on the camera image and / or the depth camera image of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 100.

[0034] In some implementations, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 105 or device 110). In some implementations, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment may be generated based on, for example, the scale of the physical environment or the positioning of an open space, a floor, a wall, etc., such that the 3D representation is configured to align with a corresponding feature of the physical environment. In some implementations, a viewpoint (e.g., of user 102) within a 3D coordinate system may be determined based on a position of the electronic device within the physical environment. The viewpoint may be determined based on, for example, image data, depth sensor data, motion sensor data, etc., which may be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0035] Figure 2 Shows the Figure 1A to Figure 1B102 is a view of virtual elements within a 3D physical environment of an XR device, wherein a user performs direct interaction. In this example, user 102 makes gestures relative to content presented in views 210a-210b of an XR environment provided by a device (e.g., device 105 or device 110 above). Views 210a-210b of the XR environment include an exemplary user interface 230 of an application (e.g., an example of virtual content) and a drawing 220 of table 120 (i.e., an example of real content). As an example, in Figure 2 In the embodiment of the present invention, the user interface 230 is a two-dimensional virtual object (e.g., having a flat front-facing surface). Providing such a view may involve determining the 3D properties of the physical environment 100 described above (e.g., the position of the table 120 in the physical environment 100, the size of the table 120, the size of the physical environment 100, etc.) and positioning the virtual content (e.g., the user interface 230) in a 3D coordinate system corresponding to the physical environment 100.

[0036] exist Figure 2 In the example of , user interface 230 includes various content items, including background portion 235 and icons 242, 244, 246, 248. Icons 242, 244, 246, 248 can be displayed on flat user interface 230. User interface 230 can be a user interface of an application, as shown in this example. For illustrative purposes, user interface 230 is simplified, and the user interface can actually include any complexity, any number of content items and / or a combination of 2D and / or 3D content. User interface 230 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0037] In this example, background portion 235 of user interface 230 is flat. In this example, background portion 235 includes all aspects (e.g., visual characteristics) of user interface 230 being displayed except icons 242, 244, 246, 248. Displaying the background portion of a user interface of an operating system or application as a flat surface may provide various advantages. For example, doing so may provide a user interface that is easy to understand or otherwise easy to use to access an application using a portion of an XR environment. In some implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within an XR environment using one or more flat background portions, although other configurations are possible.

[0038] In some implementations, the position and / or orientation of such one or more user interfaces may be determined to facilitate visibility and / or use. Within the 3D environment, one or more user interfaces may be in a fixed position and orientation. In such cases, user movement will not affect the position or orientation of the user interface within the 3D environment (e.g., such that the user interfaces remain at their respective positions or orientations and do not move relative to the user's viewpoint).

[0039] The position of the user interface within the 3D environment may be based on determining the distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on various criteria, including, but not limited to, criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, environment lighting, the presence of other people in the environment, use of the application or content by multiple users, user preferences, user input, and other factors.

[0040] In some implementations, one or more user interfaces may be body-locked content, e.g., at a distance and orientation offset relative to a portion of a user's body (e.g., their torso). For example, body-locked content of a user interface may be 0.5 meters to the left of and 45 degrees to the forward-facing vector of the user's torso. If the user's head turns and the torso remains stationary, the body-locked user interface will appear to remain stationary at 0.5 meters to the left of and 45 degrees to the forward-facing vector of the torso in the 3D environment. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will rotate with the torso and reposition within the 3D environment so that it remains 0.5 meters to the left of and 45 degrees to the new forward-facing vector of their torso.

[0041] In other implementations, user interface content is defined at a specific distance from the user, where it remains fixed relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, a body-locked user interface may be defined as 2m away and may be initially displayed to the north of the user based on the direction the user is currently facing. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2m away from the north of the user, with north now directly behind the user.

[0042] The body-locked user interface may also be configured to always remain aligned with the gravity line or horizon, so that head and / or body changes in roll orientation will not cause the body-locked user interface to move within the 3D environment. On the other hand, translational movement will cause the body-locked content to be repositioned within the 3D environment so as to maintain the distance offset.

[0043] exist Figure 2 In the example of , the user 102 moves their hand from an initial position, as shown by the position of the drawing 222 of the hand in view 210a. The hand moves along a path 250 to a later position, as shown by the position of the drawing 222 in view 210b. As the user 102 moves their hand along this path 250, the finger intersects the user interface 230. Specifically, as the finger moves along the path 250, it virtually pierces the icon 246 in the user interface 230 and thus, from the user's point of view, the fingertip portion of the finger (not shown) is optionally obscured by the user interface 230 in view 210b.

[0044] The embodiments disclosed herein explain the user's movement such as Figure 1A to Figure 1B The user 102 of the present invention moves their hand / finger along a path 250 relative to a user interface element such as an icon 246 in the user interface 230 to identify user input / interaction. The interpretation of user movement and other user activities can be based on the use of multiple potentially separate recognition processes corresponding to different input modalities to identify user intent. The use of multiple potentially separate recognition processes for different modalities can improve functionality, accuracy, efficiency, and / or provide other device-based or interaction-based benefits.

[0045] exist Figure 2 In an example of , recognizing input may involve determining that the gesture is a direct interaction and then using a direct input recognition process to recognize the gesture. For example, such a gesture may be interpreted as a tap input directed to icon 246. When making such a gesture, the user's actual movement relative to icon 246 may deviate from the ideal movement (e.g., a straight path passing through the center of the user interface element (e.g., icon 246) in a direction completely orthogonal to the plane of the user interface element). The actual path may be curved, jagged, or otherwise nonlinear, and may be angled rather than orthogonal to the plane of the user interface element. The path may have properties that make the gesture similar to other types of input gestures (e.g., swipe, drag, flick, etc.). For example, non-orthogonal motion may make the gesture similar to a swipe motion, in which the user provides input by piercing the user interface element and then moving in a direction along the plane of the user interface (e.g., horizontally or vertically).

[0046] Some embodiments disclosed herein determine that a direct interaction mode is applicable and, based on the direct interaction mode, utilize a direct interaction recognition process to distinguish or otherwise interpret user activity corresponding to direct input, such as identifying an intended user interaction based on whether and how a gesture path intersects one or more 3D regions of space. Such recognition processes may take into account actual human tendencies associated with direct interaction (e.g., a natural arching of the hand that occurs during an intended straight motion, a tendency to move based on a shoulder or other pivot position (e.g., an elbow), etc.), human perception issues (e.g., a user cannot see or know the exact position of virtual content relative to their hand), and / or other direct interaction specific issues.

[0047] Note that the user's movements in the real world (e.g., physical environment 100) correspond to movements within a 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to real-world objects including the user). Thus, the user moves his hand in the physical environment 100, for example, through empty space, but the hand (i.e., a drawing or representation of the hand) intersects and / or pierces the user interface 230 of the XR environment based on the physical environment. In this way, the user directly interacts virtually with the virtual content.

[0048] Figure 3 Shows the Figure 1A to Figure 1B 305 of the XR environment, wherein the user performs indirect interaction. In this example, user 102 performs a gesture while viewing content presented in view 305 of the XR environment provided by a device (e.g., device 105 or device 110) (e.g., when the user's 102 gaze is directed toward the content). View 305 of the XR environment includes Figure 2 An exemplary user interface 230 of FIG. Figure 3 In the example of , user 102 makes a pinch gesture with their hands as shown in drawing 222 while gazing at user interface icon 246 along gaze direction path 310. In this example, this user activity (e.g., a pinch gesture along with a gaze directed at a UI element) corresponds to a user intent to interact with user interface icon 246.

[0049] Implementations disclosed herein interpret user activity (such as user 102 forming a pinch gesture in conjunction with a gaze directed at a UI element) to identify a user / interaction. For example, such user activity may be interpreted as a tap input directed at icon 246, such as selecting icon 246 in user interface 230, which causes the device to activate icon 246 and optionally display content (e.g., a user interface) corresponding to icon 246. However, when performing such actions, the timing between the user's gaze direction and / or gesture and detecting the gaze at the target with which the user intended to associate the gesture may not be fully executed and / or timed.

[0050] Some embodiments disclosed herein determine that an indirect interaction mode is applicable. Based on the indirect interaction mode, an indirect interaction identification process is used to identify an expected user interaction based on the user activity. For example, the process can be based on a gesture path (e.g., similar to Figure 3 The intended user interaction is identified by determining whether and how the path 310 in the image (i) intersects one or more 3D regions in the space. Such an identification process may take into account actual human tendencies associated with indirect interactions (e.g., eye saccades, eye fixations, and other natural human gaze behaviors, hand arching movements, retractions that do not correspond to intended insertion directions, etc.), human perception issues (e.g., users cannot see or know the exact location of virtual content relative to their hands), and / or other indirect interaction specific issues.

[0051] Some implementations determine an interaction mode, e.g., a direct interaction mode or an indirect interaction mode, such that user behavior can be interpreted by a dedicated (or otherwise separate) recognition process for the appropriate interaction type, e.g., using a direct interaction recognition process for direct interactions and an indirect interaction recognition process for indirect interactions. Such dedicated (or otherwise separate) process utilization may be more efficient, more accurate, or provide other benefits relative to using a single recognition process configured to recognize multiple types of (e.g., both direct and indirect) interactions.

[0052] Figure 4 An exemplary determination of a direct interaction mode based on a 3D area 430 surrounding a user interface 400 is shown. In this example, a user 102 is using a device 105 (e.g., implemented as an HMD) to view and interact with an XR environment that includes the user interface 400. The 3D area 430 surrounding the user interface 400 is determined by the device 105. Note that in this example, the dotted lines indicating the boundaries of the 3D area 430 are for illustrative purposes and are not visible to the user 102. The 3D area 430 is used to determine whether a direct interaction mode or an indirect interaction mode is applicable at different points in time based on user activity. In this example, the interaction mode is determined based on whether a hand 422 of the user 102 intersects the 3D area 430. In this case, in Figure 5 In the example of FIG. 4 , the hand 422 does intersect the 3D area 430 (e.g., breaks through the boundary of the 3D area 430), and thus the interaction mode is determined to be a direct interaction mode. Thus, the user's activity (e.g., hand movement, etc.) is evaluated, for example, using a direct interaction recognition process (such as providing a selection of pointing to the user interface 400).

[0053] Figure 5 4 shows an exemplary determination of a direct interaction mode based on a 3D area 430 and a focus area 510 around a user interface 400. In this example, a user 102 is using a device 105 (e.g., implemented as an HMD) to view and interact with an XR environment including the user interface 400. The 3D area 430 around the user interface 400 is determined by the device 105, as described above with reference to FIG. Figure 4 Similarly discussed. Note that in this example, the dotted lines indicating the boundaries of the 3D area 430 are for illustrative purposes and are not visible to the user 102. The focus area 510 is also a 3D area (e.g., a conical area centered on a centerline corresponding to the user's head direction or the forward-facing direction of the device 105). The user's head direction can be determined based on an estimated posture (e.g., the position and / or orientation of the head using sensors on the device 105). For example, the head direction can be determined based on determining the position of a plane corresponding to the user's cheek and extending a vector from a position on the user's face in a direction orthogonal to the plane. In another example, the head direction can be determined in part based on the orientation of the HMD worn on the head, for example, based on device posture tracking technology to determine a vector directly referenced to the device rather than based on the user's face. The user device direction can be determined via sensor data (e.g., image data, depth data, motion data, VIO process, SLAM process, etc.). The region of interest 510 may be determined based on a field of view provided in the view provided by the device 105 (e.g., corresponding to a central portion of the field of view (e.g., central 30%, 50%, 70%, etc.)) such that the region of interest 510 has predetermined three-dimensional characteristics (e.g., a fixed volume relative to the centerline discussed above). In other specific implementations, the region of interest 510 corresponds to a field of view provided in the view provided by the device 105.

[0054] 3D area 430 and focus area 510 are used to determine at different points in time whether direct interaction mode or indirect interaction mode is applicable to detected user activity. In this example, the interaction mode is determined based on determining that hand 422 of user 102 intersects 3D area 430 at intersection 515 and whether intersection 515 is within the focus area. In this case, intersection 515 is also within focus area 510, and therefore, the interaction mode is determined to be direct interaction mode. Therefore, for example, the user's activities (e.g., hand movements, etc.) are evaluated using a direct interaction recognition process, as discussed previously herein. The interaction mode may or may not also be applicable with respect to potential interactions with other (e.g., nearby) user interface elements (such as second user interface 520). For example, in Figure 5 Because the hand 422 intersects with the 3D area 430 surrounding the user interface 400 (e.g., at the intersection 515) instead of the 3D area surrounding the second user interface 520, when the device 105 performs a selection operation pointing to the user interface 400, the device 105 does not perform a selection operation pointing to the second user interface 520 based on the user's activity (e.g., movement of the hand 422).

[0055] Figure 6 Another exemplary determination of a direct interaction mode based on a 3D area 620 and a focus area 610 around a virtual object (e.g., a second user interface 520) is shown. In this example, the user 102 is using the device 105 to view a video including Figure 5 400 and the XR environment of the second user interface 520 of the user interface 400 and the second user interface 520 and interact with it. The 3D area 620 around the virtual object (e.g., the second user interface 520) is determined by the device 105. It should be noted that in this example, the dotted line indicating the boundary of the 3D area 620 is for illustrative purposes and is not visible to the user 102. In some specific implementations, the 3D area 620 has one or more characteristics of the 3D area 520 discussed above. The focus area 610 is also a 3D area (e.g., a conical area centered on a center line corresponding to the user's head direction or the forward-facing direction of the device 105). In some specific implementations, the focus area 610 has one or more characteristics of the focus area 510 discussed above. The 3D area 620 and the focus area 610 are used to determine at different points in time whether the direct interaction mode or the indirect interaction mode is applicable to the detected user activity. In this example, the interaction mode is dedicated to the interaction with the second user interface 520 (and is therefore determined to be not applicable to the user interface 400).

[0056] Based on determining that the hand 422 of the user 102 is at an intersection with the 3D area 620 (e.g., similar to Figure 5The interaction mode for the second user interface 520 is determined by intersecting at the intersection point 515 in the image and whether the intersection point is within the focus area. Figure 6 , the intersection point is within the focus area 610, and therefore, the interaction mode is determined to be a direct interaction mode. Therefore, for example, the user's activities (eg, hand movements, etc.) are evaluated using a direct interaction recognition process, as similarly described above.

[0057] It should be noted that the intersection point may correspond to a single location (e.g., an x, y, z location), to 2D, or to a 3D area at which the intersection occurs. Whether the intersection point is considered to be within the region of interest may be based on whether there is any intersection, such as any point on the hand 422 intersecting with any point in the region of interest 610. In another example, whether the intersection point is considered to be within the region of interest is based on other criteria, such as whether at least a portion of a given size of the hand 422 is within the region of interest 610. In another example, a specific point on the hand (e.g., a point on or near the end of a specific finger) is used to determine whether the intersection point is considered to be within the region of interest, such as when the specific point on the hand 422 is within the region of interest 610.

[0058] Figure 7 An exemplary determination of a direct interaction mode based on a time threshold is shown. Figure 6 4. The direct interaction mode with the second user interface 520 is determined to remain active for a period of time (e.g., 0.1, 1, 5, 10, 15, 30, etc. seconds) after the determination is made. Thus, even if the user looks up (changes the focus area 710) to cover the user interface 400 instead of the second user interface 520, as shown in FIG. Figure 7 As shown, the direct interaction mode determined for the second user interface 520 remains active. In other words, within the threshold amount of time, the direct interaction mode still applies to the interaction of the hand 422 with the second user interface 520 (e.g., selection interaction, movement interaction, etc.). Once the above threshold amount of time expires, the interaction mode can be determined again, for example, as direct or indirect and / or with respect to the same or different user interface elements. In some specific implementations, additional or alternative heuristics (e.g., in addition to time-based thresholds) are used to determine when to re-determine the interaction mode, for example, based on changes in content, changes in the environment, movement of the user with certain characteristics.

[0059] Fig. 8A An exemplary determination of an indirect interaction mode based on a 3D area around a virtual object is shown. In this example, user 102 is using device 105 to view a 3D area around a virtual object. Figures 4 to 7102. The 3D region 430 surrounding the user interface 400 is determined by the device 105 as previously discussed herein. Note that in this example, the dashed lines indicating the boundaries of the 3D region 430 are for illustrative purposes and are not visible to the user 102. The 3D region 430 is used to determine whether the direct interaction mode or the indirect interaction mode is applicable to the detected user activity at different points in time. In this example, as Fig. 8A As shown, the interaction mode is determined based on whether the hand 422 of the user 102 intersects with the 3D area 430. In this case, the hand 422 does not intersect with the 3D area 430, and therefore, the interaction mode is determined to be an indirect interaction mode. Therefore, the user's activities (e.g., hand movements together with the gaze direction 805, etc.) are evaluated using, for example, an indirect interaction recognition process (such as providing a selection of pointing to the user interface 400), as similar to what is described herein.

[0060] In some implementations, a third interaction mode (e.g., a mode that does not allow interaction) can be determined based on one or more criteria being satisfied. For example, such a mode can be triggered by a user providing input to specifically enter a mode in which the user's activities will not interfere with the content (e.g., by providing relative interaction with the content). In another example, such a mode can be manually or automatically triggered for a low power mode.

[0061] Figure 8B An exemplary determination of an indirect interaction mode based on a 3D area 620 and a focus area 850 around a virtual object (i.e., a second user interface 520) is shown. In this example, the user 102 is using the device 105 to view and interact with an XR environment including the user interface 400 and the second user interface 520. The 3D area 620 around the second user interface 520 is determined by the device 105, as similarly described above. Note that in this example, the dotted lines indicating the boundaries of the 3D area 620 are for illustrative purposes and are not visible to the user 102. The focus area 850 is also a 3D area (e.g., a conical area centered on a centerline 855 corresponding to the user's head direction or the forward-facing direction of the device 105), as similarly described above.

[0062] 3D area 620 and focus area 850 are used to determine whether direct interaction mode or indirect interaction mode is applicable at different time points. In this example, the interaction mode is determined based on determining that hand 422 of user 102 intersects with 3D area 620 at an intersection and whether the intersection is also within focus area 850. In this case, hand 422 of user 102 does intersect with 3D area 620, but the intersection is not within focus area 850, and therefore the interaction mode is determined to be indirect interaction mode. Therefore, for example, an indirect interaction recognition process is used to evaluate the user's activities (e.g., hand movements and gaze direction 805, etc.), as described similarly above. With respect to potential interactions with nearby user interface elements (such as user interface 400), the interaction mode may also be applicable or not, as described similarly above.

[0063] Fig. 9 Indirect interaction recognition is shown. In this example, sensor data on device 105 is used to recognize a user interaction made by user 102, for example, based on outward-facing image sensor data, depth sensor data, eye sensor data, motion sensor data, etc. Such data can be monitored for an engagement condition corresponding to the start of a user interaction. In some implementations, engagement is triggered based on detecting a hand motion (e.g., raising a hand or raising a hand while making a particular gesture).

[0064] In this example, at block 910, the process includes detecting that the user 102 has positioned their hand 422 within the field of view of the outward-facing image sensor (e.g., within Fig. 9 The process may include detecting a particular hand configuration (e.g., claw, flat hand, stable hand in any configuration, etc.) as an indication of hand engagement, or may simply include detecting the presence of hand 422 within the sensor's field of view.

[0065] At block 920, the process includes identifying an object using the user gaze data. In this example, the process includes identifying a gaze direction 905 of the user 102 at a user interface element 915 of the user interface 400, such as Fig. 9 shown.

[0066] At block 930, the process includes displaying feedback based on the object identified by the gaze. In this example, the feedback graphically distinguishes user interface element 915 (e.g., from other user interface elements displayed in user interface 400) to indicate that user interface element 915 now has a different state (e.g., a "hover" state similar to the state of a traditional UI icon when a cursor is over an item without clicking / tapping). In some implementations, displaying user interface element 915 with feedback includes highlighting or bolding user interface element 915, displaying user interface element 915 with a focus ring or glow, increasing the size of user interface element 915, and / or changing the color of user interface element 915.

[0067] At block 940, the process includes identifying a gesture to be associated with the identified object. In this example, the user is gazing at user interface element 915 in gaze direction 905 when device 105 detects (or at least within a threshold time relative to) a pinch gesture performed by hand 422. In this example, the pinch gesture is interpreted as initiating an action on user interface element 915, e.g., resulting in a cursor "click" event similar to a traditional UI icon (during which a cursor is positioned over the icon and a trigger such as a mouse click or trackpad tap is received) or a selection action similar to a touch screen "tap" event.

[0068] Fig. 9 An example of identifying indirect user interactions is shown. Numerous other types of indirect interactions may be identified, for example, based on one or more user actions to identify user interface elements and / or provide input (e.g., no-action / hover type input, selection type input, input with direction, path, speed, acceleration, etc.). Inputs in 3D space similar to inputs on a 2D interface may be identified, such as inputs similar to mouse movements, mouse button clicks, touch screen touch events, trackpad events, joystick events, game controller events, etc.

[0069] Fig.101000a . A transition from a direct interaction mode to an indirect interaction mode is shown. In this example, at a first point in time, a direct interaction mode is determined based on a user's hand 422 within a 3D region 1030 around the user interface 1000a, as similarly discussed herein. In this example, the interaction mode is determined based on whether the hand 422 of the user 102 intersects with the 3D region 1030. In this case, the hand 422 does intersect with the 3D region 1030, and therefore the interaction mode is determined to be a direct interaction mode. Therefore, for example, a direct interaction recognition process is used to evaluate the user's activities (e.g., hand movements, etc.), as previously discussed above. The direct interaction recognition process may include identifying a direct touch interaction with a UI element 1014. The direct touch may initiate an action corresponding to the UI element 1014, such as triggering the display of the user interface 1000b.

[0070] Then, at a second point in time, the direct interaction mode is exited based on the user hand 422 being outside of the 3D area 1030 around the user interface 1000b.

[0071] At a second point in time, an indirect interaction mode may be determined based on the user's hand 422 being outside of the 3D region 1030 surrounding the user interface 1000b. In this example, the interaction mode is determined based on whether the hand 422 of the user 102 intersects with the 3D region 1030, as previously discussed herein. In this case, the hand 422 does not intersect with the 3D region 1030, and therefore, the interaction mode is determined to be an indirect interaction mode. Thus, for example, an indirect interaction recognition process is used to evaluate the user's activities (e.g., hand movement, gaze direction, etc.). The direct interaction recognition process may include identifying a gaze-pinch interaction with the UI element 1015, for example, identifying that a pinch of the hand 422 occurs when the gaze direction 1005 points to the UI element 1015. The interaction may initiate an action corresponding to the UI element 1015, for example, triggering a change in the user interface 1000b, such as a change in the appearance of the user interface 1000b or one or more elements displayed in the user interface 1000b.

[0072] During the user experience, the user's interaction mode may transition from direct to indirect, and vice versa, providing the user with significant flexibility in interacting with UI content in a manner that best suits the user and / or situation as a benefit. The transition may be accompanied by feedback indicating to the user that a switch in interaction mode has occurred and / or identifying the new / active mode.

[0073] Exemplary direct and indirect interaction identification

[0074] In some cases, a user interface (or a portion of a user interface) may be presented at a location close to the current position of the user's hand. For example, when interacting with the user interface of the instrument in front, the user may rest their hand on their lap or table surface, and the user interface may initiate the display of another user interface element (such as a panel, keyboard, color picker, or other element close to the hand position). Doing so can provide the user with an intuitive, easy, efficient, etc. environment to use additional user interface elements, such as through direct touch interaction. However, at a given time when such a UI element is displayed, the user may instead intend to interact indirectly with the main user interface window displayed in front of it (e.g., based on gaze direction and triggering activities such as pinching), and may, for example, inadvertently perform a pinch (or other indirect triggering gesture) on a UI element displayed close to the user's hand. Some specific implementations provide rules, logic, or policies to interpret user activities in these and other situations based on the user's intentions.

[0075] Fig.11A The direct interaction target selection identification process is shown. In this example, the focus area 1120 is generated and used to accurately respond to the user tap interaction performed by the user 102 pushing the hand 422 to virtually touch the user interface 400. In this example, the focus area 1120 is a cone formed around the ray 1122 (e.g., a cone with a 40° cone half angle). The ray 1122 can be determined based on the user's gaze direction and / or head posture direction (e.g., using the average between the gaze direction and the head posture direction) to determine the direction of the user's intention. In some specific implementations, using a relatively wide focus area (e.g., a 30+° cone half angle or other angles determined to correspond to the beginning of peripheral vision) can provide various benefits. Such a wide focus area can help ensure that intentional direct touch interactions that may be wider than the user's focus direction (e.g., the user reaches a relatively far left, far right, far up, and / or far down to start a direct touch scroll interaction) are identified and treated as direct touch interactions. The focus area 1120 can be used within target discovery. Parameters (e.g., cone half angle) may be used to adjust the width of the cone for different situations and / or use cases. If the interaction position / pose (e.g., if hand 422 is determined to intersect user interface 400) is within focus region 1120 (e.g., Fig.11A If the target is recently in the focus area 1120 (as shown in the figure), the system can enable direct interaction (e.g., direct hover, touch, pinch, etc.). A pinch in the direct hover range can be automatically determined as a direct pinch.

[0076] FIG. 11B to FIG. 11CA distinction between direct triggering interactions and indirect triggering interactions (e.g., pinching) is shown. In this example, a focus area 1120 and / or an inspection area 1130 are generated and used to accurately respond to a triggering (e.g., pinching) interaction performed by user 102 using hand 422 (e.g., when the user pinches hand 422 on user interface 400). In this example, focus area 1120 is a cone formed around ray 1122 (e.g., a cone with a 40° cone half angle). In this example, inspection area 1130 is a sphere centered at position 1132 where the user's gaze ray intersects user interface 400. The sphere can have a size determined based on various criteria, for example, a 35 cm diameter sphere (e.g., a sphere of approximately 1.0 foot diameter) can ensure that the sphere will include a keyboard displayed within user interface 400. To interpret the user action (i.e., pinching hand 422 at hit location 1140 on user interface 400), the system determines whether hit location 1140 is within inspection region 1130. If hit location 1140 is within inspection region 1130, the pinch of hand 422 is determined to be a direct interaction, i.e., a pinch at location 1140, rather than an indirect interaction, i.e., a pinch associated with location 1132 where the user's gaze ray intersects user interface 400. In contrast, if hit location 1150 associated with the pinch is outside of inspection region 1130 (e.g., Fig. 11C As shown), the pinch of hand 422 is determined to be an indirect interaction, i.e., the pinch will be associated with location 1132 where the user's gaze ray intersects with user interface 400, rather than a direct interaction, i.e., a pinch at location 1150.

[0077] Focus area 1120 and / or inspection area 1130 may additionally or alternatively be used to provide appropriate hover feedback / state determination. When the user's hand approaches the user interface, if hand 422 is within inspection area 1130, proximity / finger-based direct hover feedback may be displayed (e.g., highlighting the location on the UI closest to the user's hand). However, if hand 422 is not within inspection area 1130, such direct hover feedback may be suppressed, i.e., not shown. The system may apply criteria to ensure that both direct hover feedback and indirect hover feedback are not displayed when the user is both gazing and interacting directly with a UI element such as a keyboard. The system may also use such criteria to handle situations where a UI element is close to the user (e.g., a keyboard is close to the user's stationary hand position) and the main user interface is at a greater distance. If the user is gazing at a near UI element, direct interaction with that near UI element via gestures will be enabled, whereas if the user is gazing at the primary UI, gestures (outside inspection area 1130) (even if located at a near UI element) will be treated as indirect interactions with the location on the primary UI where the user is gazing.

[0078] In some implementations, interpreting a pinch as a direct pinch or an indirect pinch is based on whether the pinch is associated with a hit location on the user interface that is within an inspection zone. A relatively small / narrow inspection zone (e.g., a sphere with a 35 cm radius) around an interaction gesture can be used as the inspection zone. If the gaze location on the UI is outside the inspection zone (e.g., sphere), gaze-based hovering can be enabled and / or the pinch gesture can be directed to the gaze location, i.e., providing indirect interaction.

[0079] In some implementations, triggering (e.g., pinching) and non-triggering (e.g., tapping, scrolling, etc.) interactions are treated differently. For example, in a situation where a UI element is close to the user (e.g., a keyboard is close to the user's resting hand position) and the main user interface is at a greater distance, the system may use the inspection area 1130 to distinguish triggering gestures (e.g., pinching) such that the user, for example, only performs a direct pinching interaction on a neighboring UI element (e.g., keyboard) when the user is looking at or near the neighboring UI element. In contrast, the system may not use the inspection area for non-triggering interactions such that the user, for example, may perform a direct non-triggering gesture (e.g., tapping, scrolling, etc.) on a neighboring UI element even when the user is not looking at or near the neighboring UI element.

[0080] The inspection area can be used to both: (a) determine whether to process a trigger gesture (e.g., a pinch) as a direct trigger gesture (e.g., a direct pinch) versus an indirect trigger gesture (e.g., an indirect pinch); and / or (b) provide appropriate hover feedback / state (e.g., providing direct hover feedback / state (e.g., based on gaze) and / or indirect hover feedback / state (based on finger position proximity to the UI)).

[0081] FIG. 11D to FIG. 11E Examples of hover feedback / state determination are shown. In these examples, region of interest 1120 and / or inspection region 1130 are generated and used to accurately respond to gaze and hand activities. In this example, region of interest 1120 is a cone formed around ray 1122. In this example, inspection region 1130 is a sphere centered at location 1132 where the user's gaze ray intersects user interface 400. In order to interpret the user action (i.e., gaze location 1132 while positioning hand 422 near location 1160), the system determines whether the location associated with the direct interaction (e.g., location 1160) is within inspection region 1130. If the location is within the inspection region (e.g., Fig.11D ), gaze hovering is disabled, i.e., the UI element at position 1132 is not hovered. In this example, the UI element at position 1160 associated with direct interaction is hovered.

[0082] In contrast, if the position associated with the direct interaction is outside the inspection area (e.g. Fig.11E), gaze hover is enabled, i.e., the UI element at position 1132 is hovered. In this example, the UI element at position 1160 associated with the direct interaction also remains hovered. Such criteria can prevent potential confusion (which might otherwise occur) caused by direct hover and indirect hover feedback being too close together (e.g., two different keys highlighted on a virtual keyboard). Once the user gazes away (outside of inspection area 1130), gaze hover reappears (as shown in FIG. 11A ). Fig.11E ), indicating to the user that the pinch will be sent to / associated with an indirect interaction at the gaze location (i.e., location 1132). Fig.11D In , if the user were to pinch, the pinch would be a direct interaction at location 1160, and if the user were to tap, the tap would be a direct interaction at location 1160. Fig.11E , if the user were to pinch, the pinch would be an indirect interaction at location 1132 , and if the user were to tap, the tap would be a direct interaction at location 1170 .

[0083] Hover persistence

[0084] FIG. 12A to FIG. 12D Hover persistence is shown. Fig. 12A In the example, the focus area 1210 is determined based on the user's interest direction (e.g., based on the gaze direction and / or the head posture direction). Based on the user interface button 1222 of the user interface element 1220 being within the focus area 1210 and the hand 422 of the user 102 being close to the button 1222, the button 1222 is hovered, for example, provided with hover feedback and / or is in a hovering state. Fig. 12A Afterwards, user 102 looks up from user interface element 1220 to user interface 400, as shown in FIG. Fig. 12B As shown. Based on determining that the position of hand 422 remains close to the hovered button 1222, the hover state / feedback of button 1222 is maintained even if button 1222 is no longer in focus area 1210. The initial hover determination is persistent in the following sense: as long as hand 422 remains close to the hovered button 1222, the initial hover determination is maintained when the gaze is moved away. In addition, the system identifies that the user is gazing at UI feature portion 1115 and determines to provide gaze hover feedback for UI feature portion 1115. Therefore, at the same time, finger hover feedback is provided for button 1222, and gaze hover feedback is provided for UI feature portion 1115.

[0085] Such dual feedback can indicate to the user how various additional interactions will be handled, for example, how triggering (e.g., pinching) and non-triggering (e.g., tapping, scrolling, etc.) gestures will be handed over. In addition, the state of the UI component (e.g., hovering or not hovering) can determine how the user activity is interpreted. FIG. 12C to FIG. 12DIn the example of , triggering and non-triggering gestures are handled differently, taking into account the state of the UI component. In this example, the system responds to a non-triggering gesture (e.g., a tap) on the hover button 1222 (e.g., Fig. 12C ) as a direct interaction with button 1222. In contrast, the system responds to a trigger gesture (e.g., pinch) on hover button 1222 (e.g., Fig.12D ), as an indirect interaction based on gaze direction (i.e., with UI feature 1115). FIG. 12C to FIG. 12D In the example of , if the user hovers their finger over the button and then looks away and pinches, the pinch will be guided as an indirect interaction. However, the user can still look away and perform a non-trigger direct touch interaction.

[0086] Fig.13 1 is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data. In this example, hand data (e.g., "hand data") is input to a hand processing block 1310. An upsampling and prediction block 1312 upsamples the hand data (e.g., adds more frames) and makes predictions about future hand positions, paths, trajectories, velocities, accelerations, etc. In some implementations, in the hand input process, the internal state of the upsampler is updated with new observation data from a hand computer vision-based tracking system. In the interaction detection process, data is received about when the next frame will be presented, which calculates an interpolated / predicted timestamp and queries the upsampler for joint data corresponding to that timestamp. These steps may occur asynchronously and / or at different frame rates. The process may typically query the upsample at different timestamps between updates from the computer vision tracking system for several predictions, for example, hand tracking may run at 30Hz while the display may render at 90Hz.

[0087] The hand and viewpoint correction (POVc) blending block 1314 can use the corrected and uncorrected hand joint data to mitigate POVc-coupled hand jumps that may occur, for example, due to background depth and camera switching. The hand and POVc blending block can calculate new hand pose information to account for the difference between where the user's hand is in the real world (e.g., tracked by a computer vision system) and where the hand appears to be from the perspective of the user viewing the display. The algorithm can smoothly blend the real and POV-corrected versions of the hand skeleton to create hand pose information that is optimized for input recognition in a usage scenario where hand registration to the UI is important (e.g., during direct input, where the finger must be aligned with the button). When the user's hand is outside the user's field of view (FOV) and / or the field of view of the sensor, it may not be desirable to expend resources or may not otherwise provide correction, resulting in a jump in the hand position. The process may include, for example, smoothly blending hand data over time based on whether the user is in direct or indirect mode, whether the hand is near a UI element, or other factors.

[0088] Eye data (e.g., "eye data") is input to gaze processing block 1320. Gaze detection block 1322 removes glances, blinks and / or other gaze loss events (e.g., leaving only the gaze segment corresponding to gaze). Exemplary glance detection process can use gaze confidence, tracking state, pupil center, pupil diameter, interpupillary distance (IPD), gaze ray data and speed data to detect glances and blinks for removal and / or to identify gaze for gaze interaction. It can be distinguished between gaze events as gaze events and gaze events as glances for more accurate gaze-based input. Gaze filtering block 1324 filters and / or smoothes gaze data signals. This can involve smoothing gaze data during gaze. Filtering can be closely coupled to gaze motion classification. Gaze processing block 1320 can lock / remember the last smoothed gaze position for those temporary periods, or can utilize additional algorithms / strategies to "fill in" the data between gazes.

[0089] The processed hand and eye data is provided to the interaction detection block 1330, as Fig.13As shown. Note that the interaction detection block 1330 may additionally use UI geometry data including data about the user interface of the application (e.g., the appearance of the user interface, the location of the user interface elements within the user interface, the behavior of the user interface elements within the user interface (e.g., interactivity), etc.). The UI geometry data may be provided at a different rate than the hand and / or eye data. In one example, UI data is received at a first rate (e.g., 90fps) while hand and gaze data are received asynchronously at a slower average rate (e.g., 60fps). The interaction detection block 1330 may additionally use a frame target time, for example, a prediction of when the frame will be rendered by the system. The interaction detection block 1330 may, for example, run asynchronously when UI updates and / or gaze / hand data are received. In some specific implementations, as discussed below, the interaction detection block 1330 includes a gaze blur hit test and focus block 1332, an unexpected rejection block 1334, and a direct and indirect mode detection block 1336.

[0090] Gaze fuzzy hit test and focus area block 1332 identifies virtual elements (e.g., UI elements) at which the user's gaze is considered to be associated with (e.g., pointing to) a focus area, for example, based on head or device orientation / direction. An exemplary gaze fuzzy hit test process may include using gaze ray data, confidence data, gesture data (e.g., hand movement classification), gaze clustering expansion data, etc. to relax / tighten the gaze area / cone (e.g., focus area) based on the accuracy of gaze tracking and / or user behavior. The process may include utilizing UI geometry from a system process based on UI information provided by one or more applications, such as identifying interaction targets (e.g., which UI elements are to be associated with a given user activity) based on a 3D representation of the UI element. The use of such UI information may facilitate more accurate hit testing. In addition to UI geometry, fuzzy hit testing may query more detailed data, for example, data about remote effects, gestures, or other contexts.

[0091] The accidental rejection block 1334 identifies hand and gaze behavior instances that are rejected (i.e., associated with subconscious user behavior that is not an intentional interaction) but are still associated with virtual elements under appropriate conditions. The accidental rejection block 1334 attempts to understand the intention behind the user's motion to better interpret the motion as, for example, an intentional UI element interaction or other matter. An exemplary accidental rejection process can use hand joint data, interaction state history, gaze focus area data, occlusion data, etc. to reject (or cancel) unintentional user interactions. The process may include utilizing UI geometry from a 3D simulation system based on UI information provided by an application, such as distinguishing intentional user activities from unexpected activities based on nearby UI elements. The use of such UI information can facilitate more accurate accidental rejection. For example, based on determining (determined from data declared by the application) that the UI of the application is 2D / plane and therefore the motion in the Z direction is unlikely to be the expected input directed to the UI, that is, it may not be an unexpected motion intended to trigger a UI interaction, and therefore the input support process can be more confident in rejecting user activities as unexpected, and the user motion in the Z direction can be identified as unexpected.

[0092] Some implementations support two interaction modes (e.g., direct interaction mode and indirect interaction mode). Some implementations support other combinations of interaction modes (e.g., direct interaction mode, indirect interaction mode, gaze-only interaction mode, and / or peripheral device interaction). A process can be used to determine which, if any, of several interaction models a user is executing. This can involve identifying and ignoring unexpected behavior, and also disambiguating which interaction model is expected when the user wants to interact intentionally. In this example, the direct vs. indirect mode detection block 1336 uses processed hand and eye data to determine an interaction mode for interpreting user activity, e.g., selecting a direct interaction mode or an indirect interaction mode, as previously discussed herein.

[0093] If direct interaction mode is selected, the hand and eye data (processed and / or unprocessed) is provided to a direct interaction recognition block 1340, which provides direct input to the user interface. In some specific implementations, the direct interaction recognition block 1340 includes a trajectory correction block 1342 and a hover, make, break block 1344. The trajectory correction block 1342 can adjust the trajectory of the user's motion, such as adjusting the user's hand motion to take into account the user's behavior in the 3D environment, such as so that the motion data better corresponds to the user's expected motion / expected interaction. The hover, make, break block 1344 can use gestures, motion paths, speeds, accelerations, etc. to perform various recognition processes to identify certain types of direct interactions with user interface elements, such as a "hover" interaction based on detecting that a hand / finger is close to but not touching a user interface element, a "make" interaction based on detecting a point (in 3D space / time) at which a hand / finger has contacted a user interface element, a "break" interaction based on detecting a point (in 3D space / time) at which a hand / finger has stopped contacting a user interface element, etc.

[0094] Many types of direct interactions can be recognized and provided as user input. In another example, a direct interaction is recognized by recognizing that a user contacts a slider user interface element, moves a hand to the left a distance "X" while in contact with the slider, and then retracts his hand (e.g., moves his hand away from the slider user interface element) to break contact with the slider. The slider can be moved to the left, for example, a distance X, based on this input. In another example, a user makes a similar motion to provide input that imparts velocity on a user interface element, such as providing a 3D swipe gesture on a user interface to cause the user interface to begin moving and continue moving after the hand breaks contact, where the continued motion is based on the velocity already imparted on the UI object, such as similar to when a user swipes upward on a multi-touch interface of a mobile phone.

[0095] If the indirect interaction mode is selected, the hand and eye data (processed and / or unprocessed) is provided to the indirect interaction recognition block 1350, which provides indirect input to the user interface. In some embodiments, the indirect interaction recognition block 1350 includes a user-centric motion block 1352 and a pinch and gaze fusion block 1354. The user-centric motion block 1352 accounts for user-centric motion in the hand data (e.g., accounting for the fact that the user may be performing a motion that pivots around a rotation point (e.g., shoulder, elbow, wrist, etc.)). The user-centric motion block 1352 can be used to map the user's hand motion to an object or affordance representation that is separate from the hand, such as an affordance representation displayed on the application content.

[0096] The pinch and gaze fusion box 1354 determines which gaze data is associated with a user activity (such as a pinch gesture) indicating an intention to interact with a UI element. Various types of indirect interactions can be recognized and provided as user input. In one example, the user pinches the fingers together and moves the pinched hand while gazing at the UI element to provide a movement input pointing to the UI element. In another example, the user quickly pinches and releases the fingers while gazing at the UI element to provide a selection event (e.g., similar to a touch screen tap or mouse click input event).

[0097] like Fig.13 The input support process shown in the example of may perform one or more core functions. In some specific implementations, these functions include, but are not limited to, obtaining gaze data and associating the user's gaze with UI elements associated with one or more applications. Doing so may involve performing a fuzzy hit testing process. Doing so may involve creating a focus area. A focus area may be considered a spatial area / region. Larger objects may be partially inside / partially outside the focus area, so that the system needs to test where the user is interacting on the object to understand whether the interaction point is within the user's focus area. The focus area may include some or all parts of a set of zero or more UI elements that a user may focus on at a given time. Focus areas can be used to disambiguate between interaction types (e.g., direct or indirect input modalities), disambiguate which application a user is interacting with, and / or disambiguate which UI elements are the targets of the user's intent to interact with UI elements.

[0098] In some implementations, the input support process relies heavily on the simulation system to provide a geometric representation of the user interface for hit testing against hand data and gaze data. The input support process may use a collision world (e.g., a 3D geometry abstraction) provided by the simulation system. The input support process may use user interface metadata information (e.g., UI priority, hierarchical display ordering, transparency information, etc.) to facilitate appropriate UI target selection. The input support process may utilize a separate hit testing process that produces 3D world hit test results. The input support process may utilize a hit testing process that uses a hierarchical tree provided by the application (e.g., declaring UI elements, relationships, and / or rendering, such as information about which areas are transparent, rendered on top, etc.).

[0099] In some implementations, the input supports the execution of a process Fig.13The functionality shown is to provide an input support process that generates data for an application. The input support process can generate an interaction state of one or more UI elements, such as identifying whether the interaction is direct or indirect, whether the interaction is a hover / close proximity interaction (e.g., associated with user activity that is not linked to UI interaction behavior) or a gesture / pinch / touch interaction (e.g., associated with user activity that indicates intentional behavior of the UI).

[0100] In some implementations, the input support process includes providing an interaction gesture, such as a point of trajectory correction on a UI element configured to drive a touch screen tap-type input.

[0101] In some implementations, the input support process includes providing a manipulator pose corresponding to, for example, the position and / or orientation of the hand itself. The manipulator pose can provide a 3D position of a stable hand center or pinch centroid (e.g., based on contact of two fingers of the hand). The manipulator pose can provide the position and / or orientation of the manipulator driving the interaction, for example, for direct touch interaction, index fingertip information (e.g., position), and for indirect pinch interaction, pinch information (e.g., pinch centroid).

[0102] In some implementations, the input support process includes providing an activity target (e.g., a UI element).

[0103] In one example, a user initiates an interaction by pinching while gazing at a UI element. The application receives an interaction gesture on the UI element and recognizes the user's intention to interact with the UI element. The user continues to pinch and moves their hand to the left. The application receives more interaction gestures, such as a set of positions on the user interface based on the 3D position of the hand when it moves to the left, and responds by moving the associated UI element to the left. In this example, by pinching while gazing at a UI object and then moving the pinched hand to the left, the user provides an input that the application can recognize to move the object to the left. The application can respond accordingly without receiving (and in fact not receiving) information about the user's gaze direction and / or specific hand characteristics. In this example, the application only receives interaction gesture data and can therefore be an application that can only receive interaction gesture data as a proxy for touch input data. Manipulator gesture data such as the 3D position of the hand when it moves to the left can be provided to the application in addition or alternatively, and the application can determine the movement of the associated object based on the changed manipulator gesture position. Applications can be provided with information based on user changes, for example, providing user-centric acceleration deltas similar to delta updates for a trackpad, which communicates the amount the user has moved from one frame to another, rather than absolute position. For example, communicating changes via a separate API can help us optimize a set of signals to accurately represent motion (e.g., for scrolling) separately from signals that also need absolute position accuracy (e.g., for drawing).

[0104] In some implementations, the input support process includes receiving a gaze direction that may potentially be associated with different UI elements, for example, because the gaze direction is between UI elements or because UI elements overlap each other. The fuzzy hit testing discussed above and other such processes can help disambiguate the user's intent to identify the appropriate UI element in such cases.

[0105] In some implementations, the input support process includes recognizing two-handed 3D gestures, such as a two-handed zoom gesture, and providing information to an application corresponding to a multi-touch touchscreen input gesture. For example, doing so may involve generating an interaction gesture for each hand that may be interpreted by the application's 2D touch-based gesture recognition process as two touch points moving closer or further away from each other, which may be interpreted as a pinch-to-zoom touchscreen gesture (e.g., zooming in and out of a user interface).

[0106] In some implementations, applications are developed for use on mobile platforms that use single-touch and / or multi-touch input gestures. Applications may be developed by application developers, include calls to gesture APIs and then specify what to do for certain types of touch events, e.g., do "X" for tap events, do "Y" for swipe events, etc. It may be desirable to use such applications on new or different platforms that provide input modalities other than (or in addition to) single-touch and / or multi-touch input gestures. Rather than implementing all new 3D gestures and gesture recognition processes within applications, some implementations disclosed herein interpret 3D user activity (e.g., hand position, gaze direction, etc.) and send proxy data to applications corresponding to where the touch would have been if the 3D user activity had been performed as a 2D touch input gesture. The input support process may provide such proxy data to such applications, and in doing so, enable a large number of existing 2D / touch-based applications to be easily imported and used within new 3D systems that use new and different 3D user activity-based inputs. The input support process may make additional information available so that newer / modified applications can take advantage of / use more information about the user's 3D activity. Alternatively, this information may be utilized by older, unmodified applications that link with an XR-compatible version of the UI framework, which may utilize this additional data on behalf of the app in the background (e.g., a binary-compatible app using a standard long press recognizer may benefit from an XR-based OS implementation that uses the z-depth of a touch to more accurately recognize a long press than purely 2D native methods).

[0107] In some embodiments, to enable the use of both traditional 2D / touch-based applications and newer applications with additional 3D recognition capabilities, the input support process may provide multiple types of information, such as interaction gesture data for supporting touch-based input recognition for traditional applications and manipulator gesture data for supporting applications with additional 3D recognition capabilities.

[0108] Interpreting 3D user activity as 2D touch-based activity may require overcoming various challenges. For example, a user's hand movement in 3D space while performing a touch gesture may be equivalent to a touchscreen tap gesture, but may lack the precision expected of a touchscreen tap gesture. A touch on a touchscreen involves contact with a physical surface where the finger rests, whereas a motion to "virtually" touch a UI in a 3D XR environment may involve the user's hand piercing a UI element (e.g., as shown in the figure). Figure 2210b of ). The user may intend to tap the surface of the virtual button, but actually pierce the surface at one location and then retract the hand / finger at a slightly different location. This user activity involving two different UI element locations (e.g., a first entry location and a second exit location corresponding to the UI element) can be interpreted (by the input support process) as a tap at a single point using various algorithms and / or machine learning processes (e.g., performing trajectory correction). The application receives the single location (from the input support process) and recognizes it as a touch event. Thus, the input support process can classify and / or interpret user activity to account for the unique circumstances of interacting within a 3D environment, and encapsulate the data provided to the application so that the application itself does not need to distinguish between actions intended by the user and actions actually performed by the user.

[0109] When providing data to the application, the input support process can take into account sensor inaccuracies / limitations, such as taking into account the fact that the identified gaze direction may be different from the actual gaze direction. The application itself does not need to take such inaccuracies into account.

[0110] In formatting an application's data in a format that the application will understand, the input support process can resolve ambiguities, account for inaccuracies, and / or repackage input provided in input modalities that the application itself does not understand / needs to understand into a format that the application does recognize.

[0111] In short, the input support process can perform various processes to interpret raw 3D data for consumption by applications, so that applications (and their developers) do not need to perform those processes via manually programmed processes. Additionally, performing such processes by a shared OS process can unify the input experience across all apps, so that users receive consistent UI response behavior.

[0112] In addition, a shared input support process (e.g., used by multiple applications) can enable multiple interaction modalities, for example, providing different ways for a user to select a button, and abstracting the user activity data associated with those different modalities for the same type of interaction to provide the same data to the application for different types of input. Thus, input modalities can be changed and / or added over time without requiring changes to the application, as long as the input support process can abstract new or changed input modality user activity into a format that the application understands.

[0113] In some implementations, an application utilizes a gesture recognizer running within an application process. The gesture recognizer may include a state machine that categorizes input received in an input data stream, e.g., a tap started, a tap in progress, a tap ended. An input support process may provide data to an application that is recognized by a gesture recognizer running within an application process as a particular type of input (e.g., a tap) that may correspond to a single input modality (such as a touch-based input modality) dedicated to the application. The input support process may configure the data provided to the application to ensure that the application's process accurately recognizes user activity, e.g., by providing the data in a form expected by the gesture recognizer. In one example, the input support process converts user activity involving user gaze and gestures into data that can be recognized as touch input. In another example, the input support process converts user activity involving a 6DOF controller and gaze direction into data that can be recognized as touch input.

[0114] Application developers do not have to write applications for different input modality platforms. Instead, application developers can write applications for touch platforms (or generic 2D input modalities), and the application can be imported with little or no effort to work within the 3D user activity platform. A single application can be used on mobile devices, laptop devices, tablet devices, desktop devices, and 3D XR devices. The techniques disclosed herein can achieve binary code compatibility, that is, applications can be executed in different environments that utilize different input modalities.

[0115] In some embodiments, the input support process provides gesture markers and / or gesture classifications to the application, for example, indicating to the application that the user has raised their hand or that a particular gesture has been recognized as a tap, swipe, scroll, etc., as a hint that the application can use when recognizing a gesture.

[0116] In some specific implementations, the input support process facilitates the application to recognize gaze-only input without providing raw gaze data to the application. This may involve periodically providing gaze data (i.e., the position on the UI element that the user is gazing at). Only gaze data may be associated with an interaction state, and data may be provided to the application based on gaze-only data only when the gaze data is associated with a specific interaction state, such as providing data when a gaze hover state is applicable. Such data may be provided only based on explicit user authorization. For example, when a user stares at a UI element for more than a threshold amount of time (e.g., 0.01, 0.1, 0.5, 1, 2, 3, 4, 5, etc. seconds), the application may be notified, and an affordance (e.g., a dot) may be displayed. If the user then views the affordance, the application may be notified that the UI element being stared at has been hit / selected. In another example, staring at a dot within a UI element for more than a threshold amount of time may trigger hover feedback, which may be handled outside the process or by the application. A gaze-only input modality can be another input modality that is abstracted into data provided to an application so that the application can interpret the input, for example, using a general or touch-based gesture recognition process.

[0117] In some implementations, the input support process determines the interaction state of each hand (e.g., direct vs. indirect) and / or the interaction state associated with the user's gaze (e.g., gaze only or not only gaze).

[0118] Fig.14 1 is a flow chart illustrating a method 1400 for interpreting user activities as user interactions with virtual elements positioned within a 3D space. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1400. In some implementations, method 1400 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1400 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0119] At block 1402, method 1400 includes displaying an XR environment corresponding to a 3D space, wherein the XR environment includes a virtual object depicted at a location in the 3D space. The virtual object may be a user interface element, such as a button, icon, text input field, slider, or menu item, that is rendered as part of a virtual UI displayed at a fixed location or otherwise within the XR environment, e.g., a virtual 2D menu with buttons displayed a few feet in front of a user in XR.

[0120] At box 1404, method 1400 includes tracking the user's hand position in 3D space based on sensor data. This may involve obtaining user position data corresponding to the 3D position of a part of the user (e.g., a finger / hand) in 3D space. The user position data may be obtained based on sensor data (e.g., image sensor, depth sensor data, motion sensor data, etc.), and may provide the path of the part of the user over a period of time. In one example, the user position data may be a skeletal representation of the user generated periodically (e.g., 30fps) based on an outward-facing image / depth sensor on the HMD. Such user position data may provide the path of a part of the skeleton (e.g., a finger skeleton portion). In other words, for example, by tracking the orientation of the skeleton at multiple moments, the path of the finger portion of the skeleton relative to the 3D space over time may be determined.

[0121] In some implementations, the portion of the user corresponds to a point on or in a finger of the user. The portion of the user may correspond to a point on or in a hand of the user. The user position data may correspond to a position within a skeletal representation of the user that is generated periodically (e.g., at multiple points in time during a period of time).

[0122] At block 1406, method 1400 includes determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space. Determining the interaction mode may include selecting whether the interaction mode is direct or indirect based on the user hand position and the position of the virtual object in the 3D space. Determining the interaction mode may involve selecting a direct interaction mode based on the hand position intersecting a 3D region associated with the virtual object in the 3D space. Determining the interaction mode may involve selecting an indirect interaction mode based on the hand position being outside of a 3D region associated with the virtual object in the 3D space.

[0123] At block 1408, method 1400 includes interpreting the gesture using an interaction recognition process associated with the determined interaction mode. In some implementations, the method involves tracking the position of the gesture in 3D space and identifying one or more interaction points based on the tracking. The method may involve identifying one or more interaction points based on a gaze direction.

[0124] In some implementations, determining the interaction mode includes selecting a direct interaction mode in which a virtual element is identified based on an intersection of a hand position and a 3D area associated with a virtual object in 3D space, and the interaction recognition process is a direct interaction recognition process applied to a gesture. Determining the interaction mode may involve selecting the direct interaction mode based on: determining a focus area based on a gaze direction or head posture of a user; determining an intersection point where the hand position and the 3D area associated with the virtual object in the 3D space intersect; and determining that the intersection point is within the focus area. In some implementations, the interaction mode is a direct interaction mode, and method 1400 includes interrupting interaction detection for a predetermined period of time (e.g., 0.1, 1, 2, 4, 5, 10, 15, etc. seconds) after recognition of a gesture.

[0125] Some implementations initiate selection of a virtual object based on the interaction mode being a direct interaction mode and the gesture being a direct touch with the virtual object. Some implementations initiate hovering of a virtual object based on the interaction mode being a direct interaction mode and the hand position being close to the position of the virtual object and within the focus area. Hovering can be an interaction similar to a mouse or cursor hover in a traditional computing (e.g., desktop, laptop, etc.) environment, in which a cursor is positioned over or on a UI element without an activation or selection command, during which hover feedback can be provided, such as highlighting or otherwise distinguishing the element. When the position of the virtual object is no longer in the focus area, the hovering of the virtual object can be automatically maintained, for example, for a predetermined period of time, such as 10 seconds, 30 seconds, etc.

[0126] In some implementations, determining the interaction mode includes selecting an indirect interaction mode in which a virtual element is identified based on an intersection of a gaze direction and an area associated with a virtual object, and the interaction recognition process is an indirect interaction recognition process applied to a gesture. Determining the interaction mode may involve selecting the indirect interaction mode based on: determining a focus area based on a gaze direction or a head gesture of a user; determining an intersection point where the hand position intersects a 3D area associated with the virtual object in the 3D space; and determining that the intersection point is outside the focus area.

[0127] Some implementations initiate selection of a virtual object based on the interaction mode being an indirect interaction mode and the gesture being a pinch that occurs when gaze is directed toward the virtual object. In some implementations, the interaction mode is an indirect interaction mode, and interpreting the gesture includes: recognizing that the gesture is a pinch; and identifying the virtual object based on the gaze direction occurring during the pinch. In some implementations, the interaction mode is an indirect interaction mode, and interpreting the gesture includes: recognizing that the gesture corresponds to a hand engagement; identifying the virtual object based on the gaze direction occurring during the hand engagement; and initiating hovering of the virtual object based on the hand engagement.

[0128] In some implementations, method 1400 is performed by an electronic device that is a head-mounted device (HMD), and / or the XR environment is a virtual reality environment or an augmented reality environment.

[0129] In some implementations, the system or method detects when a user rests their hand (or arm) on their knee, the armrest of a chair they are sitting in, or another physical object for or in an indirect mode. The user's hand or arm supported by another physical object can be used to determine to use an indirect interaction mode and / or to determine how to interpret gestures performed by the hand.

[0130] In some embodiments, the method is performed by executing instructions (stored in a non-transitory computer-readable medium). The method may obtain a user hand position in a 3D space based on sensor data, wherein the hand position is associated with a gesture. The method determines that an arm or hand associated with the hand position is supported by (e.g., resting on) a physical object (e.g., a user's knee, a user's leg, an armrest, etc.). Based on determining that the arm or hand is supported by the physical object, the method interprets the gesture using an interaction recognition process associated with an indirect interaction mode. The gesture may be interpreted as a user interaction in an extended reality (XR) environment corresponding to the 3D space. Determining to utilize an indirect interaction mode may be based on determining that the arm or hand is supported by a physical object. The method may initiate an action (e.g., selection of a virtual object) based on recognizing a gesture (e.g., based on recognizing a gesture that occurs when gaze is directed toward a virtual object).

[0131] In the indirect interaction mode, a virtual element may be identified based on an intersection of a gaze direction and a region associated with a virtual object.

[0132] The interaction recognition process is an indirect interaction recognition process of the identification gesture of the application. In some specific implementations, recognizing the gesture involves recognizing a pinch.

[0133] In some embodiments, the method determines that the arm or hand associated with the hand position is supported by the physical object based on determining a first position of the arm or hand, determining a second position of the physical object, and determining that a relationship between the first position and the second position satisfies a criterion (for example, determining based on image data, depth data, etc. that a skeletal representation of the user's hand is within a threshold distance of the surface of the physical object).

[0134] In some implementations, method 1400 provides data corresponding to user activity to an application. The data may enable the application to interpret the user activity. The data may be specific to the interaction mode. In one example, for a direct interaction mode, the data includes an interaction gesture with the index fingertip as a manipulator gesture and trajectory corrections, for example, direct touch specific trajectory corrections may be applied to the interaction gesture. In another example, for an indirect interaction mode, the data includes a pinch centroid as a manipulator gesture and an interaction gesture or user-centric data. This may involve running different signal chains to generate interaction gestures and other indirect specific "user-centric increments".

[0135] Fig.15 1 is a block diagram of an electronic device 1500. Device 1500 illustrates an exemplary device configuration of electronic device 105 or 110. Although some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1500 includes one or more processing units 1502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1506, one or more communication interfaces 1508 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1510, one or more output devices 1512 (e.g., including a display), one or more internal-facing and / or external-facing image sensor systems 1514, memory 1520, and one or more communication buses 1504 for interconnecting these and various other components.

[0136] In some implementations, one or more communication buses 1504 include circuits that interconnect system components and control communications between system components. In some implementations, one or more I / O devices and sensors 1506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0137] In some implementations, the one or more output devices 1512 include one or more displays configured to present a view of a 3D environment to a user. In some implementations, the one or more displays correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, the one or more displays correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 1500 includes a single display. In another example, the device 1500 includes a display for each eye of the user.

[0138] In some implementations, one or more output devices 1512 include one or more audio generating devices. In some implementations, one or more output devices 1512 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sound (e.g., 3D audio effects). Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sound can involve converting sound waves (e.g., using head-related transfer functions (HRTFs), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) that are emitted from one or more points in a 3D environment. Spatialized sound can induce the listener's brain to interpret the sound as if it occurs at one or more points in a 3D environment (e.g., from one or more specific sound sources), even though the actual sound may be generated by speakers in other locations. One or more output devices 1512 may be additionally or alternatively configured to generate tactile sensations.

[0139] In some implementations, the one or more image sensor systems 1514 are configured to obtain image data corresponding to at least a portion of the physical environment. For example, the one or more image sensor systems 1514 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1514 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1514 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0140] Memory 1520 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 1520 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 1520 optionally includes one or more storage devices remotely located from one or more processing units 1502. Memory 1520 includes non-transitory computer-readable storage media.

[0141] In some implementations, the memory 1520 or a non-transitory computer-readable storage medium of the memory 1520 stores an optional operating system 1530 and one or more instruction sets 1540. The operating system 1530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1540 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1540 is software that can be executed by one or more processing units 1502 to implement one or more of the techniques described herein.

[0142] The instruction set 1540 includes a user interaction instruction set 1542 that is configured to, when executed, identify and / or interpret user gestures and other activities as described herein. The instruction set 1540 may be embodied as a single software executable file or as a plurality of software executable files.

[0143] Although instruction set 1540 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. In addition, the drawings serve more as functional descriptions of the various features present in a particular implementation, as opposed to the structural schematics of the implementations described herein. As will be appreciated by those of ordinary skill in the art, the items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0144] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0145] As described above, one aspect of the present technology is to collect and use sensor data that may include user data to improve the user experience of electronic devices. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0146] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0147] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0148] Regardless of the foregoing, the present disclosure also contemplates implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0149] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data. For example,

[0150] Preferences or settings are inferred based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to the content delivery service, or publicly available information to select and deliver content to the user.

[0151] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access their stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0152] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0153] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0154] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0155] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0156] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0157] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0158] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0159] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0160] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At an electronic device having a processor: Obtaining a position of a virtual object in an extended reality (XR) environment corresponding to a three-dimensional (3D) space; obtaining a user's hand position in the 3D space based on the sensor data, wherein the hand position is associated with a gesture; determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space; and The gesture is interpreted using an interaction recognition process associated with the determined interaction mode. 2 . The method of claim 1 , wherein determining the interaction mode comprises selecting whether the interaction mode is direct or indirect based on the user hand position and the position of the virtual object in the 3D space.

3. The method according to claim 1, wherein determining the interaction mode comprises selecting a direct interaction mode in which the virtual object is identified based on an intersection of the hand position and a 3D region associated with the virtual object in the 3D space, and The interaction recognition process is a direct interaction recognition process applied to the gesture. 4 . The method of claim 1 , wherein determining the interaction mode comprises selecting a direct interaction mode based on an intersection of the hand position and a 3D area associated with the virtual object in the 3D space.

5. The method of claim 1 , wherein determining the interaction mode comprises selecting a direct interaction mode based on: determining a region of interest based on a gaze direction or a head pose of the user; determining an intersection point where the hand position intersects a 3D region associated with the virtual object in the 3D space; as well as It is determined that the intersection point is within the region of interest. 6 . The method of claim 1 , wherein the interaction mode is a direct interaction mode, and the method further comprises interrupting interaction detection for a predetermined period of time after recognition of the gesture. 7 . The method of claim 1 , further comprising initiating selection of the virtual object based on the interaction mode being a direct interaction mode and the gesture being a direct touch with the virtual object.

8. The method of claim 1, further comprising initiating hovering of the virtual object based on the interaction mode being a direct interaction mode and the hand position satisfying a proximity criterion with respect to the position of the virtual object and being within a focus area. 9 . The method of claim 8 , further comprising maintaining the hovering of the virtual object when the position of the virtual object is no longer in the focus area.

10. The method according to claim 1, wherein determining the interaction mode comprises selecting an indirect interaction mode in which the virtual object is identified based on an intersection of a gaze direction and a region associated with the virtual object, and The interaction recognition process is an indirect interaction recognition process applied to the gesture.

11. The method of claim 1 , wherein determining the interaction mode comprises selecting an indirect interaction mode based on the hand position being outside of a 3D region associated with the virtual object in the 3D space.

12. The method of claim 1 , wherein determining the interaction mode comprises selecting an indirect interaction mode based on: determining a region of interest based on a gaze direction or a head pose of the user; determining an intersection point where the hand position intersects a 3D region associated with the virtual object in the 3D space; as well as It is determined that the intersection point is outside the region of interest.

13. The method of claim 1, further comprising initiating selection of the virtual object based on the interaction mode being an indirect interaction mode and the gesture being a pinch that occurs when gaze is directed toward the virtual object.

14. The method of claim 1, wherein the interaction mode is an indirect interaction mode, and interpreting the gesture comprises: recognizing that the gesture is a pinch; as well as The virtual object is identified based on a gaze direction occurring during the pinching.

15. The method of claim 1, wherein the interaction mode is an indirect interaction mode, and interpreting the gesture comprises: recognizing that the gesture corresponds to a hand engagement; identifying the virtual object based on a gaze direction occurring during the hand engagement; as well as Hovering of the virtual object is initiated based on the hand engagement.

16. The method according to claim 1, further comprising: Identify gestures; Tracking a position of the gesture in the 3D space; as well as One or more interaction points are identified based on the tracking. The method of claim 16 , wherein identifying the one or more interaction points is based on a gaze direction.

18. The method of claim 1, wherein the electronic device is a head mounted device, and the XR environment is a virtual reality environment or an augmented reality environment.

19. The method of claim 1, further comprising providing data corresponding to the user activity to an application for the application to interpret the user activity based on the data, wherein the data is specific to the interaction mode.

20. The method of claim 19, wherein for the direct interaction mode, the data includes an interaction gesture of the index fingertip as a manipulator gesture and trajectory correction.

21. The method of claim 19, wherein for an indirect interaction mode, the data includes a pinch center of mass as a manipulator gesture and an interaction gesture or user-centric data.

22. A system, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Obtaining a position of a virtual object in an extended reality (XR) environment corresponding to a three-dimensional (3D) space; obtaining a user's hand position in the 3D space based on the sensor data, wherein the hand position is associated with a gesture; determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space; and The gesture is interpreted using an interaction recognition process associated with the determined interaction mode.

23. The system of claim 22, wherein determining the interaction mode comprises selecting whether the interaction mode is direct or indirect based on the user hand position and the position of the virtual object in the 3D space.

24. The system according to claim 22, wherein determining the interaction mode comprises selecting a direct interaction mode in which the virtual element is identified based on an intersection of the hand position and a 3D region associated with the virtual object in the 3D space, and The interaction recognition process is a direct interaction recognition process applied to the gesture.

25. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising: Obtaining a position of a virtual object in an extended reality (XR) environment corresponding to a three-dimensional (3D) space; obtaining a user's hand position in the 3D space based on the sensor data, wherein the hand position is associated with a gesture; determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space; and The gesture is interpreted using an interaction recognition process associated with the determined interaction mode.