Navigation of interactable elements

Abstract

A method includes receiving movement data, moving an indicator of a position on a display in response to the movement data, determining that the indicator is within a threshold distance of an interactable element, and in response to the determination that the indicator is within the threshold distance of the interactable element, moving the indicator toward the interactable element.

Description

NAVIGATION OF INTERACTABLE ELEMENTSCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 563,129, filed March 8, 2024, the disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Users of extended reality devices can have difficulty navigating to interactable elements within a display.SUMMARY

[0003] This document describes systems and techniques for improving how users of a device (e.g., an XR (extended reality) device, or other mixed reality device, computing device, mobile device) can access and navigate a virtual interface (e.g., an XR scene) using a positional indicator (pointer, cursor, etc.) that is displayed in the virtual interface, or a selector that is not displayed in the virtual interface. The systems and techniques use cursor snapping, e.g., virtual magnetic attraction, to position the cursor on interactable regions or elements (e.g., links, buttons, icons, characters, etc.) of a displayed virtual interface.

[0004] Example implementations and techniques are described with respect to the corresponding drawings. Features and aspects of one implementation can be included in other implementations, though those features and aspects may not be specifically shown or described with respect to a given example. In some implementations, features or aspects can be added, omitted or replaced, depending on the particular implementation.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIGs. 1 A to IF are diagrams illustrating examples of navigating a scene using cursor snapping of a cursor, or positional indicator.

[0006] FIGs. 2A and 2B are diagrams illustrating an example of navigating a scene using cursor snapping and hand tracking.

[0007] FIG. 3 is a diagram illustrating example relationships between input modality, threshold distance and cursor snapping actions.

[0008] FIG. 4 is a diagram illustrating an example of a system for implementing cursor movement.

[0009] FIG. 5 is a diagram illustrating another example of displaying a positional indicator in a scene.

[0010] FIGs. 6A to 6C are diagrams illustrating an example of cursor snapping based on a threshold magnetic distance, e.g., a threshold distance in virtual space.

[0011] FIG. 7 is a diagram illustrating an example of scene-dependent cursor snapping in a displayed scene.

[0012] FIGs. 8A, 8B, and 8C are diagrams illustrating interactable element selection and cursor snapping hysteresis.

[0013] FIG. 9 is a block diagram of a computing system that can perform functions described herein.

[0014] FIGs. 10A, 10B, and 10C show an example of a head-mounted device.

[0015] FIG. 10D shows another implementation of a head-mounted device.

[0016] FIG. 11 is a flowchart showing a method performed by a computing system.

[0017] Like reference numbers refer to like elements.DETAILED DESCRIPTION

[0018] Technical problems exist with current XR system-level experiences where it is difficult to navigate a cursor (or other positional indicator) within a virtual interface and / or to properly position the cursor on interactable elements or regions of the virtual interface, which interactable element or region can be difficult to identify, find and select in at least some instances. A user may, for example, move the cursor past the interactable element, or move the cursor back and forth without properly positioning the cursor, due to the lack of accuracy in controlling the cursor within the XR environment.

[0019] The technical solutions described herein address the technical problems by using cursor snapping to reposition a cursor, or other positional indicator, to correspond with an interactable element of a virtual display, such as an XR scene display, an augmented-reality (AR) scene display, a virtual-reality (VR) scene display, and / or a mixed-reality (MR) scene display(collectively hereafter “scene”). Again, as noted above, such cursor snapping can be implemented as simulating magnetic attraction between a positional indicator (cursor) and interactable elements or regions of a scene. When the user moves the cursor to within a threshold distance of the interactable element, the computing system moves the cursor toward the interactable element to assist the user in contacting the interactable element with the cursor.

[0020] This document describes systems and techniques for cursor snapping that respond to intersection of a positional indicator within a scene in virtual space and a distance, e g., in virtual space, of the positional indicator from interactable elements or regions of the scene.

[0021] Different input modalities can be used for positional indicator movement and location within a scene. For instance, head position, eye tracking or gaze direction, an XR / AR / VR / MR controller, and / or hand tracking can be used for positional indicator input.

[0022] Visual display of a positional indicator, which can also be considered a pointer or cursor, can be changed based on the input modality used. For instance, virtual hand rays can be used for hand tracking input or controller input, while a two-dimensional positional indicator can be used for eye tracking or head position tracking input. In some implementations, a pointer, cursor, or other selector is not displayed by the computing system. The selector may have a location maintained by the computing system, without the computing system displaying the location of the selector. In some implementations, the computing system displays the location of the selector when the location of the selector is on or near an interactable target and / or interactable element. The computing system can move the selector toward the interactable element when the interactable element is within a threshold distance of the interactable element, even if the location of the selector is not displayed. The selector otherwise has features of the positional indicator (or “indicator”) described herein, and can be considered and example of a positional indicator.

[0023] Technical advantages of the technical solutions include improved navigation experience: users can easily navigate between interactable elements or regions of a scene, including easily identifying such interactable elements as a result of cursor snapping. Technical advantages of the technical solutions also include ergonomics: by varying magnetic distance, e.g., a threshold distance triggering snapping, as well as visual display of a positional indicator based on corresponding input modality, visual distraction and physical strain, such as eye and / or neck strain, can be reduced. Another technical advantage of the technical solutions is to compensatefor jitter, random noise, and / or small errors in the input method and / or control of the positional indicators, which could otherwise make targeting and / or selecting small targets or interactable elements difficult.

[0024] FIGs. lAto IF are diagrams illustrating examples of navigating a scene 100 using cursor snapping of a cursor 110, which is an example of a positional indicator. In the examples of FIGs. lAto IF, a computing system controls movement and / or positioning of the cursor 110 in the scene 100 based, at least in part, on movement data. Movement data can include input indicating a desire by the user to move the positional indicator or cursor 110. In some implementations, the movement data includes head position data. In some implementations, the movement data includes eye gaze direction data. In some implementations, the movement data includes hand movement data. In some implementations, the movement data includes controller movement data. In some implementations, the movement data includes a combination of two, three, or all four of head position data, eye gaze direction data, hand movement data, and / or controller movement data. The positional indicator, such as a cursor, can be an indicator of a position on the display that presents the scene 100. The computing system can receive and / or process movement data based on head position tracking, e.g., using a six-degree-of-freedom (6- DOF) positional sensor, such as an inertial measurement unit (IMU), a gyroscope, or the like. In some implementations, the computing system receives and / or processes movement data to control cursor movement and / or positioning in the examples of the FIGs. 1 A to IF based on eye tracking using at least one user-facing camera included in the computing system, such as gazetracking cameras. In some implementations, a 6-DOF positional sensor and / or one more user facing cameras can be included in an XR / AR / VR / MR (extended Reality / Augmented Reality / Virtual Reality / Mixed Reality) device, such as a head mounted device. In some implementations, other modalities can be received and / or processed for cursor movement, such as hand tracking (using a world-facing camera), or hand-held controllers. The cursor 110 can indicate an intersection point with the scene 100, e.g., x-y coordinates in virtual space.

[0025] The scene 100 includes a number of scene elements, e.g., displayed elements of the scene 100, which can be referred to as nodes, or panels. In some implementations, such nodes or panels are displayed elements of respective applications in virtual space. For example, the scene 100 includes a node 105 that includes a plurality of interactable elements, or regions, such as interactable element 120. The node 105 also includes non-interactable regions or elements thatare included in the display of the node 105. Interactable elements or regions can be associated with at least one respective action, where an action associated with an interactable element can be triggered by a user interacting with that interactable element, such as by selecting, clicking, hovering, swiping, and the like. The computing system can respond to an interaction with an interactable element by performing an action associated with the interactable element. The action can be a response by the computing system to the interaction with the interactable element. In some examples, the response to the interaction with the interactable element includes launching an application. In some examples, the response to the interaction with the interactable element includes closing an application. In some examples, the response to the interaction with the interactable element includes selecting options within an application. In some examples, the response to the interaction with the interactable element includes changing a presentation of elements within an application. In some examples, the response to the interaction with the interactable element includes retrieving or viewing a document or portion of a document referenced or addressed by a link, magnifying an image, or shooting at an enemy in a game, as non-limiting examples. In comparison, non-interactable regions do not have any associated actions.

[0026] In this example, the cursor 110 is moving toward the interactable element 120, as indicated by arrow D. As the cursor 110 moves within the scene 100, the computing system that renders the scene 100 can monitor a distance of the cursor 110 from the interactable elements of the scene 100. In some implementations, a distance 125 between the cursor 110 and the interactable element 120 is measured between a center of the cursor 110 to a center of the interactable element 120. In some implementations, a distance 125 between the cursor 110 and the interactable element 120 is measured from a portion of a border of the cursor 110 that is closest to the interactable element 120 to a portion of the border of the interactable element 120 that is closest to the cursor 110. In some implementations, a distance 125 between the cursor 110 and the interactable element 120 is measured any other portion of the cursor 110 to the interactable element 120. In some implementations, a center of an interactable element is a point with least variance in distances between the point and portions of the border of the interactable element. In some implementations, a center of an interactive element is a center of mass of the interactable element. In some implementations, a center of the interactable element is an interior portion of the interactable element.

[0027] When the computing system determines that the distance 125 between the cursor 110 and the interactable element 120 is within a threshold distance, e.g., in virtual space, to a closest border of an interactable element, the computing system can cause the cursor 110 to move toward the interactable element. Based on determining that the distance 125 is within the threshold distance, the computing system can move the cursor 110 toward the interactable element independent of input from the user. In some implementations, the computing system accelerates and / or increases a speed of movement of the cursor 110 toward the interactable element, complementing the movement of the cursor that is caused by the user input. The computing system can stop moving the cursor 110 toward the interactable element in response to the cursor 110 contacting the interactable element. Moving the cursor 110 toward the interactable element can include snapping the cursor 110 to that interactable element. For instance, as shown in FIG. IB, the cursor 110 has been snapped to the interactable element 120. In this example, center snapping can be performed, where the cursor 110 is snapped to a center of the interactable element, e g., when the cursor moves to within a threshold distance of the interactable element.

[0028] In some implementations, the scene 100, cursor 110, interactable element 120, and other elements presented by a display are world-locked. In some implementations, the scene 100, cursor 110, interactable element 120, and other elements maintain fixed and / or constant locations with respect to external objects (objects that are outside and / or not connected to the computing system presenting the scene 100). For example, the location of the interactable element can remain constant with respect to an external object while the display moves with respect to the external object, and / or the location of the positional indicator can remain constant with respect to the external object while the display moves with respect to the external object.

[0029] In some implementations, the speed of movement of the cursor 110 toward the interactable element 120 caused by the computing system is a function of the distance 125 between the cursor 110 and the interactable element 120. In some implementations, the speed of movement of the cursor 110 toward the interactable element 120 caused by the computing system is higher as the cursor 110 is closer to the interactable element 120 and lower when the cursor 110 is farther from the interactable element 120. The increase in speed of movement of the cursor 110 toward the interactable element 120 caused by the computing system can, for example, be inversely proportional to the square of the distance 125, causing the speed of movement of the cursor 110 toward the interactable element 120 to be inversely proportional tothe distance 125, simulating a magnetic force of attraction.

[0030] In some implementations, while navigating a scene, a cursor displayed in that scene can be positioned such that it is within a threshold distance of more than one interactable element. In such instances, other factors can be considered when determining which interactable element the cursor is moved toward or snapped to, such as respective distances of the interactable elements from the cursor, direction of travel of the cursor relative to positions of the interactable elements, previous actions while navigating the scene, and / or structure and content of displayed nodes in the scene, as some examples. In some implementations, the computing system moves the cursor 110 toward the interactable element that the cursor 110 is closest to and within the threshold distance of. In some implementations, the computing system moves the cursor 110 toward an interactable element based on the cursor being within the threshold distance of the interactable element and already moving toward the interactable element based on the movement data. Moving the cursor 110 toward an interactable element based on the cursor 110 being within the threshold distance of the interactable element and already moving toward the interactable element based on the movement data prevents the computing system from moving the cursor 110 toward an interactable element that the user is moving the cursor 110 away from.

[0031] FIGs. 1C and ID further illustrate center snapping of the cursor 110. For instance, in FIG. 1C, the cursor 110 is shown as being center-snapped to an interactable element 130. In FIG. ID, the cursor 110 is shown as being center-snapped to an interactable element 140. The cursor snapping illustrated in FIGs. 1C and ID can occur as a result of movement of the cursor 110 in the scene 100, e.g., based on positional information for a corresponding input modality that controls the cursor 110 or positional indicator, such as head tracking, eye tracking, hand tracking, and / or controller input. The cursor snapping can occur in response to the input modality moving the cursor 110 or positional indicator to within a threshold distance of the interactable element 130. Center-snapping can include the computing system moving the cursor 110 to a center of an interactable element. The computing system can move the cursor 110 to the center of the interactable element based on the distance between the cursor 110 and the interactable element being within the threshold distance, based on the interactable element to which the cursor 110 is center snapped being the closest interactable element to the cursor 110, and / or based on the cursor 110 moving toward the interactable element in response to movement data received and / or processed by the computing system.

[0032] In some implementations, the computing system moves the cursor to a center or border of a smaller shape (such as a rounded rectangle) embedded within an interactable element. Moving the cursor to the center or border of the smaller shape within the interactable element can improve the user experience when the interactable area of the interactable element is smaller than a bounding box of the interactable element.

[0033] FIGs. IE and IF illustrate an example of interaction with an interactable element 150 in the scene 100. As shown in FIG. IE, the cursor 110 has been snapped to an interactable element 150 of the node 105, e.g., as a result of movement of the cursor 110 to within a threshold distance of a border of the interactable element 150. As also shown in FIG. IE, a hand 115 of a user is performing a pinch gesture to interact with the interactable element 150. In this example, display of the hand 115 in the scene 100 can be based on image data of a user’s hand acquired by a world-facing camera of the computing system such as an XR / AR / VR / MR device. As shown in FIG. IF, in response to the pinch gesture of FIG. IE, the node 105 (not shown in FIG. IF) in the scene 100 has been closed.

[0034] FIGs. 2A and 2B are diagrams illustrating an example of navigating a scene 200 using cursor snapping and hand tracking. In this example, the scene 200 includes the same nodes as the scene 100. In other implementations, the computing system can display other nodes in the scene. In the example of FIGs. 2A and 2B, the computing system processes hand position data and / or hand movement to determine cursor movement. For instance, the computing system tracks movement of a hand 215 using a world-facing camera and the computing system displays a hand ray 210 as a cursor. The hand ray 210 includes a positional indicator 210a and a ray 210b, both of which move in correspondence with detected movement of the hand 215. The positional indicator 210a can indicate an intersection point, e.g., x-y coordinates in virtual space, of the ray 210b with the scene 200 presented by the computing system.

[0035] In this example, the hand ray 210, in correspondence with movement of the hand 215, is moving toward an interactable element 240, as indicated by the arrow D. Once the positional indicator 210a of the hand ray 210 is positioned such that snapping to the interactable element 240 is triggered, such as based on a distance of the positional indicator 210a from the interactable element 240 being within the threshold distance, both the positional indicator 210a and the ray 210b are snapped to the closest border of the interactable element 240, as shown in FIG. 2B. Such snapping, which is referred to herein as closest border snapping or bordersnapping, can be triggered based on various criteria, such as those discussed herein. For instance, once the positional indicator 210a is within a threshold distance, e.g., a magnetic distance, of a border (edge) of the interactable element 240, closest border snapping can be triggered. In some implementations, additional and / or other criteria can be evaluated to determine whether or not cursor snapping should occur, and to which interactable element of a given scene that the displayed cursor should be snapped to.

[0036] FIG. 3 is a diagram illustrating example relationships between input modality, threshold distance, and cursor snapping actions. In some implementations, the threshold distance that determines whether the computing system moves the positional indicator toward the interactable element is based on the input modality. In FIG. 3, the left-most column indicates various input modalities, which include a mouse (or other pointing device, such as a trackpad), head gaze (head position), a controller (e.g., XR / AR / VR / MR controller), hand tracking (e.g. hand ray), and eye tracking (e.g. eye gaze determined by one or more gaze-tracking cameras). The center column indicates normalized examples of distance thresholds, e.g., in virtual space, for triggering movement of the positional indicator such as cursor snapping actions for each of the input modalities, while the right-most column indicates the movement such as the cursor snapping action that is associated with each input modality. Snapping can include moving the cursor or positional indicator directly to the closest border or center of the interactable element without showing the positional indicator in an intermediate position between the closest border or center and the location of the positional indicator when the determination was made to snap the cursor to the interactable element, rather than gradually moving the cursor or positional indicator toward the interactable element.

[0037] As shown in FIG. 3, in this example, no cursor snapping is triggered when a mouse is used, because a computer mouse can be moved accurately without need for assistance in moving a positional indicator over an interactable element. Accordingly, no threshold distance is associated with use of a computer mouse. Further in this example, for head gaze, controller and hand tracking input modalities, closest border snapping is used, which can reduce visual distraction for these input modalities by moving the positional indicator to a closest border of the interactable element and thereby reducing movement of the positional indicator compared to moving the positional indicator to a center of the interactable element. For head gaze (head position) input modalities, a larger threshold distance can be used to reduce head movementwhen navigating a scene. This, in turn, can reduce neck strain associated with head movement as input for facilitating cursor movement. As further shown in FIG. 3, for eye tracking as an input modality, center snapping is used, which can minimize visual distraction and improve identification of interactable elements in a displayed scene. Also, the normalized distance threshold can be selected to reduce eye strain.

[0038] In some implementations, the computing system varies the threshold distance dynamically based on current circumstances and / or recent events. In some implementations, the computing system can vary the threshold distance based on light conditions, such as increasing the threshold distance in lower light conditions (which may increase hand tracking jitter) and decreasing the threshold distance in higher light conditions. In some implementations, the computing system can increase the threshold distance based on a threshold number of small movements of the positional indicator within a threshold time period, which may indicate that the user has been attempting to move the positional indicator onto a positional indicator.

[0039] FIG. 4 is a diagram illustrating an example of a system 400 for implementing cursor movement. Cursor movement can include movement of the positional indicator and / or cursor snapping. The system 400 includes an application 410, a composer 420 (which can be included in an XR / AR / VR / MR operating system), and positional indicator movement 430. Tracking and pointer snapping is an example of positional indicator movement. The application 410 can be one of a plurality of applications with corresponding nodes of a scene that is displayed by the system 400. That is, the composer 420 can implement display of nodes in an XR / AR / VR / MR environment, including displaying nodes associated with applications, such as the application 410.

[0040] The application 410 includes logic 412, which can include executable code of the application 410. The application 410 also includes a library 414 that is used by the logic 412 and / or operates in conjunction with the logic 412 during execution of the application 410. The library 414 includes views 416 of the application 410, which can include interactable elements and / or other visual elements associated with the application 410. The library 414 also includes a surface generator 418, which can be configured to provide information for displaying one or more nodes of the application 410 in virtual space, e.g., two-dimensional panels and / or three- dimensional elements.

[0041] In this example, the composer 420 can communicate with the application 410 to receiveinformation for display of one or more nodes of the application 410. The composer 420 can also communicate with the application 410 to identify interactable elements within those one or more nodes. A surface handler 422 of the composer 420 can receive information for views of the application 410 (and other applications), and then aggregate and manage display of those views. For instance, the surface handler 422 can, based on information provided by the application 410 and other applications, determine respective locations of interactable elements of the application 410 in virtual space, e.g., x-y coordinates in an associated scene including the views (nodes).

[0042] The surface handler 422 can generate a local state 424, which is a representation of content of the surface of a given view or node in its own XR / AR / VR / MR space (e.g., x, y coordinates corresponding with the view). A global state 436 of the positional indicator movement 430 can be a representation of various surfaces for views or nodes in world space (e.g., three-dimensional (3D) positions and scale corresponding to world dimensions), such as for positioning display of virtual content with displayed real world elements of an XR / AR / VR / MR display.

[0043] In this example, positional indicator movement 430 is responsible for handling input information (positional indicator input via input pipeline 432) and dispatching events, e.g., collisions between a positional indicator and an interactable element, such as determining that a positional indicator is within a threshold distance of an interactable element or multiple interactable elements. That is, positional indicator movement 430 is responsible for handling collisions between a positional indicator (e.g., hand-rays) and different virtual surfaces displayed in world-space in an XR / AR / VR / MR displayed scene 434.

[0044] FIG. 5 is a diagram illustrating another example of displaying a positional indicator in a scene 500. As shown in FIG. 5, a positional indicator of a hand-ray 510 is located within and / or pointing into a virtual panel 505, where the virtual panel 505 is a surface containing a view of an application. In some implementations, a displayed surface of the virtual panel 505 has a world- locked location, maintaining a fixed position and / or constant location with respect to an external object and / or surrounding physical environment while the display presenting the virtual panel moves, and is rendered by a display device (e.g., an XR / AR / VR / MR headset). In some implementations, 6-DOF positional tracking can be used to cause spatializing of the virtual panel 505.

[0045] In the example of FIG. 5, a panel collider 540 can determine if the positional indicatorof the hand-ray 510 is aimed at the virtual panel 505 (e.g., is with the virtual panel 505). In some implementations, the panel collider 540 is used to handle / determine occurrence of intersection 550 between the positional indicator of the hand-ray 510 and the virtual panel 505. In some implementations, a location of the intersection 550 is represented in Cartesian coordinates such as an ‘x’ value representing a horizontal position and a ‘y’ value representing a vertical position. Once an intersection 550, which can be considered a collision, between the positional indicator of the hand-ray 510 and the virtual panel 505 is detected, an internal state of the virtual panel 505 can be analyzed to find interactable areas used for cursor snapping functionality.

[0046] FIGs. 6A to 6C are diagrams illustrating an example of cursor snapping based on a threshold magnetic distance, e.g., a threshold distance in virtual space. In this example, a panel 605 (or node) includes three interactable elements or regions, interactable element 650A, interactable element 650B and interactable element 650C. Based on information from an input modality, such as those described herein, an intersection between a cursor 610 and the panel 605, e g., a two-dimensional panel, is determined and displayed in the panel 605, e.g., within a larger scene in virtual space. FIG. 6B illustrates respective points on the borders of interactable element 650A, interactable element 650B and interactable element 650C that are closest to the intersection with the panel 605 indicated by the cursor 610. For instance, point 620 is the closest point of a border of interactable element 650Ato the cursor 610, point 630 is the closest point of a border of interactable element 650B to the cursor 610, and point 640 is the closest point of a border of interactable element 650C to the cursor 610.

[0047] Distances between the cursor 610 and the interactable elements 650A, 650B, 650C can be based on distances between the cursor 610 and the points 620, 630, 640. A distance between the cursor 610 and the interactable element 650Acan be based on a distance 660A between the point 620 and the cursor 610. A distance between the cursor 610 and the interactable element 650B can be based on a distance 660B between the point 630 and the cursor 610. A distance between the cursor 610 and the interactable element 650C can be based on a distance 660C between the cursor 610 and the interactable element 650C. In this example, the distance 660C is less than and / or shorter than the distances 660A and 660B because point 640 is the closest of points 620, 630 and 640 to the cursor 610. The cursor 610 is closer to the interactable element 650C than interactable element 650A or interactable element 650B. Accordingly, even if points 620, 630 and 640 are all within a threshold distance to the cursor 610, as shown in FIG. 6C, thecursor 610 moves toward and / or snaps to interactable element 650C. As described herein, e.g., with respect to FIG. 3, the threshold distance can correspond with an input modality being used for movement of the cursor 610. Depending on the particular implementation and / or the associated input modality, movement of the cursor 610 such as center snapping or closest border snapping to interactable element 650C can be performed by the computing system.

[0048] FIG. 7 is a diagram illustrating an example of scene-dependent cursor movement in a displayed scene 700. Cursor snapping is an example of cursor movement. As shown in FIG. 7, the displayed scene 700 includes a virtual panel (node 720), e.g., a browser. The displayed scene 700 also includes a node 730 (e.g., for an application), and a node 740, e.g., a taskbar. As shown in FIG. 7, even when there is not a collision between a positional indicator, e g., a hand-ray 710, and any of the nodes 720, 730 and 740, evaluation of the scene 700 can still be performed for implementing cursor movement such as cursor snapping. For instance, in this example, a determination can be made that a border of an interactable element 730C of the node 730 is within a threshold distance of the positional indicator. This determination can then trigger snapping of the positional indicator to the interactable element 730C of the node 730 (e.g., closest-border snapping or center snapping). Such operation can improve ergonomics of cursor snapping, e.g., allowing an interactable element to be more readily identified, even in the absence of a positional indicator and virtual panel collision.

[0049] FIGs. 8A, 8B, and 8C are diagrams illustrating interactable element selection and cursor snapping hysteresis. In this example, a panel 805 (node) includes two interactable elements, interactable element 805 A and interactable element 805B. In an initial state shown in FIG. 8 A, a cursor 810 is located outside the borders of both interactable element 805A and interactable element 805B. In a subsequent state of FIG. 8B, the cursor 810 is within the borders of interactable element 805A, after being moved to, snapped to, and / or navigated within the borders of interactable element 805 A. As shown in FIG. 8B, in the state of FIG. 8B, after the cursor 810 enters interactable element 805A, the virtual margins of interactable element 805A increase as compared to the initial state shown in FIG. 8A. The cursor 810 can then be moved toward and / or enter interactable element 805B, and snapped to interactable element 805B, as shown in FIG. 8C. As a result, the virtual margins of B increase, while the virtual margins of interactable element 805A decrease to those of the initial state shown in FIG. 8A. In this example, hysteresis can be applied when triggering snapping functions and / or movement functions. For instance, a differentthreshold distance can be used when moving and / or snapping between interactable element 805A and interactable element 805B, as compared to a threshold distance used when moving and / or snapping to interactable element 805A or interactable element 805B from the initial state shown in FIG. 8A. In some implementations, a hysteresis threshold distance can be less than a threshold distance for moving or snapping to an element from an initial state, such as the initial state shown in FIG. 8A.

[0050] FIG. 9 is a block diagram of a computing system 900 that can perform functions described herein. The computing system 900 can be implemented by a head-mounted device such as the smartglasses shown in FIGs. 10A, 10B, and 10C or goggles shown in FIG. 10D, by a computing device such as a server in communication with the head-mounted device, by a computing device such as a desktop computer, laptop computer, tablet, or smartphone, or have functions distributed between multiple such computing devices.

[0051] The computing system 900 can include a movement data processor 902. The movement data processor 902 can process and / or determine movement data. The movement data processor 902 can determine movement data based on input received by the computing system 900. The movement data processor 902 can receive the input by various input modalities, such as head position data, eye gaze direction data, hand movement data, or controller movement data, as nonlimiting examples. In some implementations, the input processed by the movement data processor 902 includes head position data. The head position data can indicate a position of a head of a user wearing the computing system 900. The head position data can be based on accelerometer data, inertial measurement unit (IMU) data, and / or camera data capturing a scene, as non-limiting examples. In some implementations, the input processed by the movement data processor 902 includes eye gaze direction data. Eye gaze direction data can be captured by one or more gaze-tracking cameras included in the computing system 900. The eye gaze direction data can indicate a direction that eyes of the users are pointing and / or looking at. In some implementations, the input processed by the movement data processor 902 includes hand movement data. The hand movement data can include gestures and / or pointing indications. The pointing indications can indicate a location on the display at which the user would like to place the cursor and / or positional indicator. The hand movement data can be captured by one or more cameras included in the computing system 900. In some implementations, the input processed by the movement data processor 902 includes controller movement data. The controller can includea computer mouse, a trackpad, or a trackball, as non-limiting examples. The controller can indicate a direction of movement desired by the user.

[0052] The computing system 900 can include a positional indicator controller 904. The positional indicator controller 904 can control position and / or movement of the positional indicator such as a cursor. The positional indicator controller 904 can control the position and / or movement of the positional indicator based on the input processed by the movement data processor 902 and / or proximity of the positional indicator to an interactable element.

[0053] The positional indicator controller 904 can include a distance determiner 906. The distance determiner 906 can determine a distance between the positional indicator and one or more interactable elements. The distance determiner 906 can determine the distance based on a distance between a center of the positional indicator and a center of the interactable element, based on a closest portion of a border of the positional indicator to a closest portion of a border of the interactable element, or from a center of one of the positional indicator or interactable element to a closest portion of a border of the other of the positional indicator or interactable element, as non-limiting examples. The distance determiner 906 can measure the distance in pixels or in units such as inches or centimeters at a depth to which the computing system 900 presents the positional indicator and interactable element to the user, as non-limiting examples.

[0054] The computing system 900 can include a threshold comparator 908. The threshold comparator 908 can compare the distance of the positional indicator from the interactable element determined by the distance determiner 906 to a threshold distance, to determine whether to move the positional indicator toward the interactable element. In some implementations, the threshold distance is based on the input modality. The threshold comparator 908 can apply different threshold distances for different input modalities. FIG. 3 shows examples of different threshold distances for different input modalities.

[0055] After determining the threshold distance, the threshold comparator 908 can compare the distance determined by the distance determiner 906 to the threshold distance to determine whether the distance satisfies the threshold distance. In some implementations, the distance satisfies the threshold distance if the distance is less than or equal to the threshold distance. In some implementations, the distance satisfies the threshold distance if the distance is less than the threshold distance.

[0056] The positional indicator controller 904 can include a direction determiner 910. Thedirection determiner 910 can determine a direction that the positional indicator is or was moving within a predetermined time period. The direction determiner 910 can determine whether the positional indicator was moving toward an interactable element. In some implementations, the direction determiner 910 can determine whether the positional indicator is or was moving toward an interactable element that is closest to the positional indicator.

[0057] The positional indicator controller 904 can include a movement controller 912. The movement controller 912 can control movement of the positional indicator. The movement controller 912 can cause the positional indicator to move based on movement data processed by the movement data processor 902. The movement controller 912 can cause the positional indicator to move toward an interactable element based on the threshold comparator 908 determining that the distance of the positional indicator from the interactable element satisfying the distance threshold, based on the interactable element being a closest interactable element to the positional indicator, and / or based on the direction determiner 910 determining that the positional indicator is moving toward the interactable element.

[0058] The computing system 900 can include a hysteresis controller 914. The hysteresis controller 914 can provide a hysteresis feature by causing the positional indicator to remain snapped to or on or along an interactable element when the positional indicator would otherwise move toward a different interactable element. As described with respect to FIGs. 8A, 8B, and 8C, after the positional indicator has snapped to, moved alongside, and / or moved onto an interactable element, and / or after or in response to the movement controller 912 moving the positional indicator toward the interactable element based on the distance between the positional indicator and the interactable element satisfying the threshold distance, the hysteresis controller 914 can implement the hysteresis feature. The hysteresis feature can include enlarging, and / or expanding the borders of, the interactable element to which the positional indicator is snapped, moved alongside, and / or moved onto. Enlarging and / or expanding the borders of the interactable element delays movement of the positional element toward another interactable element until the positional indicator has moved further toward the other interactable element.

[0059] The computing system 900 can include a selection processor 916. The selection processor 916 can process and / or determine selections of interactable elements. The selection processor 916 can process or determine selections of interactable elements based on the positional indicator being snapped to, along, or on the interactable element and / or based onreceiving another input, such as a predetermined audible input, a predetermined hand gesture, a predetermined head movement, or a predetermined eye movement, as non-limiting examples. A selection can include and / or be equivalent to an input such as pressing or selecting a button, inputting a character or word, selecting an item from a menu, or dragging an object, as nonlimiting examples.

[0060] The computing system 900 can include an action processor 918. The action processor 918 can cause the computing system 900 to perform an action based on a selection of an interactable element determined by the selection processor 916. The action processor 918 can cause the computing system 900 to, in response to the selection processor 916 determining that the user has selected an interactable element, launch an application, select an item or object within an application, or move an object, as non-limiting examples.

[0061] The computing system 900 can include at least one processor 920. The at least one processor 920 can execute instructions, such as instructions stored in at least one memory device 922, to cause the computing system 900 to perform any combination of methods, functions, and / or techniques described herein.

[0062] The computing system 900 can include at least one memory device 922. The at least one memory device 922 can include a non-transitory computer-readable storage medium. The at least one memory device 922 can store data and instructions thereon that, when executed by at least one processor, such as the processor 920, are configured to cause the computing system 900 to perform any combination of methods, functions, and / or techniques described herein.Accordingly, in any of the implementations described herein (even if not explicitly noted in connection with a particular implementation), software (e g., processing modules, stored instructions) and / or hardware (e.g., processor, memory devices, etc.) associated with, or included in, the computing system 900 can be configured to perform, alone, or in combination with the computing system 900, any combination of methods, functions, and / or techniques described herein. The at least one memory device 922 can include a gesture library. The gesture library can include predetermined gestures. The predetermined gestures can include hand formations and / or movements and associated actions that causes the positional indicator controller 904 to move the positional indicator.

[0063] The computing system 900 may include at least one input / output node 924. The at least one input / output node 924 may receive and / or send data, such as from and / or to, a server, and / ormay receive input and provide output from and to a user. The input and output functions may be combined into a single node, or may be divided into separate input and output nodes. The input / output node 924 can include a microphone, a camera (such as a front-facing camera or gaze-tracking camera), an IMU, a display, a speaker, one or more buttons, and / or one or more wired or wireless interfaces for communicating with other computing devices such as a headmounted device and / or a computing device in communication with the head-mounted device.

[0064] FIGs. 10A, 10B, and 10C show an implementation of a head-mounted device 1000. The head-mounted device 1000 is an example of the computing system 900. As shown in FIGs. 10A, 10B, and 10C, the head-mounted device 1000 includes a frame 1002. The frame 1002 includes a front frame portion defined by rim portions 1003 A, 1003B surrounding respective optical portions in the form of lenses 1007A, 1007B, with a bridge portion 1009 connecting the rim portions 1003 A, 1003B. Arm portions 1005 A, 1005B are coupled pivotably or rotatably coupled, to the front frame by hinge portions 1012A, 1012B at the respective rim portion 1003 A, 1003B. In some implementations, the lenses 1007A, 1007B may be corrective / pre scription lenses. In some implementations, the lenses 1007A, 1007B may be an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters. Displays 1010A, 1010B may be coupled in a portion of the frame 1002. In the implementation shown in FIG. 10B, the displays 1010A, 1010B are coupled in the arm portions 1005 A, 1005B and / or rim portions 1003 A, 1003B of the frame 1002. In some implementations, the headmounted device 1000 can also include an audio output device 1016 (such as one or more speakers), an illumination device 1018, at least one processor 1011 (which is an example of the processor 920 shown and described with respect to FIG. 9), an outward-facing image sensor 1014, and gaze-tracking cameras 1019A, 1019B that can capture images of eyes of a user to track a gaze of the user. In some implementations, the head-mounted device 1000 may include a see-through near-eye display. The displays 1010A, 1010B may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world through the lenses 1007A, 1007B, next to content (such as digital images, user interface elements, virtual content, and the like) generated by the displays1010A, 1O1OB. In some implementations, waveguide optics may be used to depict content on the displays 1010A, 1010B via outcoupled light 1020A, 1020B. The images projected by the displays 1010A, 1010B onto the lenses 1007A, 1007B may be translucent, allowing the user to see the images projected by the displays 1010A, 1010B as well as physical objects beyond the head-mounted device 1000.

[0065] FIG. 10D shows another implementation of the head-mounted device 1000. In this implementation, the head-mounted device 1000 is in goggle form, with a display included in the head-mounted device 1000 and a housing supporting the display enclosing the face and / or eyes of the user. This implementation of the head-mounted device 1000 can support a virtual reality (VR) experience in which the user sees only what is presented by the display included in the head-mounted device 1000.

[0066] FIG. 11 is a flowchart showing a method 1100 performed by a computing system. The method 1100 can be performed by the computing system 900. The method 1100 includes receiving movement data (1102). The method 1100 includes moving an indicator (1104). Moving the indicator (1104) includes moving an indicator of a position on a display in response to the movement data. The method 1100 includes determining whether a distance is within a threshold (1106). Determining whether the distance is within the threshold (1106) includes determining that the indicator is within a threshold distance of an interactable element. The method 1100 includes moving the indicator (1108). Moving the indicator (1108) includes, in response to the determination that the indicator is within the threshold distance of the interactable element, moving the indicator toward the interactable element.

[0067] In some implementations, the method 1100 further includes determining that a direction of the movement of the indicator in response to the movement data is toward the interactable element. Moving the indicator toward the interactable element is in response to the determination that the distance is within the threshold distance and the determination that the direction of the movement of the indicator in response to the movement data is toward the interactable element.

[0068] In some implementations, a location of the interactable element remains constant with respect to an external object while the display moves with respect to the external object. A location of the indicator remains constant with respect to the external object while the display moves with respect to the external object.

[0069] In some implementations, a speed of moving the indicator toward the interactableelement in response to the determination that the distance is within the threshold distance is a function of distance of the indicator from the interactable element.

[0070] In some implementations, moving the indicator toward the interactable element includes snapping the indicator to a location along a border of the interactable element closest to the indicator.

[0071] In some implementations, the method 1100 further includes determining a center of the interactable element. Moving the indicator toward the interactable element includes snapping the indicator to the center of the interactable element.

[0072] In some implementations, the method 1100 further includes expanding a location of the interactable element in response to either moving the indicator to the location along the border of the interactable element closest to the indicator or snapping the indicator to the center of the interactable element.

[0073] In some implementations, the determination that the indicator is within the threshold distance of the interactable element is performed in response to determining that the indicator is located on a panel, the panel including the interactable element.

[0074] In some implementations, the interactable element is a first interactable element, the method further includes determining that the indicator is within the threshold distance of a second interactable element, the method 1100 further includes determining that the indicator is closer to the first interactable element than the second interactable element, and moving the indicator toward the first interactable element is performed in response to the determination that the indicator is closer to the first interactable element than the second interactable element.

[0075] In some implementations, the method 1100 further includes receiving a selection of the interactable element after moving the indicator toward the interactable element.

[0076] In some implementations, the method 1100 further includes performing an action in response to receiving a selection of the interactable element.

[0077] In some implementations, the threshold distance is based on a modality of receiving the movement data.

[0078] In some implementations, the movement data includes head position data. In some implementations, the movement data includes eye gaze direction data. In some implementations, the movement data includes hand movement data. In some implementations, the movement data includes controller movement data.

[0079] Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0080] Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (applicationspecific integrated circuit).

[0081] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0082] To provide for interaction with a user, implementations may be implemented on acomputer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0083] Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0084] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the embodiments.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving movement data; moving an indicator of a position on a display in response to the movement data; determining that the indicator is within a threshold distance of an interactable element; and in response to the determination that the indicator is within the threshold distance of the interactable element, moving the indicator toward the interactable element.

2. The method of claim 1, further comprising: determining that a direction of the movement of the indicator in response to the movement data is toward the interactable element, wherein moving the indicator toward the interactable element is in response to the determination that the distance is within the threshold distance and the determination that the direction of the movement of the indicator in response to the movement data is toward the interactable element.

3. The method of either of claims 1 or 2, wherein: a location of the interactable element remains constant with respect to an external object while the display moves with respect to the external object; and a location of the indicator remains constant with respect to the external object while the display moves with respect to the external object.

4. The method of any of claims 1 to 3, wherein a speed of moving the indicator toward the interactable element in response to the determination that the distance is within the threshold distance is a function of distance of the indicator from the interactable element.

5. The method of any of claims 1-4, wherein moving the indicator toward the interactable element includes snapping the indicator to a location along a border of the interactable element closest to the indicator.

6. The method of any of claims 1-4, further comprising: determining a center of the interactable element, wherein moving the indicator toward the interactable element includes snapping the indicator to the center of the interactable element.

7. The method of either of claims 5 or 6, further comprising expanding a location of the interactable element in response to either moving the indicator to the location along the border of the interactable element closest to the indicator or snapping the indicator to the center of the interactable element.

8. The method of any of the preceding claims, wherein the determination that the indicator is within the threshold distance of the interactable element is performed in response to determining that the indicator is located on a panel, the panel including the interactable element.

9. The method of any of the preceding claims, wherein: the interactable element is a first interactable element; the method further includes determining that the indicator is within the threshold distance of a second interactable element; the method further includes determining that the indicator is closer to the first interactable element than the second interactable element; and moving the indicator toward the first interactable element is performed in response to the determination that the indicator is closer to the first interactable element than the second interactable element.

10. The method of any of the preceding claims, further comprising receiving a selection of the interactable element after moving the indicator toward the interactable element.

11. The method of any of the preceding claims, further comprising performing an action in response to receiving a selection of the interactable element.

12. The method of any of the preceding claims, wherein the threshold distance is based on a modality of receiving the movement data.

13. The method of any of the preceding claims, wherein the movement data includes at least one of: head position data; eye gaze direction data; hand movement data; or controller movement data.

14. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, are configured to cause a computing system to perform the method of any of the preceding claims.

15. A computing system comprising: at least one processor; and a computer-readable storage medium comprising instructions stored thereon that, when executed by the at least one processor, are configured to perform the method of any of claims 1- 13.

Images