Mixed eye-tracking in wearable devices

Abstract

A method comprising: determining the gaze characteristics of a user of a wearable device; determining the head movements of the user; determining a gesture based on the gaze characteristics, head movements, and the correlation between the gaze characteristics and head movements; selecting UI elements of a headlock user interface (UI) that operates on the wearable device based on the gesture; and triggering a UI action based on the selected UI elements.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application is a continuation of and claims priority to U.S. Patent Application No. 18 / 477,188, filed on September 28, 2023, the disclosure of which is hereby incorporated by reference in its entirety into the present specification.

[0002] Embodiments relate to eye gaze tracking for wearable devices.

Background Art

[0003] Wearable devices including augmented reality (AR) glasses allow users to quickly access the functionality of general-purpose computers throughout their day during any activity without interruption, by visually presenting content within the user's field of view.

Summary of the Invention

[0004] Embodiments relate to combining head movement and eye-gaze to designate elements (e.g., icons, menus, etc.) to interact with a user interface (UI) of a wearable device (e.g., AR headset, AR glasses, etc.). More specifically, the present subject matter relates to generating gestures based on the correlation between eye movements and head movements associated with UI elements. The gestures can be used to make discrete UI selections including, for example, selecting an icon, selecting a menu item, looking away from the UI (which may trigger closing the UI), and the like.

[0005] In a general embodiment, a device, system, non-temporary computer-readable medium (which stores computer-executable program code that can be run on a computer system), and / or method may perform a process by a method that includes determining the gaze characteristics of a user of a wearable device, determining the user's head movements, determining gestures based on the gaze characteristics, head movements, and correlations between the gaze characteristics and head movements, selecting user interface (UI) elements of a headlock UI to operate on the wearable device based on the gestures, and triggering UI actions based on the selected UI elements.

[0006] Exemplary embodiments will be better understood from the following detailed description and accompanying drawings. In the drawings, similar elements are represented by the same reference numerals, which are given for illustrative purposes only and are not limiting to the exemplary embodiments. [Brief explanation of the drawing]

[0007] [Figure 1A] This section provides a schematic representation of a scenario for performing mixed eye-tracking, using exemplary embodiments. [Figure 1B] This section provides a schematic representation of a scenario for performing mixed eye-tracking, using exemplary embodiments. [Figure 1C] This section provides a schematic representation of a scenario for performing mixed eye-tracking, using exemplary embodiments. [Figure 1D] This section provides a schematic representation of a scenario for performing mixed eye-tracking, using exemplary embodiments. [Figure 1E] This section provides a schematic representation of a scenario for performing mixed eye-tracking, using exemplary embodiments. [Figure 2] A block diagram of the data flow of a wearable device according to an exemplary embodiment is shown. [Figure 3A] A graph of the gaze position in eye-tracking according to an exemplary embodiment is shown. [Figure 3B] A graph of head tracking motion position is shown according to an exemplary embodiment. [Figure 3C] A graph of eye-tracking movement in an exemplary embodiment is shown. [Figure 3D] A graph showing the gaze position and gaze movement in an exemplary embodiment of eye tracking is shown. [Figure 3E] Graphs of gaze position, gaze movement, and head movement in an exemplary embodiment of eye tracking are shown. [Figure 4-1] A-C show block diagrams of parts of the user interface (UI) design of a wearable device, including sectors, according to exemplary embodiments. [Figure 4-2] D to F show block diagrams of a portion of the user interface (UI) design for a wearable device, including sectors and UI elements, according to an exemplary embodiment. [Figure 5] A method for generating gesture data according to an exemplary embodiment is shown. [Figure 6A] This shows an example of a head-mounted wearable device worn by a user. [Figure 6B] Figure 6A is a front view of an exemplary head-mounted wearable device. [Figure 6C] Figure 6A is a rear view of an exemplary head-mounted wearable device. [Figure 7] This figure shows an exemplary apparatus for performing mixed eye-tracking according to an exemplary embodiment. [Figure 8] A block diagram of a method for operating a wearable device according to an exemplary embodiment is shown. [Modes for carrying out the invention]

[0008] It should be noted that these drawings illustrate the general characteristics of the methods and / or structures used in a particular exemplary embodiment and are intended to supplement the descriptive descriptions provided below. However, these drawings are not to scale and may not accurately reflect the exact structural or performance characteristics of any given embodiment and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiment. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various drawings is intended to indicate the existence of similar or identical elements or features.

[0009] Users of computing devices may require one or more peripheral devices to interact with the computing device's software (e.g., user interface) and / or hardware (e.g., camera). For example, a typical computing device might use a mouse, keyboard, or touch interface. However, a technical challenge associated with wearable devices is that they may not have the necessary interfaces to connect a mouse, keyboard, touch interface, etc. to the wearable device. Furthermore, carrying peripheral devices can degrade the user experience. A technical solution might be to perform the functions of peripheral devices (e.g., functions similar to those of peripheral devices on a computing device) on the wearable device based on the user's gaze and / or head movements. The technical effect of performing peripheral device functions on a wearable device based on the user's gaze and / or head movements is that the user can interact with the wearable device's software (e.g., user interface) and / or hardware (e.g., camera) without using physical peripheral devices, resulting in an improved user experience.

[0010] The embodiments relate to interacting with the user interface (UI) of a wearable device (e.g., an AR headset, AR glasses, etc.) by specifying elements (e.g., icons, menus, etc.) by gaze and / or head movements. More specifically, the subject relates to making discrete UI selections using gaze and / or head movements, including, for example, selecting icons, selecting menu items, and diverting gaze from the UI (which may trigger, for example, closing the UI). Selected UI elements can trigger actions associated with the UI. In exemplary embodiments, the UI design may include sectors (e.g., blocks, N×N blocks, quadrants, etc.) containing UI elements. Sectors can improve the precision of UI element selection. In some embodiments, the subject relates to generating gestures based on differential eye movements within the UI. For example, gestures can be generated based on eye movements from a first sector to a second sector.

[0011] As discussed above, wearable devices, including augmented reality (AR) glasses, can enable users to quickly access the functions of a general-purpose computer by visually presenting content within the user's field of view. However, a problem with current wearable technology is the lack of a mechanism for hands-free interaction with the UI without accessory devices or user interruptions (e.g., user touching the wearable device). Therefore, there is a need for novel technologies that enable intuitive, uninterrupted, and reliable interaction with user interface (UI) elements. Interaction with UI elements should not be affected by movement (e.g., head movement independent of the UI) or motion (e.g., user walking). An exemplary embodiment solves this problem by providing a system for specifying UI elements using a gesture interface based on gaze and / or head movement.

[0012] Exemplary embodiments include an interaction system for a head-locked UI. The system can process inputs including head movements (e.g., 3-degree-of-freedom (DoF) head movements, acceleration, gyro, etc.) and gaze characteristics (e.g., saccades, gaze vectors, timestamps associated with gaze vectors, and / or velocities associated with gaze vectors, and / or the like), as well as a UI having a sectorized (e.g., N×N blocks or quadrants) head-locked visual UI. The system can be controlled based on the user's natural interaction with the wearable device, including moving the gaze toward visual UI elements. In some embodiments, the gaze can be correlated with corresponding head movements (e.g., to confirm detection of a gaze gesture) to generate gestures for making discrete UI selections. In addition to the benefits of user experience, the system can relax the accuracy and calibration requirements of gaze tracking technology, thereby resulting in cost reduction compared with conventional gaze tracking systems.

[0013] FIGS. 1A to 1E show schematic representations of scenarios for performing hybrid gaze tracking according to exemplary embodiment(s). As shown in FIGS. 1A to 1E, a user 105 may wear a wearable device 110 (e.g., an AR headset, AR glasses, etc.). The wearable device 110 may have a display area 115 on the lens (e.g., one or more lenses) of the wearable device 110. The display area 115 may be configured to display a user interface (UI) 120. The UI 120 may include a plurality of UI elements 125, 130, 135, 140.

[0014] Some embodiments can relate to making discrete UI selections, including, for example, selecting one of the UI elements 125, 130, 135, 140 (e.g., as icons, menu items, folders, radio buttons, and / or the like) via gaze and / or head movement. The gaze and / or head movement can correspond to a gesture, and the gesture can trigger the selection of one of the UI elements 125, 130, 135, 140. In some embodiments, the correspondence between gaze and head movement can be used to indicate and / or identify a gesture.

[0015] For example, referring to FIG. 1A, arrow H indicates the direction of the head movement of the user 105. Furthermore, arrow E indicates the direction of the gaze movement of the user 105. In FIG. 1A, the direction of the head movement and the direction of the gaze movement are the same direction. That direction is illustrated as a rightward direction. However, the direction may be any direction, such as a leftward direction, rightward direction, upward direction, downward direction, and any direction therebetween. When the direction of the head movement and the direction of the gaze movement are the same direction, and the movement distance satisfies a criterion (e.g., is less than a threshold movement distance), the gaze within the boundary of the UI 120 can indicate that a gesture is being performed. Furthermore, the position of the gaze (e.g., a gaze point described later) can indicate the selection of one of the UI elements 125, 130, 135, 140. In other words, small gaze and / or head movements can indicate that the user 105 is performing a gesture. Furthermore, movements in the same direction can confirm that the user 105 is performing a gesture.

[0016] When the direction of the head movement and the direction of the gaze movement are the same direction, and the movement distance does not satisfy the criterion (e.g., exceeds the threshold movement distance), the movement can indicate that the user is observing something in the real world (e.g., not looking within the UI 120). In other words, large gaze and / or head movements can indicate that the user 105 is not performing a gesture, and is instead observing the real world and / or looking outside the boundary of the UI 120.

[0017] For example, referring to Figure 1B, arrow H indicates the direction of user 105's head movement. Furthermore, arrow E indicates the direction of user 105's gaze movement. In Figure 1B, the direction of head movement and the direction of gaze movement are different (e.g., opposite). The directions are illustrated as head movement to the right and gaze movement to the left. However, these directions could be any (opposite) direction, such as left, right, up, down, and any direction in between. When the direction of head movement and the direction of gaze movement are different, user 105 may not have intended to perform the gesture. For example, some events can cause user 105 to move their head, such as being pushed (e.g., a walking user bumping into another person). At the same time, the user's eyes will try to maintain their gaze on approximately the same position on UI 120 (e.g., the point of fixation described later). Therefore, the direction of head movement and the direction of gaze movement will be different. Therefore, head movements and eye movements can indicate that user 105 does not intend to perform the gesture.

[0018] For example, referring to Figure 1C, arrow H indicates the direction of user 105's head movement. Furthermore, arrow E indicates the direction of user 105's gaze movement. In Figure 1C, the direction of head movement and the direction of gaze movement may be the same or different (e.g., opposite). The directions are illustrated as head movement toward the right and gaze movement toward the left or right. However, the directions may be any (opposite) direction, such as left, right, up, down, and any direction in between. If the direction of head movement and the direction of gaze movement are the same or different, and the distance of the head movement does not meet a criterion (e.g., exceeds the threshold movement distance), the movement may indicate that the user is observing something in the real world (e.g., not looking within UI120). In other words, a large head movement may indicate that user 105 is not performing a gesture and is instead observing the real world and / or looking outside the boundaries of UI120.

[0019] For example, referring to Figure 1D, line H (e.g., not an arrow) indicates that user 105's head movement is minimal to zero. Furthermore, arrow E indicates the direction of user 105's gaze movement. In Figure 1D, the direction of gaze movement does not influence the determination of whether user 105 intends to perform a gesture. That direction is illustrated as left. However, that direction can be any direction, such as left, right, up, down, and any direction in between. If head movement is minimal to zero and the distance of the gaze movement meets a criterion (e.g., less than the threshold movement distance), then gaze within the boundaries of UI 120 can indicate that a gesture is being performed. Furthermore, the position of the gaze on UI 120 (e.g., the point of fixation described later) can indicate a selection of one of UI elements 125, 130, 135, or 140. In other words, gaze movement alone can indicate that user 105 is performing a gesture.

[0020] For example, referring to Figure 1E, arrow H indicates the direction of user 105's head movement. Furthermore, line E (e.g., not an arrow) indicates that user 105's gaze movement is minimal to zero. In Figure 1E, the direction of head movement is illustrated as the head movement being directed to the right. However, the direction of head movement can be any direction, such as left, right, up, down, and any direction in between. If the distance of the head movement does not meet a criterion (e.g., exceeds the threshold movement distance), the movement may indicate that the user is observing something in the real world (e.g., not looking within UI120). In other words, a large head movement may indicate that user 105 is not performing a gesture and is instead observing the real world and / or looking outside the boundaries of UI120.

[0021] In any of the embodiments shown in Figures 1A to 1E, once a gesture by user 105 is performed (or determined to be performed), the user's intent can be determined using the final gaze position on UI 120 (e.g., the gaze point described later). For example, the gesture can be used to determine which UI elements 125, 130, 135, and 140 user intends to interact with. Interaction with one of the UI elements 125, 130, 135, and 140 may be configured to trigger a UI action based on the selected UI element. For example, UI element 140 is illustrated as a folder. Therefore, interaction with one of the UI elements 140 may result in opening that folder within the display area 115. Figure 2 can further illustrate how user 105 interacts with UI 120 on the wearable device 110.

[0022] Figure 2 shows a block diagram of the data flow of a wearable device 110 according to an exemplary embodiment. The data flow is associated with a system for specifying UI elements of a UI that operates on the wearable device 110 using a combined gesture interface of the user's head movements and gaze. The wearable device 110 may be a head-mounted device. For example, the wearable device 110 may be an AR / VR headset, smart glasses, etc. As shown in Figure 2, the system includes a gaze module 205, a head movement module 210, a gesture module 215, and a UI 120. The UI 120 includes a UI element module 220 and a UI action module 225.

[0023] The gaze module 205 may be configured to track the gaze of the user of the wearable device 110. The gaze module 205 may be configured to determine the gaze characteristics of the user of the wearable device. The gaze characteristics may relate to the gaze (e.g., eye movements) of the user of the wearable device 110 (e.g., the person wearing it). For example, the wearable device 110 may include hardware and software that sense eye movements and generate data including the orientation, velocity, direction, position, and / or similar of the eye movements before, during, and after the eye movements. The gaze characteristics may be selected, determined, generated, calculated, and / or similar actions performed using the generated data. In some embodiments, the gaze module 205 may be configured to use the generated data to select, determine, generate, calculate, and / or similar actions performed on the gaze characteristics(s).

[0024] For example, referring to Figures 1A-1E, the gaze module 205 may be configured to generate gaze movement including gaze direction (e.g., arrow E) and gaze movement distance, and to determine whether the gaze movement distance meets a criterion indicating that user 105 intends to perform a gesture (e.g., is less than a threshold movement distance). If the gaze module 205 determines that user 105 intends to perform a gesture, the gaze module 205 may be configured to communicate data associated with the gaze movement and / or data associated with gaze characteristics(s) to the gesture module 215.

[0025] The head motion module 210 may be configured to determine the head movements of the user of the wearable device 110. For example, the wearable device may include an IMU configured to generate IMU data. The IMU data may include head IMU data 733 (e.g., rotational speed, acceleration, position, orientation, velocity, and / or similar). The IMU data may be acquired by the head motion module 210 (e.g., received from the IMU) and used to track the head movements of the user wearing the wearable device. The head motion module 210 may be configured to generate, store, and / or communicate head motion data. The head motion module 210 may be configured to generate, store, and / or communicate head movements, including, for example, head motion vectors. For example, head motion data may include tracking (e.g., storage of timestamped data) including head position, orientation, and direction. Head motion vectors can be calculated based on the head motion data.

[0026] For example, referring to Figures 1A-1E, the head motion module 210 may be configured to generate head motion including the direction of head motion (e.g., arrow H) and head movement distance, and to determine whether the head movement distance meets the criteria indicating the user 105's intention to perform the gesture. If the head motion module 210 determines that the user 105 intends to perform the gesture, the head motion module 210 may be configured to communicate data and / or IMU data related to the head motion to the gesture module 215.

[0027] The gesture module 215 may be configured to determine a gesture based on gaze characteristics, head movements, and the correlation between gaze characteristics and head movements. In an exemplary embodiment, a gesture can be determined by combining the current gaze position, head movements, and gaze movements using eye tracking. For example, a gesture can be determined by determining the current gaze position, head movements, and gaze movements, and then combining these three as a gesture. For example, referring to Figure 3E, a gesture can be determined based on gaze position 305, head movements along line 320, and gaze movements along line 325. Furthermore, each time the user's gaze changes, the head movement should be along the same vector as the gaze vector, thereby confirming the gaze. Moving along the same vector can demonstrate a correlation between gaze characteristics and head movements. The gesture module 215 may be configured to determine a point of fixation based on the gesture. The point of fixation may be a location on the UI that the user is looking at or viewing. The point of fixation will be described in more detail below.

[0028] UI120 can be configured as an interface between the user of the wearable device 110 and the hardware of the wearable device 110. UI120 can be a visual UI (for example, the interface is visible on the display of the wearable device 110). UI120 can be a head-locked visual UI. When the user interface is head-locked, the wearable device 110 moves (for example, when worn on the head), and various objects from the physical environment outside the wearable device 110 are visible to the user, and the user interface can remain in a stationary position on the display of the wearable device 110 while the wearable device 110 is moving. In contrast, when the world-locked, UI120 can remain stationary relative to objects from the physical environment outside the wearable device 110 while the wearable device 110 is moving. In such a situation, the user interface elements can move within the display.

[0029] UI120 includes a UI element module 220 and a UI behavior module 225. UI elements (e.g., UI elements 125, 130, 135, 140) may be icons, menus, etc. The UI element module 220 may be configured to manage the UI elements (or other UI elements) that are visually displayed on UI120. For example, the UI element module 220 may be configured to manage the position of UI elements, the behaviors triggered by UI elements, the dimensions of UI elements, the characteristics of UI elements (e.g., color), and the relationships between UI elements (e.g., relationships with other UI elements).

[0030] The UI element(s) module 220 may be configured to select UI elements of a headlock UI operating on a wearable device based on gestures. For example, the UI element(s) module 220 may be configured to select UI elements based on gaze associated with a gesture. In some embodiments, the UI element(s) module 220 may include a map (e.g., a table) containing a mapping between gaze points and UI elements. Thus, the UI element(s) module 220 may be configured to select UI elements from the map using gaze points determined by the gesture module 215. UI elements can be searched within the map by selecting (e.g., filtering) gaze points within the map.

[0031] The UI Action Module 225 can be configured to trigger UI actions. UI actions can be triggered by UI elements. For example, interaction with an icon can trigger (e.g., to display) a contact(s). UI actions can be triggered without interaction with an icon. In response to the selection of a UI element, the UI Action Module 225 can be configured to trigger UI actions (e.g., based on the UI element). In some embodiments, the UI Action Module 225 can include a map (e.g., a table) containing a mapping between UI elements and UI actions. Thus, the UI Action Module 225 can be configured to select a UI action from the map using a UI element determined by the UI Element Module 220. UI elements can be searched within the map by selecting (e.g., filtering) UI elements within the map.

[0032] For example, UI120 may be shown or hidden without interaction with the icon. For example, UI120 may be shown or hidden based on criteria. For example, UI120 may be hidden after a threshold period of no user interaction. UI behavior may be associated with components of the wearable device 110 and may be triggered by UI elements. For example, interaction with the icon may trigger (e.g.) the camera of the wearable device 110 to capture an image.

[0033] In a visual UI, a user typically looks at the UI element they are interacting with. For example, referring to Figure 3A, the user may currently be looking at the UI element represented by gaze position 305. From the perspective of absolute gaze tracking, gaze position 305 is located at a point along the x-axis 315 and y-axis 310 within the UI (e.g., UI120). When a user moves their head, the movement is usually nonlinear. For example, referring to Figure 3B, the user may have head movement along line 320. When a user moves their gaze (e.g., eye movement), the movement is usually linear. For example, referring to Figure 3C, the user may have gaze movement along line 325.

[0034] In exemplary embodiments, determining a gesture using eye tracking can be achieved by combining the current gaze position (Figure 3A) and gaze movement (Figure 3C). For example, a gesture may be determined by determining the current gaze position 305, determining the gaze movement (e.g., along line 325), and combining these two as a gesture. In some embodiments, a gesture may be associated with gaze-based user input configured to interact with elements of a UI operating on a wearable device. For example, referring to Figure 3D, a gesture may be determined based on the gaze position 305 and the gaze movement along line 325. The technique shown in Figure 3D is an effective tool for determining gestures. However, this technique may rely on margins and / or padding within the UI to identify UI elements that may be selected based on the gesture, which may reduce the available space for content. Furthermore, the gaze position may depend on the accuracy of the calibration.

[0035] In exemplary embodiments, determining a gesture using eye tracking can be achieved by combining the current gaze position (Figure 3A), head movement (Figure 3B), and eye movement (Figure 3C). For example, a gesture may be determined by determining the current gaze position 305, determining head movement (e.g., along line 320), determining eye movement (e.g., along line 325), and combining these three as a gesture. Referring to Figure 3E, for example, a gesture can be determined based on the gaze position 305, head movement along line 320, and eye movement along line 325.

[0036] In exemplary embodiments, the vector may include direction and velocity data. Direction and velocity may be determined using eye-tracking data (e.g., eye-tracking data 770, which will be described in more detail below). For example, the eye-tracking data may include a glint vector. The glint vector may be based on a glint spot based on the relative position of the pupil center and the corneal reflection. The eye-tracking vector may be a vector from the glint spot to the pupil center indicating the direction in which the eyeball is facing relative to the camera. Thus, eye movement along line 325 may be based on the eye-tracking vector and the eye-tracking position 305. In some embodiments, the eye-tracking data may include eye-tracking characteristics, which may be and / or include the eye-tracking vector, glint vector, glint spot, a timestamp associated with the eye-tracking vector, and / or velocity associated with the eye-tracking vector.

[0037] Figures 4-1A, B, and C show partial block diagrams of a user interface (UI) design for a wearable device, including sectors, according to an exemplary embodiment. As shown in Figure 4-1A, the UI 300 may include multiple sectors (shown as blocks or grids). In Figure 4-1B, sector 305 may include multiple pixels 307 associated with the display of the wearable device. More specifically, the multiple pixels 307 may be associated with the UI 300 displayed on the display of the wearable device (for example, the UI 300 does not necessarily encompass the entire display of the wearable device).

[0038] The point of focus may correspond to one or more of a plurality of pixels 307. The point of focus may correspond to one or more of a plurality of pixels 307 that display at least a portion of UI elements that the user can interact with to trigger an action within the UI, on a wearable device, and / or on a device communicably coupled to a wearable device. The location of the point of focus (e.g., the point of focus position) may be (or be based on) Cartesian coordinates associated with the x-axis (e.g., x-axis 315) and y-axis (e.g., y-axis 310) associated with the UI 300. Thus, the point of focus and / or the point of focus position may be determined based on the Cartesian coordinates of the pixels. The Cartesian coordinates of the pixels may be determined by and / or using gaze tracking data (e.g., gaze data 770, which is described in more detail below).

[0039] In some embodiments, the accuracy of determining the pixel position may be insufficient for determining the gaze point position. Therefore, the gaze point may correspond to one of several sectors (e.g., sector 305). For example, each of the several sectors (e.g., sector 305) may have a range on the x-axis (e.g., x-axis 315) and a range on the y-axis (e.g., y-axis 310) associated with UI300. Thus, the gaze point and / or gaze point position may be determined based on the Cartesian coordinates of the current, previous, and / or later gaze position. The Cartesian coordinates of the gaze position may be determined by and / or using gaze tracking data (e.g., gaze data 770, which will be described in more detail below). If the gaze position is within the range on the x-axis and the range on the y-axis, the gaze point position may correspond to the associated sector.

[0040] In some embodiments, the accuracy of determining the gaze position may not be sufficient for use in determining the gaze point location of a UI element or part thereof. Alternatively or additionally, the pixel density (PD) of a sector may fall below a threshold number of pixels. As a result, the UI element (e.g., UI element 442) may not be rendered with sufficient quality. Therefore, the gaze point may correspond to a minimum number of sectors. As shown in Figure 4-1C, the gaze point may include an N×N grid of sectors (e.g., sector 305). For example, each N×N (shown as 2×2 in Figure 4-1C) grid of a sector may have ranges on the x-axis (e.g., x-axis 315) and y-axis (e.g., y-axis 310) associated with UI 300. Therefore, the gaze point and / or gaze point location may be determined based on the Cartesian coordinates of the gaze position. As an example, the Cartesian coordinates of the gaze position may be determined by and / or using gaze tracking data (e.g., gaze data 770, which will be described in more detail below). When the line of sight is within a range on the x-axis and a range on the y-axis, the point of gaze may correspond to an N×N grid of the associated sector. In some embodiments, each of the multiple sectors and / or the N×N grid of the sectors may have a corresponding unique identifier or ID.

[0041] Figure 4-2D shows a partial block diagram of the UI300 design for a wearable device. The UI may include multiple sectors 302, 304, 306, 308, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, and 440. One or more of sectors 302-340 may contain UI element 442. In an exemplary embodiment, to improve the precision of the selection of UI element 442, the UI300 may be composed of an N×N grid of sectors. In other words, the UI300 may have a design such that one UI element may be contained within an N×N grid of sectors. For example, UI element 442 is shown as being within sectors 304, 306, 414, and 416 (e.g., a 2×2 sector or quadrant).

[0042] UI300 may include discrete pointing vectors 444 that point from an N×N grid in one sector to an N×N grid in another sector (three vectors are shown as an example). As described above, in the exemplary embodiment, the N×N grid in each sector may include UI elements 442. In the exemplary embodiment, the gesture module 215 may store (for example, in memory) a dataset representing predefined gaze movements, i.e., information, such as predefined gaze movements (sometimes called gaze saccades), and fixation points represented by, for example, discrete pointing vectors 444 in the UI. For example, the pointing vectors 444 may be identified and / or generated when UI300 is designed and stored in the gesture module 215.

[0043] In some embodiments, UI300 may include multiple UI elements. Furthermore, a data map may be associated with the UI. The data map may include references to multiple UI elements, references to gaze points associated with multiple UI elements (e.g., one-to-one references), references to vectors from a first gaze point to a second gaze point (e.g., pointing vectors), and / or gaze saccads (may include multiple) associated with multiple UI elements. In some embodiments, gaze saccads associated with UI elements can be used to identify whether the gaze is moving within a UI element or to another UI element, for example. For example, a stored gaze saccad may be a value (may include multiple) representing a threshold (or range) (e.g., a numerical value (may include multiple)), where if the gaze falls below the threshold, movement may be within a UI element, and if it exceeds the threshold, movement may be directed to another UI element. For example, a UI element may be a menu containing multiple menu items. An associated gaze saccad may indicate whether the gaze is moving between menu items (e.g., a slow gaze saccad) or to a different UI element (e.g., a fast gaze saccad).

[0044] Referring to Figures 1, 2E, and 3, the gaze module 205 can receive and / or generate gaze tracking data (e.g., gaze data 770, which will be described in more detail below). The gaze module 205 can determine whether the user's gaze has changed, and if so, can communicate the gaze tracking data to the gesture module 215. Based on the gaze tracking data, the gesture module 215 can determine the gaze characteristics(s) of the user of the wearable device 110. The gaze characteristics(s) may include gaze saccades(s) and gaze points(s), and can generate vectors. Based on the generated vectors, the gaze module 205 may be configured to select the pointing vector 444 of the UI 300 (e.g., using a stored map, as described below in detail).

[0045] Simultaneously, the head motion module 210 receives head motion data (e.g., IMU data 733, which will be described in more detail below) from, for example, the integrated inertial measurement unit (IMU) of the wearable device 110. The head motion module 210 can determine whether the user's head is moving (e.g., whether it exceeds a threshold amount of motion, acceleration, velocity, and / or similar), and if the head is moving, it can communicate the head motion data to the gesture module 215. The gesture module 215 can determine the head motion. For example, the gesture module 215 can generate a head motion vector. The head motion vector may include the direction, velocity, acceleration, etc., associated with the head motion. In exemplary embodiments, since the UI is a headlock UI, head motion may be minimal. Therefore, in gesture determination, gaze may be weighted more heavily than head motion. For example, in some embodiments, gaze characteristics may be determined based on the head motion vector. For example, if the head motion vector is in the same direction as the gaze vector (for example, if the gaze characteristic includes the gaze vector), then the gaze characteristic can be confirmed to be correct.

[0046] In an exemplary embodiment, if the user's gaze shifts from one point of focus to another, and the head movement follows the same vector, UI elements can be identified and highlighted based on the gesture. For example, the gesture module 215 can communicate with the UI element module 220 to identify UI elements based on selected pointing vectors 444 and / or head movement vectors. For example, the UI element module 220 may include a lookup table containing a list of UI elements and their corresponding pointing vectors 444. In an exemplary embodiment, each UI element may have one or more corresponding pointing vectors 444. In other words, since the initial point of focus on UI 120 can be anywhere on UI 120, a UI element may have multiple corresponding pointing vectors.

[0047] The selected pointing vector 444 may indicate that the gesture is associated with sectors of an N×N grid, including sectors 304, 306, 414, and 416. Thus, the UI element 442 (as identified within or by the UI element module 220) is selected by the gesture module 215 (see E in Figure 4-2). Furthermore, the UI element 442 can be highlighted. As an example, to highlight the UI element 442, it can be resized (e.g., enlarged), as shown F in Figure 4-2. Other techniques for highlighting the UI element 442 are also within the scope of this disclosure. For example, the color may be changed, the UI element may float or be made 4D, or a box may surround the UI element.

[0048] As mentioned above, since the UI is a headlock UI, head movement is minimal. Therefore, gestures can be indicated with minimal head movement. However, head movement exceeding a threshold amount of movement, acceleration, velocity, and / or similar may indicate that the user is not looking at the UI and is instead looking at the surrounding environment (sometimes called the world view). Therefore, head movement exceeding a threshold amount of movement, acceleration, velocity, and / or similar may not indicate a gesture and may not trigger other UI actions.

[0049] Furthermore, gesture errors may prevent UI actions from being executed. Conversely, the failure to execute a UI action may result in a gesture error. In some embodiments, gesture errors may include false detection of gaze, false detection of head movement, and / or detection of missed interactions or gestures. In exemplary embodiments, the user may re-execute a gesture for correction. Corrective movements are distinctly different from normal movements and can be classified as pointing vectors. For example, pointing vectors can be multidimensional. For example, a pointing vector can move away from a UI element and then back towards it. For example, a pointing vector moving away from a UI element may be a predefined direction indicating a re-execution of a previous gesture (e.g., up, down, left, right, etc.).

[0050] In exemplary embodiments, a gesture can be determined using either head movement or eye movement. For example, if the gaze module 205 fails to generate eye movement data, or if the head movement module 210 fails to generate head movement data, the other (eye movement data or head movement data) can be used to determine the gesture. For example, a pointing vector can be generated using either eye movement data or head movement data, and the gesture can be determined based on the pointing vector. Furthermore, in exemplary embodiments, the gesture module 215 may be configured to switch between using eye movement data, head movement data, or both eye movement data and head movement data to determine the gesture. In other words, since both head tracking and gaze tracking are motion-oriented, the gesture module 215 may be configured to smoothly switch between eye movement data and head movement data without affecting the user interacting with the UI 120 if either eye movement data or head movement data is unavailable.

[0051] Figure 5 shows a method for generating gesture data according to an exemplary embodiment. As shown in Figure 5, gaze is generated in step S505. For example, this method may be performed in a wearable device. The wearable device may include an eye-tracking element, a visual positioning system (VPS), and / or an eye-tracking function (e.g., gaze module 205). For example, referring to Figure 7 below, the wearable device may include VPS data 752 which includes at least one of second position data 754, second orientation data 756, and second velocity data 758. Thus, tracking gaze may include generating at least one of second position data 754, second orientation data 756, and second velocity data 758.

[0052] In step S510, gaze characteristics are determined based on the gaze. For example, gaze characteristics may include position data, orientation data, velocity, etc. In some embodiments, gaze characteristics may be derived from position data, orientation data, velocity, etc. For example, gaze characteristics may include eye saccades, point of fixation (e.g., position on the UI), etc. In some embodiments, gaze characteristics may be determined, generated, calculated, and / or similarly performed based on neural network data 742, gaze data 752, and / or VPS data 752. As an example, orientation may be calculated based on first position data 744 and second position data 754.

[0053] In step S515, head motion data is acquired. For example, a wearable device may include a head tracking element, an inertial measurement unit (IMU), and / or a head tracking function (e.g., a head motion module 210). For example, referring to Figure 7 below, the wearable device may include IMU data 733. For example, the IMU data 733 may include rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739, which are derived from gyroscope and accelerometer measurements associated with the IMU. In some embodiments, head motion data may be derived from rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739, and / or similar. For example, head motion data may include head motion vectors derived from rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739, and / or similar. Head motion vectors may include the direction, velocity, and acceleration associated with head movement.

[0054] In step S520, head motion is determined based on head motion data. For example, head motion may include a head motion vector that includes direction, velocity, acceleration, etc. In an exemplary embodiment, the direction of head motion and the velocity of head motion based on the velocity data 739 can be calculated using position data 737, orientation data 738, and velocity data 739 (for example, using two or more consecutive position data 737 and consecutive orientation data 738).

[0055] In step S525, at least one pointing vector is determined based on gaze characteristics and head movements. For example, a wearable device may have a UI that operates on the wearable device's display. The UI may have an associated map containing a plurality of pointing vectors associated with UI elements. The map may also include gaze points associated with UI elements. Thus, a pointing vector can include gaze points on the UI and a vector (e.g., the direction from one gaze point to another). In some embodiments, determining a pointing vector may include selecting a pointing vector. For example, determining a pointing vector may include searching (e.g., filtering) within the map based on variables that constitute the pointing vector based on UI elements, gaze characteristics and / or head movements.

[0056] Augmented reality and virtual reality (AR / VR) systems can utilize wearable devices such as head-mounted displays (HMDs) and smart glasses. In some wearable devices, gestures used to select UI elements may be based on gaze patterns and the user's head movements. Gestures may be based on the correlation between gaze patterns and the user's head movements. Figures 6A, 6B, and 6C illustrate smart glasses as an example of a wearable device configured to use the correlation between gaze patterns and the user's head movements as gestures used to select UI elements on the smart glasses' UI.

[0057] Figure 6A shows a user wearing exemplary smart glasses 500, which include display, eye / gaze tracking, and computing / processing functions. Figure 6B is a front view of the exemplary smart glasses 500 shown in Figure 6A, and Figure 6C is a rear view.

[0058] An exemplary smart glasses 500 may be configured to track the user's gaze, determine the characteristics of the tracked user's gaze, determine the head movements of the user wearing the smart glasses 500, and generate gestures based on the gaze characteristics and the user's head movements. The exemplary smart glasses 500 may be configured to display a user interface (UI) on the smart glasses 500's display. The UI may be a headlock UI. The generated gestures may be used to select UI elements of the headlock UI. The selected UI elements may be configured to trigger UI actions associated with the headlock UI.

[0059] An exemplary smart glasses 500 includes a frame 610. The frame 610 includes a front frame portion 620 and a pair of temple arm portions 630 rotatably coupled to the front frame portion 620 by their respective hinge portions 640. The front frame portion 620 includes a rim portion 623 surrounding each optical portion in the form of a lens 627 and a bridge portion 629 connecting the rim portions 623. The temple arm portions 630 are pivotably or rotatably coupled to the front frame portion 620, for example, at the periphery of each rim portion 623. In some embodiments, the lens 627 is a corrective / prescription lens. In some embodiments, the lens 627 is an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters.

[0060] In some embodiments, the smart glasses 500 include a display device 505 that outputs visual content, thereby allowing the user to view the visual content. In the embodiments shown in Figures 6B and 6C, the display device 505 is provided on one of the two arm sections 630, for illustrative purposes only. The display device 505 may be provided on each of the two arm sections 630 to provide binocular output of content. In some embodiments, the display device 505 may be a see-through near-eye display. In some embodiments, the display device 505 may be configured to project light from a display source onto a portion of teleprompter glass that functions as a beam splitter positioned at an angle (e.g., 30 to 45 degrees). The beam splitter may tolerate reflectivity and transmittance values ​​that allow light from the display source to be partially reflected while the remaining light is transmitted. Such an optical design may allow the user to view physical items in the real world through, for example, lens 627, while simultaneously viewing content output by the display device 505 (e.g., digital images, user interface elements, virtual content, etc.). In some embodiments, waveguide optics may be used to render content on the display device 505. Due to errors in head pose data, digital images may be rendered with a deviation from physical items in the real world. Therefore, in exemplary embodiments, harmonic exponential filters may be used in the head pose signal processing pipeline to correct for errors in head pose data.

[0061] As described above, the display device 505 can output visual content, thereby making that visual content visible to the user. In some embodiments, the visual content may include (and / or be) a user interface (UI). The UI may include at least one UI element. The UI element may include screens, windows, buttons, menus, toggles, icons, and / or other visual elements used by the user to interact with the smart glasses 500. Typically, a pointing device (e.g., a mouse) can be used to manipulate the cursor and make selections in the UI operating on the computing device. However, pointing devices may not be practical for use in wearable devices. Therefore, gestures can be used to interact with the UI operating on the smart glasses 500. Gestures may be generated based on gaze characteristics (may be multiple) associated with eye-tracking of the user's eyes in the smart glasses 500 and the user's head movements in the smart glasses 500.

[0062] In some embodiments, the UI can be head-locked in such a way that it remains in the same position on the display 505 when the user moves their head and / or when the smart glasses 500 move. For example, superimposed AR content may move with the user's head movements while remaining within the user's field of view (FOV). As an example, in a head-locked visual UI, the position and orientation of the superimposed AR content in the AR display depend not on any features of the real-world scene, but solely on the position and orientation of the viewer's head (as if the AR object or content were rigidly attached to the viewer's head by a rigid rod). The superimposed AR content may move across the entire real-world scene in the AR display in instantaneous response to the viewer's head movements of any type (e.g., all amplitudes, frequencies, and directions, 6 degrees of freedom (6DoF) movements). For example, a circular or sinusoidal frequency head movement may cause the superimposed AR content to move across the entire real-world scene at the same circular or sinusoidal frequency as the head movement. Furthermore, for example, head movements such as tilting or rolling will cause the superimposed AR content to tilt from the vertical line of gravity by the same angle as the tilt or rotation of the head movement that moves it away from the vertical line of gravity.

[0063] In some embodiments, the sensing system 611 may include various sensing devices, and the control system 612 may include various control system devices, for example, one or more processors 614 operably coupled to components of the control system 612. In some embodiments, the control system 612 may include a communication module that provides communication and information exchange between the smart glasses 500 and other external devices. In some embodiments, the smart glasses 500 include an eye-tracking device 615 for detecting and tracking the direction and movement of the gaze. The eye-tracking device 615 may include sensors 617, 619 (e.g., cameras). Data captured by the eye-tracking device 615 may be processed to detect and track the direction and movement of the gaze as user input. In the embodiments shown in Figures 6B and 6C, the eye-tracking device 615 is provided on one of the two arm sections 630, for illustrative purposes only. In the exemplary configuration shown in Figures 6B and 6C, the eye-tracking device 615 is mounted on the same arm 630 as the display device 505, thereby allowing the user's gaze to be tracked not only for objects in the physical environment but also for content output for display by the display device 505. In some embodiments, the gaze, or eye-tracking device 615, may be mounted on each of the two arm 630 to provide eye-tracking for each of the user's two eyes. In some embodiments, the display device 505 may be mounted on each of the two arm 630 to provide binocular display of visual content.

[0064] In some embodiments, the smart glasses 500 include an audio output device 506 (e.g., one or more speakers), a lighting device 508, a sensing system 611, a control system 612, at least one processor 614, and one or more outward-facing image sensors or outward-facing cameras 616. In exemplary embodiments, the outward-facing image sensors or outward-facing cameras 616 can be used to generate image data used to generate head pose data. The outward-facing camera 616 may include an inertial measurement unit (IMU). Alternatively (or additionally), the IMU may be a separate module included in the smart glasses 500. For example, the IMU may be included in one of two arm sections 630. The IMU may include a pair of gyroscopes configured to measure rotational speed and an accelerometer configured to measure the acceleration of the outward-facing camera 616 as the camera moves with the head and / or body, or with the user. In exemplary embodiments, the outward-facing camera 616 and / or IMU may be used in the head pose data generation operation (or processing pipeline).

[0065] Figure 7 shows an example of a processing circuit 720. In an exemplary embodiment, the processing circuit 720 may include a circuit (e.g., a signal processing pipeline) configured to generate eye-tracking data. Eye-tracking data may be generated based on image data and / or IMU data.

[0066] In some embodiments, one or more components of the processing circuit 720 may be, or include, a processor (e.g., a processing unit 724) configured to process instructions stored in the memory 726. Examples of such instructions, as depicted in Figure 7, include the IMU manager 730, the neural network manager 740, the visual positioning system manager 750, and the gesture manager 760. Furthermore, as shown in Figure 7, the memory 726 is configured to store various data, which will be described in relation to each manager that uses such data.

[0067] The IMU manager 730 is configured to acquire IMU data 733. In some embodiments, the IMU manager 730 acquires the IMU data 733 wirelessly (for example, from the IMU associated with the outward-facing camera 616). As shown in Figure 7, the IMU manager 730 includes an error compensation manager 731 and an integration manager 732.

[0068] The error compensation manager 731 is configured, for example, to receive the IMU output (IMU data 733) from the IMU manager 730 and compensate for the error in the IMU output using IMU-specific parameter values. The error compensation manager 731 is further configured to generate the IMU data 733 after performing the error compensation.

[0069] The integration manager 732 is configured to perform an integral operation (e.g., the sum of time-dependent values) on the IMU data 733. The rotational velocity data 734 is integrated over time to generate orientation. Furthermore, the acceleration data 735 is integrated twice over time to generate position. Thus, the integration manager 732 generates 6DoF attitude (position and orientation) from the IMU output, i.e., the rotational velocity data 734 and the acceleration data 735.

[0070] IMU data 733 represents gyro and accelerometer measurements in the world coordinate system (as opposed to the local coordinate system, i.e., the IMU's coordinate system), compensated for errors (may include errors), as well as acceleration data 735. Furthermore, IMU data 733 includes 6DoF attitude and motion data, position data 737, orientation data 738, and velocity data 739, derived from the gyro and accelerometer measurements. Finally, in some embodiments, IMU data 733 also includes IMU temperature data 736, which may indicate further errors in the rotational velocity data 734 and acceleration data 735.

[0071] The neural network manager 740 is configured to receive rotational velocity data 734 and acceleration data 735 as input and generate neural network data 742 including first position data 744, first orientation data 746, and first velocity data 748. In some embodiments, the input rotational velocity data 734 and acceleration data 735 are generated by an error compensation manager 731 that acts on the raw IMU output values, i.e., compensates for errors using IMU-specific parameter values. As shown in Figure 7, the neural network manager 740 includes a neural network training manager 741.

[0072] The neural network training manager 741 is configured to take in training data 749 and generate neural network data 742, which includes data on layers, cost functions, and values. In some embodiments, the training data 749 includes, for example, motion data obtained from measurements of a user wearing smart glasses 500 and moving their head and other parts of their body, and ground truth 6DoF posture data obtained from those measurements. In some embodiments, the training data 749 includes measured rotational velocity and acceleration from motion, paired with measured 6DoF posture and velocity.

[0073] Furthermore, in some embodiments, the neural network manager 740 uses historical data from the IMU to generate first position data 744, first orientation data 746, and first velocity data 748. For example, this historical data is used to extend the training data 749 with a mapping of past rotational speed, acceleration, and temperature to the resulting 6DoF attitude and motion results, thereby further improving the neural network. In some embodiments, the neural network represented by the neural network manager 740 is a convolutional neural network, and its layers are convolutional layers.

[0074] The Visual Positioning System (VPS) manager 750 is configured to receive an image as input and generate VPS data 752, which includes second position data 754 and second orientation data 756. In some embodiments, the VPS data also includes second velocity data 758, i.e., 6DoF attitude based on the image. In some embodiments, the image is acquired using an outward-facing camera on the frame of the smart glasses 500 (e.g., outward-facing camera 616).

[0075] Furthermore, the processing circuit 720 may include a network interface 722, one or more processing units 724, and non-temporary memory 726. The network interface 722 may include, for example, an Ethernet® adapter, a Token Ring adapter, a Bluetooth® adapter, a WiFi adapter, an NFC adapter, etc., for converting electronic and / or optical signals received from a network into an electronic format for use by the processing circuit 720. The series of processing units 724 may include one or more processing chips and / or assemblies. The memory 726 may include both volatile memory (e.g., RAM) and non-volatile memory (e.g., one or more ROMs, disk drives, solid-state drives, etc.). The series of processing units 724 and the memory 726 together form a processing circuit, which is configured and arranged to perform various methods and functions described herein. Thus, the series of processing units 724 and the memory 726 together form a processing circuit, which is a signal processing pipeline configured to generate head pose data corrected using a harmonic index filter (e.g., included in the gesture manager 760).

[0076] Example 1. Figure 8 is a block diagram of how a wearable device operates according to an exemplary embodiment. As shown in Figure 8, in step S805, gaze data is acquired (e.g., received). In step S810, the gaze characteristics of the wearable device user are determined based on the gaze data. For example, the wearable device may include an eye tracking module configured to track eye movements using, for example, a rearward, inward, or eye-direction camera. The eye tracking module may be configured to generate, communicate, and / or store gaze characteristics(s). Gaze characteristics(s) may include saccades, gaze vectors, glint vectors, glint spots, timestamps associated with gaze vectors, and / or velocities associated with gaze vectors.

[0077] In step S815, head movement data is acquired (e.g., received). In step S820, the user's head movements are determined based on the head movement data. For example, a wearable device may include an IMU used to track the head movements of a user wearing the wearable device. The IMU may be configured to generate, store, and / or communicate IMU data as head movement data, which includes (or is used to generate) head movement vectors. For example, the IMU data may include tracking (e.g., storage of timestamped data) including head position, direction, orientation, etc. Head movement vectors may be calculated based on the IMU data.

[0078] In step S825, a gesture is determined based on gaze characteristics, head movements, and the correlation between gaze characteristics and head movements. In an exemplary embodiment, a gesture can be determined by combining the current gaze position, head movements, and gaze movements using eye tracking. For example, a gesture can be determined by determining the current gaze position, head movements, and gaze movements, and then combining these three as a gesture. For example, referring to Figure 3E, a gesture can be determined based on gaze position 305, head movements along line 320, and gaze movements along line 325. Furthermore, each time the user's gaze changes, the head movement should be along the same vector as the gaze vector, thereby confirming the gaze. Moving along the same vector can demonstrate a correlation between gaze characteristics and head movements. Otherwise, the gaze may be identified as an incorrect measurement (e.g., a false detection of gaze).

[0079] In step S830, a UI element of the headlock user interface (UI) is selected based on the gesture to be displayed, operated, processed, executed, and / or otherwise performed on the wearable device. In some embodiments, the gesture can be used to identify a point of gaze on the UI. In some embodiments, a map may include links between the point of gaze and the UI element located at that point of gaze. Thus, determining the gesture may include identifying the point of gaze and using a map to identify the UI element located at that point of gaze within the UI. In step S835, a UI action is triggered based on the UI element. A UI element may have an associated UI action. For example, the UI action may be configured to display an image. Thus, by interacting with the UI element located at a second point of gaze, the display of an image on the wearable device's display can be triggered. Further details relating to each of these steps are described above.

[0080] Example 2. The method of Example 1, wherein the gaze characteristics may include at least one of a gaze saccade and a point of fixation. In some embodiments, a gaze saccade may indicate that the gaze is moving within a UI element or, for example, to another UI element. For example, the UI element may be a menu containing multiple menu items. An associated gaze saccade may indicate that the gaze is moving between menu items (e.g., a slow gaze saccade) or to a different UI element (e.g., a fast gaze saccade). In some embodiments, if the gaze saccade is below a threshold, there is no need to generate a gaze-based gesture, and therefore the method of Example 1 may stop after step S810. Alternatively or additionally, the UI element may include sub-elements (e.g., menu items). Thus, a second point of fixation may be associated with a sub-element within the UI element.

[0081] Example 3. The method of Example 1, wherein head movement may include a head motion vector. For example, a wearable device may include an IMU used to track the head movement of a user wearing the wearable device. The IMU may be configured to generate, store, and / or communicate IMU data as head motion data that includes (or is used to generate) a head motion vector. For example, the IMU data may include tracking (e.g., storage of timestamped data) that includes head position, direction, orientation, etc. The head motion vector may be calculated based on the IMU data. Furthermore, each time the user's gaze changes from one point of fixation to another, the head movement should be along the same vector as the gaze vector, thereby confirming the gaze. Otherwise, the gaze may be identified as an erroneous measurement (e.g., false detection of gaze).

[0082] Example 4. The method of Example 1, wherein a UI element may be associated with an N×N block of UI, and the UI behavior changes the properties of the UI element.

[0083] Example 5. The method of Example 1, wherein the correlation between gaze characteristics and head movements can indicate that the user is looking outside the boundaries of the UI, and the UI is hidden by the UI action.

[0084] Example 6. The method of Example 1 may further include identifying a false gaze detection, a false head movement detection, or a missed UI interaction, and preventing the execution of a UI action in response to the identification of a false gaze detection, a false head movement detection, or a missed UI interaction.

[0085] Example 7. The method of Example 1 may further include determining a second gaze characteristic, determining a second head movement, selecting a second UI element using the correlation between the second gaze characteristic and the head movement characteristic as an index correlation, and triggering a user interface (UI) action based on the selected second UI element.

[0086] Example 8. The method may include one or any combination of the methods from Examples 1 to 7.

[0087] Example 9. A non-temporary computer-readable storage medium that stores instructions configured to cause a computing system to perform any of the methods of Examples 1 to 8 when executed by at least one processor.

[0088] Example 10. An apparatus comprising means for carrying out any of the methods of Examples 1 to 8.

[0089] Example 11. An apparatus comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured together with the at least one processor to cause the apparatus to perform any of the methods of at least one of Examples 1 to 8.

[0090] Generally, the user's gaze characteristics determined for selecting UI elements may relate to the characteristics of the user's gaze, which are measurable via sensor devices of a wearable device (e.g., a head-mounted device) and include the movement of at least one of the user's eyes and / or point of fixation. For example, gaze characteristics may include gaze saccades (i.e., the determination that at least one gaze saccade occurred) and at least one of the point of fixation.

[0091] Triggering a UI action based on a selected UI element can include triggering a UI action related to the selected UI element, and therefore based on the selection of the UI element.

[0092] In some embodiments, head movement may include a head motion vector, i.e., head movement for gesture determination is determined based on the head motion vector.

[0093] Furthermore, UI elements may be associated with an N×N block of UI, and UI actions may change the properties of the UI elements. An N×N block may be N columns × N rows. A block may be one or more pixels. For example, a UI action may cause a change in the properties of a selected UI element, including the display properties of the selected UI element. In one embodiment, a UI action may highlight a selected UI element in an N×N block of UI by changing its color, making the selected UI element float from the UI interface, making the selected UI element 4D, or enclosing the selected UI element in a box within the UI interface. An N×N block may be N columns of blocks × N rows of blocks. A block may be one or more pixels.

[0094] In some embodiments, the correlation between gaze characteristics and head movements can indicate that the user is looking outside the boundaries of the UI, and the UI behavior can cause the UI to be hidden.

[0095] Furthermore, false detections of gaze, false detections of head movements, or missed UI interactions may be detected during the determination of gaze characteristics, head movements, or gestures. Therefore, when determining (attempting to determine) a gesture based on gaze characteristics, head movements, and the correlation between gaze characteristics and head movements, a gesture error can be determined, where the gesture error prevents the execution of a UI action. Thus, the determination of a gesture error may include detecting an incorrect gaze, detecting an incorrect head movement, and / or detecting a missed interaction or gesture. In response to identifying any of the false detections of gaze, false detections of head movements, or missed UI interactions, the execution of a UI action can be prevented.

[0096] An exemplary embodiment may include a non-temporary computer-readable storage medium storing instructions configured, when executed by at least one processor, to cause a computing system to perform any of the methods described above. An exemplary embodiment may include a device that includes means for performing any of the methods described above. An exemplary embodiment may include a device having at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured, using at least one processor, to cause the device to perform at least one of the methods described above.

[0097] Various embodiments of the systems and technologies described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, and which is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to them.

[0098] These computer programs (also known as programs, software, software applications, or code) contain machine instructions to a programmable processor and can be executed in high-level procedural programming languages ​​and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0099] To provide user interaction, the systems and technologies described herein can be implemented on a computer having a display device (LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., mouse or trackball) on which the user can input information into the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and the input from the user can be received in any form, including acoustic, spoken language, or haptic input.

[0100] The systems and technologies described herein can be implemented in a computing system including a backend component (e.g., as a data server), a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a client computer having a graphical user interface or web browser that allows a user to interact with the modes of implementation of the systems and technologies described herein), or in any combination of such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0101] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other.

[0102] Several embodiments have been described. Nevertheless, it should be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0103] Additionally, the logic flow shown in the diagram does not require a specific sequence, i.e., a sequential order, to achieve the desired result. Furthermore, other steps may be added to the described flow, or steps may be removed from the described flow, and other components may be added to the described system, or other components may be removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

[0104] While specific features of the embodiments described herein have been illustrated, numerous modifications, substitutions, alterations, and equivalents will be conceivable to those skilled in the art. It should be understood that the appended claims are intended to encompass all such modifications and alterations that fall within the scope of the embodiments. They are presented only as examples and not as limitations, and various modifications in form and detail are permitted. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the different embodiments described.

[0105] The exemplary embodiments may include various modifications and alternative forms, which are shown as examples in the drawings and described in detail herein. However, it should be understood that the exemplary embodiments are not intended to limit the embodiments to any particular form disclosed, but rather should include all modifications, equivalents, and alternative forms that fall within the scope of the claims. Similar numbers refer to similar components throughout the description of the figures.

[0106] Some of the exemplary embodiments described above are explained as processes or methods shown as flowcharts. While flowcharts describe operations as sequential, many operations may occur in parallel, together, or simultaneously. The order of operations can also be changed. A process may terminate when its operations are completed, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0107] The methods described above, some of which are illustrated by flowcharts, can be performed by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When performed by software, firmware, middleware, or microcode, the program code or code segment for performing the required task can be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) can perform the required task.

[0108] The specific structural and functional details disclosed herein are provided solely for illustrative purposes to illustrate exemplary embodiments. However, exemplary embodiments can be embodied in many alternative forms and should not be construed as being limited only to the embodiments described herein.

[0109] The terms, first, second, etc., may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the terms and / or include one or any combination of the associated described items.

[0110] When an element is referred to as being "connected" or "joined" to another element, it will be understood that it can be directly connected or joined to the other element, or that there may be an intermediary element. In contrast, when an element is referred to as being "directly connected" or "directly joined" to another element, there is no intermediary element. Other words used to describe the relationship between elements should be interpreted similarly (e.g., between vs. directly between, adjacent vs. directly adjacent, etc.).

[0111] The terminology used herein is for the purpose of describing specific embodiments and is not intended to limit the exemplary embodiments. Where used herein, the singular forms a, an, and the are also intended to include the plural forms unless the context otherwise explicitly indicates. It should be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” where used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0112] Furthermore, please note that in some alternative embodiments, the functions / actions shown may occur in a different order than that shown in the diagrams. For example, two diagrams shown consecutively may actually be executed simultaneously, or they may be executed in reverse order depending on the functions / actions involved.

[0113] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the exemplary embodiments belong. For example, terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as their meanings in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0114] The exemplary embodiments described above and the corresponding embodiments for carrying out the inventions are presented with respect to software, or with respect to algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are intended to effectively convey to those skilled in the art the essence of their research. Where the term is used herein, and where it is used generally, an algorithm is considered to be a self-consistent sequence of steps leading to a desired result. These steps require the physical manipulation of physical quantities. Usually, but not always, these quantities take the form of optical, electrical, or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. For reasons of general use, it has sometimes proven convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0115] In the exemplary embodiments described above, references to actions and symbolic representations of behavior (e.g., in the form of a flowchart) that may be executed as a program module or functional process include routines, programs, objects, components, data structures, etc., which perform specific tasks or execute specific abstract data types and may be described and / or executed using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate array (FPGA) computers, etc.

[0116] However, it should be recognized that all these terms and similar terms should correspond to appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specified or as is evident from the description, terms such as decisions to process or compute or calculate or display refer to actions and processes of a computer system or similar electronic computing device, which manipulates data represented as physical quantities, electronic quantities, in the computer system's registers and memory and converts them into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage devices, transmission devices, or display devices.

[0117] It should also be noted that the software execution modes of the exemplary embodiments are typically encoded on some form of non-temporary program storage medium or executed via some type of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read-only memory, i.e., CD-ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cables, optical fibers, or any other suitable transmission medium known in the art. The exemplary embodiments are not limited to these aspects of any given embodiment.

[0118] Finally, while the attached claims describe specific combinations of features described herein, it should be noted that the scope of this disclosure is not limited to the specific combinations claimed below, but rather extends to encompass any combination of features of the embodiments disclosed herein, regardless of whether the specific combination is explicitly enumerated in the attached claims at this point.

Claims

1. A non-temporary computer-readable storage medium containing stored instructions, wherein, when the instructions are executed by at least one processor, the computing system... Determining the eye-tracking characteristics of wearable device users, Determining the user's head movements, The gesture is determined based on the aforementioned gaze characteristics, head movements, and the correlation between the gaze characteristics and head movements. Based on the aforementioned gesture, select a UI element of the headlock user interface (UI) displayed on the wearable device, A non-temporary computer-readable storage medium configured to trigger UI actions based on the selected UI elements.

2. The non-temporary computer-readable storage medium according to claim 1, wherein the gaze characteristics include at least one of a gaze saccade and a point of fixation.

3. The non-temporary computer-readable storage medium according to claim 1 or 2, wherein the head movement includes a head motion vector.

4. The aforementioned UI element is associated with an N x N block of the UI, The non-temporary computer-readable storage medium according to any one of claims 1 to 3, wherein the UI operation changes the characteristics of the UI elements.

5. The correlation between the gaze characteristics and the head movements indicates that the user is looking outside the boundaries of the UI. The aforementioned UI operation is to hide the UI, a non-temporary computer-readable storage medium according to any one of claims 1 to 4.

6. The instruction further states to the computing system Identifying one of the following: false eye tracking, false head movement tracking, or missed UI interactions, A non-temporary computer-readable storage medium according to any one of claims 1 to 5, which, in response to the identification of one of the following: false detection of gaze, false detection of head movement, or missed UI interaction, causes the UI action to not be performed.

7. The instruction further states to the computing system To determine the second line of sight characteristic, To determine the second head movement, The correlation between the second gaze characteristic and the head movement characteristic is used as an index correlation to select a second UI element, A non-temporary computer-readable storage medium according to claim 6, which causes a user interface (UI) operation to be triggered based on the selected second UI element.

8. It is a method, Determining the eye-tracking characteristics of wearable device users, Determining the user's head movements, Based on at least one of the gaze characteristics and head movements, it is determined whether the user of the wearable device intends to perform a gesture. In response to the user of the wearable device deciding to perform a gesture, The gesture is determined based on the aforementioned gaze characteristics, head movements, and the correlation between the gaze characteristics and head movements. Based on the aforementioned gesture, select a UI element of the headlock user interface (UI) displayed on the wearable device, A method comprising triggering a UI action based on the selected UI element.

9. The method according to claim 8, wherein the line of sight characteristics include at least one of a line of sight saccade and a point of fixation.

10. The method according to claim 8 or 9, wherein the head movement includes a head motion vector.

11. The aforementioned UI element is associated with an N x N block of the UI, The method according to any one of claims 8 to 10, wherein the UI operation changes the characteristics of the UI element.

12. The correlation between the gaze characteristics and the head movements indicates that the user is looking outside the boundaries of the UI. The method according to any one of claims 8 to 11, wherein the UI operation is to hide the UI.

13. Identifying one of the following: false eye tracking, false head movement tracking, or missed UI interactions, The method according to any one of claims 8 to 12, further comprising preventing the execution of a UI action in response to the identification of one of the following: false detection of gaze, false detection of head movement, or missed UI interaction.

14. To determine the second line of sight characteristic, To determine the second head movement, The correlation between the second gaze characteristic and the head movement characteristic is used as an index correlation to select a second UI element, The method according to claim 13, further comprising triggering a user interface (UI) action based on the selected second UI element.

15. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code, together with the at least one processor, Determining the eye-tracking characteristics of wearable device users, Determining the user's head movements, The gesture is determined based on the aforementioned gaze characteristics, head movements, and the correlation between the gaze characteristics and head movements. Based on the aforementioned gesture, select a UI element of the headlock user interface (UI) displayed on the wearable device, A device configured to trigger a UI action and perform an action based on the selected UI element.

16. The apparatus according to claim 15, wherein the line of sight characteristics include at least one of a line of sight saccade and a point of fixation.

17. The apparatus according to claim 15 or 16, wherein the head movement includes a head motion vector.

18. The aforementioned UI element is associated with an N x N block of the UI, The apparatus according to any one of claims 15 to 17, wherein the UI operation changes the characteristics of the UI elements.

19. The aforementioned computer program code further applies to the computing system. Identifying one of the following: false eye tracking, false head movement tracking, or missed UI interactions, The apparatus according to any one of claims 15 to 18, which, in response to the identification of one of the following: false detection of gaze, false detection of head movement, or missed UI interaction, prevents the execution of the UI action.

20. The aforementioned computer program code further applies to the computing system. To determine the second line of sight characteristic, To determine the second head movement, The correlation between the second gaze characteristic and the head movement characteristic is used as an index correlation to select a second UI element, The apparatus according to claim 19, which causes a user interface (UI) operation to be triggered based on the selected second UI element.

Images