Wearable device hybrid gaze tracking

Abstract

A method comprising: determining an eye gaze characteristic of a user of a wearable device; determining a head movement of the user; determining a gesture based on the eye gaze characteristic, the head movement, and a correlation between the eye gaze characteristic and the head movement; selecting a user interface (UI) element of a head-locked UI operating on the wearable device based on the gesture; and triggering a UI operation based on the selected UI element.

Description

[0001] Cross-reference to related applications

[0002] This application is a continuation-to-priority application to U.S. Application No. 18 / 477,188, filed on September 28, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The implementation involves gaze tracking using wearable devices. Background Technology

[0004] Wearable devices, including augmented reality (AR) glasses, allow users to quickly access the functionality of a general-purpose computer throughout the day without interruption by visually presenting content in the user's field of vision. Summary of the Invention

[0005] The implementation involves addressing elements (e.g., icons, menus, etc.) to interact with the user interface (UI) of wearable devices (e.g., AR headsets, AR glasses, etc.) by utilizing a combination of head movements and eye gazes. More specifically, this topic relates to generating gestures based on the correlation between eye movements and head movements associated with UI elements. These gestures can be used to make discrete UI selections, including, for example, selecting an icon, selecting a menu item, or gazing away from the UI (e.g., triggering the closing of the UI), etc.

[0006] In general, an apparatus, a system, a non-transitory computer-readable medium (on which computer-executable program code that can be executed on a computer system) and / or a method can utilize a method to perform a process comprising: determining an eye gaze characteristic of a user of a wearable device; determining a head movement of the user; determining a gesture based on the eye gaze characteristic, the head movement, and the correlation between the eye gaze characteristic and the head movement; selecting a UI element of a head-locked user interface (UI) operating on the wearable device based on the gesture; and triggering a UI operation based on the selected UI element. Attached Figure Description

[0007] The example implementation will be more fully understood from the detailed description and accompanying figures given below, in which the same elements are indicated by the same reference numerals. The figures are given by way of illustration only and therefore do not limit the example implementation.

[0008] Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 1E A graphical representation of a scene for performing hybrid gaze tracking is shown, based on an example implementation.

[0009] Figure 2 A block diagram illustrating the data flow of a wearable device according to an example implementation is shown.

[0010] Figure 3A A diagram showing eye-tracking gaze position based on an example implementation is provided.

[0011] Figure 3B A diagram showing the head tracking movement position according to the example implementation is shown.

[0012] Figure 3C A graph showing eye-tracking gaze movement is shown based on an example implementation.

[0013] Figure 3D The diagram shows eye tracking gaze position plus gaze movement according to the example implementation.

[0014] Figure 3E The diagram shows eye tracking gaze position plus gaze movement plus head movement according to the example implementation.

[0015] Figure 4A Figures 4B and 4C show block diagrams of a portion of a wearable device user interface (UI) design that includes partitions, according to an example implementation.

[0016] Figure 4D , Figure 4E and Figure 4F A block diagram showing a portion of a wearable device user interface (UI) design, including partitions and UI elements, based on an example implementation.

[0017] Figure 5 This illustrates a method for generating gesture data based on an example implementation.

[0018] Figure 6A An example head-mounted wearable device worn by a user is shown.

[0019] Figure 6B yes Figure 6A The example head-mounted wearable device shown is a front view, and Figure 6C This is a rear view of the example head-mounted wearable device.

[0020] Figure 7 This is a diagram illustrating an example device for performing hybrid gaze tracking, based on an example implementation.

[0021] Figure 8 A block diagram of a method for operating a wearable device according to an example implementation is shown.

[0022] It should be noted that these figures are intended to illustrate the general characteristics of the methods and / or structures utilized in certain example implementations and to supplement the written description provided below. However, these figures are not necessarily drawn to scale and may not accurately reflect the precise structural or performance characteristics of any given implementation, and should not be construed as limiting or restricting the range of values ​​or properties covered by the example implementations. For example, the positioning of modules and / or structural elements may be reduced or enlarged for clarity. The use of similar or identical reference numerals in the various figures is intended to indicate the presence of similar or identical elements or features. Detailed Implementation

[0023] Users of computing devices may require one or more peripheral devices to interact with the software (e.g., user interface) and / or hardware (e.g., camera) of the computing device. For example, a typical computing device may use a mouse, keyboard, touch interface, etc. However, a technical problem associated with wearable devices is that the wearable device may not have the necessary interfaces to connect a mouse, keyboard, touch interface, etc. Furthermore, carrying peripheral devices can lead to a poor user experience. A technical solution could be to perform the functions of peripheral devices on the wearable device based on the user's eye and / or head movements (e.g., similar functions as peripheral devices for computing devices). The technical effect of performing peripheral functions on the wearable device based on the user's eye and / or head movements can be to enable the user to interact with the software (e.g., user interface) and / or hardware (e.g., camera) of the wearable device without using physical peripheral devices, resulting in an improved user experience.

[0024] Implementations involve interacting with the user interface (UI) of a wearable device (e.g., AR headset, AR glasses, etc.) by addressing elements (e.g., icons, menus, etc.) using eye gaze and / or head movement. More specifically, this topic relates to using eye gaze and / or head movement for discrete UI selection, including, for example, selecting an icon, selecting a menu item, gazing away from the UI (e.g., triggering the closing of the UI), etc. The selected UI element can trigger an action associated with the UI. In example implementations, the UI design may include partitions (e.g., blocks, N×N blocks, quadrants, etc.) containing UI elements. These partitions can improve the accuracy of UI element selection. In some implementations, this topic relates to generating gestures based on different eye movements within the UI. For example, gestures may be generated based on eye movement from a first partition to a second partition.

[0025] As discussed above, wearable devices, including augmented reality (AR) glasses, can allow users to quickly access the functionality of a general-purpose computer by visually presenting content within the user's field of view. However, a problem with current wearable technology is the lack of a mechanism for hands-free UI interaction without accessory devices and / or user interference (e.g., user touching the wearable device). Therefore, there is a need for new technologies that enable intuitive, distraction-free, and reliable interaction while allowing users to interact with user interface (UI) elements. Interaction with UI elements should be unaffected by movement (e.g., head movement independent of the UI) and motion (e.g., user walking). An example implementation addresses this issue by providing a system that uses eye gaze and / or head movement gestures to address UI elements.

[0026] Example implementations include an interaction system for head-locked UIs. The system can handle inputs including head movements (e.g., three (3) degrees of freedom (DoF) head movements, acceleration, gyroscope, etc.) and eye gaze characteristics (e.g., saccades, eye gaze vectors, timestamps associated with eye gaze vectors, and / or velocities associated with eye gaze vectors, etc.) and UIs with partitioned (e.g., N×N blocks or quadrants) visual UIs and head-locked visual UIs. The system can be controlled based on natural interactions between the user and the wearable device, including moving their eyes toward visual UI elements. In some implementations, eye gazes can be correlated with corresponding head movements (e.g., to confirm eye gesture detection) to generate gestures for making discrete UI selections. In addition to user experience benefits, the system can reduce the accuracy and calibration requirements of eye-tracking technology compared to traditional eye-tracking systems, resulting in cost savings.

[0027] Figures 1A to 1E A graphical representation of a scene for performing hybrid gaze tracking, based on an example implementation, is shown. Figures 1A to 1E As shown, user 105 can wear wearable device 110 (e.g., AR headset, AR glasses, etc.). Wearable device 110 may have a display area 115 on the lenses (e.g., one or more lenses) of wearable device 110. Display area 115 can be configured to display user interface (UI) 120. UI 120 may include multiple UI elements 125, 130, 135, 140.

[0028] Some implementations may involve making discrete UI selections, including, for example, selecting one of UI elements 125, 130, 135, 140 (e.g., as an icon, menu item, folder, radio button, etc.), eye gaze, and / or head movement. Eye gaze and / or head movement may correspond to a gesture, and this gesture may result in the selection of one of UI elements 125, 130, 135, 140. In some implementations, the correspondence between eye gaze and head movement can be used to indicate and / or recognize gestures.

[0029] For example, refer to Figure 1A Arrow H indicates the direction of head movement for user 105. Further, arrow E indicates the direction of eye movement for user 105. Figure 1A In this context, the direction of head movement and the direction of eye gaze movement are in the same direction. This direction is shown as right. However, the direction can be any direction—left, right, up, down, or any direction in between. If the direction of head movement and the direction of eye gaze movement are in the same direction, and the distance moved meets a criterion (e.g., below a threshold distance), then eye gaze within the boundaries of UI 120 can indicate that a gesture is being performed. Furthermore, the location of the eye gaze (e.g., the gaze point described below) can indicate the selection of one of the UI elements 125, 130, 135, and 140. In other words, a small eye gaze and / or head movement can indicate that user 105 is performing a gesture. Furthermore, movement in the same direction can confirm that user 105 is performing a gesture.

[0030] If the direction of head movement and eye gaze movement are in the same direction, and the distance moved does not meet a criterion (e.g., exceeding a threshold distance), then the movement may indicate that the user is observing something in the real world (e.g., not within UI 120). In other words, a large eye gaze and / or head movement may indicate that the user is not performing a gesture and is instead observing the real world and / or looking outside the boundaries of UI 120.

[0031] For example, refer to Figure 1B Arrow H indicates the direction of head movement for user 105. Further, arrow E indicates the direction of eye movement for user 105. Figure 1BIn this context, the direction of head movement and the direction of eye gaze movement are in different (e.g., opposite) directions. This direction is shown as a rightward head movement and a leftward eye gaze movement. However, the direction can be any (opposite) direction, left, right, up, down, and any direction in between. If the direction of head movement and the direction of eye gaze movement are in different directions, then user 105 may not be intending to perform a gesture. For example, some event may cause user 105 to be pushed (e.g., the user walking may bump into another person), thus causing user 105's head to move. At the same time, the user's eyes attempt to maintain eye gaze at substantially the same location on UI 120 (e.g., the gaze point described below). Therefore, the direction of head movement and the direction of eye gaze movement are in different directions. Thus, head movement and eye gaze movement can indicate that user 105 is not intending to perform a gesture.

[0032] For example, refer to Figure 1C Arrow H indicates the direction of head movement for user 105. Further, arrow E indicates the direction of eye movement for user 105. Figure 1C In this context, the direction of head movement and the direction of eye gaze movement can be in the same direction or in different (e.g., opposite) directions. This direction is illustrated as a rightward head movement and a leftward or rightward eye gaze movement. However, the direction can be any (opposite) direction, left, right, up, down, and any direction in between. If the direction of head movement and the direction of eye gaze movement are in the same or different directions, and the head movement distance does not meet a criterion (e.g., exceeding a threshold distance), then the movement may indicate that the user is observing something in the real world (e.g., not within UI 120). In other words, a large head movement may indicate that the user is not performing a gesture and is instead observing the real world and / or looking outside the boundaries of UI 120.

[0033] For example, refer to Figure 1D The line (e.g., not an arrow) H indicates that user 105 has minimal to no head movement. Further, the arrow E indicates the direction of movement in which user 105's eyes are focused. Figure 1DIn this context, the direction of eye gaze movement does not affect the determination of whether user 105 is intending to perform a gesture. This direction is shown as left. However, the direction can be any direction—left, right, up, down, and any direction in between. An eye gaze within the boundaries of UI 120 can indicate that a gesture is being performed if there is a minimum of no head movement and the distance of the eye gaze movement meets a criterion (e.g., below a threshold distance of movement). Furthermore, the location of the eye gaze on UI 120 (e.g., the gaze point described below) can indicate the selection of one of UI elements 125, 130, 135, and 140. In other words, eye gaze movement alone can indicate that user 105 is performing a gesture.

[0034] For example, refer to Figure 1E Arrow H indicates the direction of head movement for user 105. Further, line (e.g., not an arrow) E indicates the minimum to no eye-focused movement for user 105. Figure 1E In this context, the direction of head movement is shown as a rightward head movement. However, the direction of head movement can be any direction—left, right, up, down, and any direction in between. If the head movement distance does not meet a criterion (e.g., exceeding a threshold distance), the movement may indicate that the user is observing something in the real world (e.g., not within UI 120). In other words, a large head movement may indicate that the user is not performing a gesture and is instead observing the real world and / or looking outside the boundaries of UI 120.

[0035] exist Figures 1A to 1E In any of the examples provided, if a gesture by user 105 has been performed (or is determined to be performed), the final eye gaze location on UI 120 (e.g., the gaze point described below) can be used to determine user 105's intent. For example, the gesture can be used to determine which of UI elements 125, 130, 135, and 140 user 105 intends to interact with. Interaction with one of the UI elements 125, 130, 135, and 140 can be configured to trigger a UI action based on the selected UI element. For example, UI element 140 is shown as a folder. Therefore, interaction with one of the UI elements 140 could cause the folder to open within display area 115. Figure 2 This can be used to further describe the interaction between user 105 and UI 120 on wearable device 110.

[0036] Figure 2A block diagram illustrating the data flow of a wearable device 110 according to an example implementation is shown. The data flow is associated with a system for addressing UI elements that operate on the wearable device 110 using a combined user head movement and eye gaze gesture interface. The wearable device 110 may be a head-mounted device. For example, the wearable device 110 may be an AR / VR head-mounted device, smart glasses, etc. Figure 2 As shown, the system includes an eye gaze module 205, a head movement module 210, a gesture module 215, and a UI 120. The UI 120 includes a UI element module 220 and a UI operation module 225.

[0037] The eye gaze module 205 can be configured to track the eye gaze of a user of the wearable device 110. The eye gaze module 205 can be configured to determine the eye gaze characteristics of the user of the wearable device. These eye gaze characteristics can be related to the gaze (e.g., eye movement) of the user of the wearable device 110 (e.g., a person wearing the wearable device). For example, the wearable device 110 may include hardware and software that senses eye movement and generates data including eye orientation, velocity, direction, position, etc., before, during, and after eye movement. The generated data can be used to select, determine, generate, calculate, etc., eye gaze characteristics. In some implementations, the eye gaze module 205 can be configured to use the generated data to select, determine, generate, calculate, etc., eye gaze characteristics.

[0038] For example, refer to Figures 1A to 1E The eye gaze module 205 can be configured to generate an eye gaze movement that includes the eye gaze direction (e.g., arrow E), the distance the eye gaze has moved, and determine whether the distance the eye gaze has moved meets the criteria for instructing the user 105 to perform a gesture (e.g., below a threshold distance). If the eye gaze module 205 determines that the user 105 intends to perform a gesture, the eye gaze module 205 can be configured to communicate data associated with the eye gaze movement and / or data associated with eye gaze characteristics to the gesture module 215.

[0039] The head movement module 210 can be configured to determine the head movement of a user of the wearable device 110. For example, the wearable device may include an IMU configured to generate IMU data. The IMU data may include IMU data 733 of the head (e.g., rotational speed, acceleration, position, orientation, velocity, etc.). The IMU data may be acquired by the head movement module 210 (e.g., received from the IMU) and used to track the head movement of the user wearing the wearable device. The head movement module 210 can be configured to generate, store, and / or communicate head movement data. The head movement module 210 can be configured to generate, store, and / or communicate head movement including, for example, a head movement vector. For example, head movement data may include tracking (e.g., storing timestamped data) including head position, orientation, etc. A head motion vector can be calculated based on the head movement data.

[0040] For example, refer to Figures 1A to 1E The head movement module 210 can be configured to generate head movement including the direction of head movement (e.g., arrow H), the distance the head has moved, and determine whether the distance the head has moved meets the criteria for instructing the user 105 to perform a gesture. If the head movement module 210 determines that the user 105 intends to perform a gesture, the head movement module 210 can be configured to communicate data and / or IMU data associated with the head movement to the gesture module 215.

[0041] The gesture module 215 can be configured to determine gestures based on eye gaze characteristics, head movement, and the correlation between eye gaze characteristics and head movement. In an example implementation, gaze tracking can be used to determine a gesture by combining the current gaze position, head movement, and gaze movement. For example, a gesture can be determined by determining the current gaze position, determining the head movement, and determining the gaze movement, and combining these three into a gesture. For example, refer to... Figure 3E The gesture can be determined based on gaze position 305, head movement along line 320, and gaze movement along line 325. Furthermore, whenever the user's eye gaze changes, the head movement should follow the same vector as the eye gaze vector, thus confirming eye gaze. Movement along the same vector indicates the correlation between eye gaze characteristics and head movement. The gesture module 215 can be configured to determine the gaze point based on the gesture. The gaze point can be a position, location, etc., on the UI that the user is looking at or viewing. The gaze point will be described in more detail below.

[0042] UI 120 can be configured as an interface between the user of wearable device 110 and the hardware of wearable device 110. UI 120 can be a visual UI (e.g., an interface visible on the display of wearable device 110). UI 120 can be a head-locked visual UI. When head-locked, the user interface can remain in a fixed position on the display of wearable device 110 while wearable device 110 (e.g., when worn on a head) moves and different objects from the physical environment outside wearable device 110 are visible to the user and move relative to the user interface. In contrast, when world-locked, UI 120 can remain stationary relative to objects from the physical environment outside wearable device 110 while wearable device 110 moves. In such cases, user interface elements can move within the display.

[0043] UI 120 includes a UI element module 220 and a UI action module 225. UI elements (e.g., UI elements 125, 130, 135, 140) can be, for example, icons, menus, etc. The UI element module 220 can be configured to manage the UI elements visually displayed on UI 120. For example, the UI element module 220 can be configured to manage the position of UI elements, actions triggered by UI elements, the dimensions of UI elements, the characteristics of UI elements (e.g., color), the relationships between UI elements (e.g., relationships with other UI elements), etc.

[0044] UI element module 220 can be configured to select UI elements of a head-locked UI that operates on a wearable device based on gestures. For example, UI element module 220 can be configured to select UI elements based on a gaze associated with a gesture. In some implementations, UI element module 220 may include a map (e.g., a table) that maps gaze points to UI elements. Therefore, UI element module 220 can be configured to select UI elements from the map using a gaze point as determined by gesture module 215. UI elements can be located in the map by selecting (e.g., filtering) a gaze point within the map.

[0045] UI action module 225 can be configured to trigger UI actions. UI actions can be triggered by UI elements. For example, interaction with an icon can cause (e.g., trigger) a contact to be displayed. UI actions can also be triggered without interaction with an icon. In response to the selection of a UI element, UI action module 225 can be configured to trigger a UI action based on that UI element. In some implementations, UI action module 225 may include a map (e.g., a table) that maps UI elements to UI actions. Therefore, UI action module 225 can be configured to select UI actions from the map using UI elements as determined by UI element module 220. A UI element can be found in the map by selecting (e.g., filtering) a UI element within the map.

[0046] For example, UI 120 can be shown or hidden without interaction with the icon. As an example, UI 120 can be shown or hidden based on criteria. For instance, UI 120 can be hidden after a threshold period of no user interaction. UI actions can be associated with components of wearable device 110 and can be triggered by UI elements. For example, interaction with an icon can cause (e.g., trigger) an image to be captured by the camera of wearable device 110.

[0047] In visual UI, users typically view the UI elements they are interacting with. For example, see references. Figure 3A The user may currently be looking at a UI element represented by gaze position 305. From an absolute eye-tracking perspective, gaze position 305 is located at a point within the UI (e.g., UI 120) along the x-axis 315 and y-axis 310. When the user moves their head, the movement is typically non-linear. For example, refer to... Figure 3B The user can move their head along line 320. When the user moves their gaze (e.g., eye movement), the movement is typically linear. For example, refer to... Figure 3C Users can have a gaze movement of 325 along the line.

[0048] In the example implementation, gaze tracking is used to determine gestures by combining the current gaze position ( Figure 3A ) and gaze movement ( Figure 3C This can be achieved through various means. For example, a gesture can be determined by determining the current gaze position 305 and the gaze movement (e.g., along line 325) and combining the two into a gesture. In some implementations, the gesture may involve gaze-based user input configured to interact with elements of a UI operating on a wearable device. For example, see [reference to...] Figure 3D Gestures can be determined based on gaze position 305 and gaze movement along line 325. Figure 3DThe technique illustrated is an effective tool for determining gestures. However, this technique can rely on margins and / or inner margins within the UI to identify UI elements that can be selected based on gestures, which may reduce the available space for content. Furthermore, gaze position can depend on the accuracy of calibration.

[0049] In the example implementation, the current gaze position can be combined ( Figure 3A ), head movement ( Figure 3B ) and gaze movement ( Figure 3C This can be achieved by using gaze tracking to determine gestures. For example, a gesture can be determined by determining the current gaze position 305, determining head movement (e.g., along line 320), and determining gaze movement (e.g., along line 325), and combining these three factors into a gesture. For example, refer to... Figure 3E The gesture can be determined based on the gaze position 305, head movement along line 320, and gaze movement along line 325.

[0050] In the example implementation, the vector may include direction and velocity data. Eye-tracking data (e.g., gaze data 770, described in more detail below) can be used to determine the direction and velocity. For example, eye-tracking data may include a blink vector. The blink vector may be based on a blink point, which is based on the relative position of the pupil center and the corneal reflection. The eye gaze vector may be a vector from the blink point to the pupil center indicating which direction the eye is pointing relative to the camera. Therefore, gaze movement along line 325 may be based on the eye gaze vector and gaze position 305. In some implementations, eye-tracking data may include eye gaze characteristics, which may be and / or may include an eye gaze vector, a blink vector, a blink point, a timestamp associated with the eye gaze vector, and / or a velocity associated with the eye gaze vector.

[0051] Figure 4A Figures 4B and 4C show block diagrams of a portion of a wearable device user interface (UI) design, including partitions, according to an example implementation. Figure 4A As shown, UI 300 may include multiple partitions (shown as blocks or grids). In Figure 4B, partition 305 may include multiple pixels 307 associated with the display of the wearable device. More specifically, the multiple pixels 307 may be associated with UI 300 displayed on the display of the wearable device (e.g., UI 300 does not necessarily encompass the entire display of the wearable device).

[0052] A gaze point may correspond to one or more pixels among a plurality of pixels 307. A gaze point may correspond to one or more pixels among a plurality of pixels 307 that display at least a portion of a UI element with which a user can interact, the interaction causing operation within the UI on and / or on a device communicatively coupled to the wearable device. The location of the gaze point (e.g., gaze point position) may be (or based on) Cartesian coordinates associated with the x-axis (e.g., x-axis 315) and y-axis (e.g., y-axis 310) associated with UI 300. Therefore, the gaze point and / or gaze point position can be determined based on the Cartesian coordinates of the pixels. The Cartesian coordinates of the pixels can be determined by and / or using eye-tracking data (e.g., gaze data 770, described in more detail below).

[0053] In some implementations, the accuracy of determining the pixel's location may not be sufficient for determining the gaze point location. Therefore, the gaze point may correspond to one of several partitions (e.g., partition 305). For example, each of the multiple partitions (e.g., partition 305) may have a range on the x-axis (e.g., x-axis 315) and a range on the y-axis (e.g., y-axis 310) associated with UI 300. Thus, the gaze point and / or gaze point location can be determined based on the Cartesian coordinates of the current, previous, and / or subsequent gaze positions. The Cartesian coordinates of the gaze position can be determined by and / or using eye-tracking data (e.g., gaze data 770, described in more detail below). If the gaze position falls within the range on the x-axis and the range on the y-axis, the gaze point location can correspond to the associated partition.

[0054] In some implementations, the accuracy of determining the gaze position may be insufficient for determining the gaze point position for a UI element or a portion thereof. Alternatively or additionally, the pixel density (PD) of the partition may be below a threshold number of pixels. As a result, the UI element (e.g., UI element 442) may not be rendered with sufficient quality. Therefore, the gaze point may correspond to a minimum number of partitions. As shown in Figure 4C, the gaze point may comprise an N×N grid of partitions (e.g., partition 305). For example, each N×N grid of the partition (shown as 2×2 in Figure 4C) may have a range on the x-axis (e.g., x-axis 315) and a range on the y-axis (e.g., y-axis 310) associated with UI 300. Thus, the gaze point and / or gaze point position can be determined based on the Cartesian coordinates of the gaze position. As an example, the Cartesian coordinates of the gaze position can be determined by and / or using eye-tracking data (e.g., gaze data 770, described in more detail below). If the gaze position falls within a range on the x-axis and y-axis, the gaze point position can correspond to an associated N×N grid of the partition. In some implementations, each of the multiple partitions and / or the N×N grid of the partition can have a corresponding unique identifier or ID.

[0055] Figure 4D A block diagram of a portion of the design of a wearable device UI 300 is shown. The UI may include multiple partitions 302, 304, 306, 308, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, and 440. One or more partitions 302 to 340 may include UI element 442. In the example implementation, UI 300 may be configured in an N×N grid of partitions to improve the accuracy of selection of UI element 442. In other words, UI 300 may be designed such that one (1) UI element can be included in an N×N grid of partitions. For example, UI element 442 is shown as being within partitions 304, 306, 414, and 416 (e.g., 2×2 partitions or quadrants).

[0056] UI 300 may include discrete pointing vectors 444 (three (3) vectors are shown as an example) for pointing from one N×N grid of a partition to another N×N grid of the partition. As mentioned above, in the example implementation, each N×N grid of the partition may contain UI elements 442. In the example implementation, gesture module 215 may store (e.g., stored in memory) predefined eye gaze movements, i.e., information, such as a dataset representing predefined eye gaze movements (sometimes referred to as eye gaze saccades) and gaze points represented by, for example, discrete pointing vectors 444 within the UI. For example, when UI 300 is designed and stored in gesture module 215, pointing vectors 444 may be recognized and / or generated.

[0057] In some implementations, UI 300 may include multiple UI elements. Further, data mapping may be associated with the UI. Data mapping may include references to multiple UI elements, references to gaze points associated with multiple UI elements (e.g., one-to-one references), references to vectors from a first gaze point to a second gaze point (e.g., pointing vectors), and / or eye gaze sacs associated with multiple UI elements. In some implementations, eye gaze sacs associated with UI elements may be used to identify whether an eye gaze is moving within a UI element or moving to, for example, another UI element. For example, the stored eye gaze sacs may be values ​​(e.g., numerical values) representing a threshold (or range) below which movement may occur within a UI element, and above which movement may point to another UI element. For example, a UI element may be a menu comprising multiple menu items. The associated eye gaze sacs may indicate whether an eye gaze is moving between menu items (e.g., a slow eye gaze sac) or moving to a different UI element (e.g., a fast eye gaze sac).

[0058] Refer to Figure 1. Figure 2 E and Figure 3, the eye gaze module 205 can receive and / or generate eye tracking data (e.g., gaze data 770, described in more detail below). The eye gaze module 205 can determine whether the user's gaze has changed, and if so, the eye tracking data can be transmitted to the gesture module 215. The gesture module 215 can determine the user's eye gaze characteristics of the wearable device 110 based on the eye tracking data. The eye gaze characteristics can include eye gaze saccades and gaze points, and generate vectors. The eye gaze module 205 can be configured to select (e.g., using a stored mapping, described in more detail below) the pointing vector 444 of the UI 300 based on the generated vectors.

[0059] Simultaneously, head movement module 210 receives head movement data (e.g., IMU data 733, described in more detail below) from, for example, the integrated inertial measurement unit (IMU) of wearable device 110. Head movement module 210 can determine whether the user's head is moving (e.g., above a threshold amount for movement, acceleration, velocity, etc.), and if the head is moving, the head movement data can be communicated to gesture module 215. Gesture module 215 can determine head movement. For example, gesture module 215 can generate a head motion vector. The head motion vector may include direction, velocity, acceleration, etc., associated with head movement. In the example implementation, head movement can be minimal because the UI is a head-locked UI. Therefore, eye gaze can be weighted more heavily in gesture determination compared to head movement. For example, in some implementations, eye gaze characteristics can be confirmed based on the head motion vector. For example, if the head motion vector is in the same direction as the eye gaze vector (e.g., when eye gaze characteristics include the eye gaze vector), then the eye gaze characteristics can be confirmed as correct.

[0060] In the example implementation, UI elements can be identified and highlighted based on gestures whenever a user's eye gaze changes from one gaze point to another and the head movement follows the same vector. For example, gesture module 215 can communicate with UI element module 220 and identify UI elements based on a selected pointing vector 444 and / or head movement vector. For example, UI element module 220 can include a lookup table containing a list of UI elements and their corresponding pointing vectors 444. In the example implementation, each UI element can have one or more corresponding pointing vectors 444. In other words, a UI element can have multiple corresponding pointing vectors because the initial gaze point on UI 120 can be anywhere on UI 120.

[0061] The selected pointing vector 444 can indicate that the gesture is associated with an N×N grid including partitions 304, 306, 414, and 416. Therefore, UI element 442 (as in or recognized by UI element module 220) is selected via gesture module 215 (see...). Figure 4E Furthermore, UI element 442 can be highlighted. For example, as shown... Figure 4F As shown, the size of UI element 442 can be modified (e.g., made larger) to highlight UI element 442. Other techniques for highlighting UI element 442 are also within the scope of this disclosure. For example, the color can be changed, the UI element can be lifted off the display or made 4D, a frame can surround the UI element, etc.

[0062] As mentioned above, head movement can be minimal because the UI is head-locked. Therefore, minimal head movement can indicate a gesture. However, head movement exceeding a threshold of motion, acceleration, velocity, etc., can indicate that the user is not looking at the UI and is instead looking out at the surrounding environment (sometimes called world view). Therefore, head movement exceeding a threshold of motion, acceleration, velocity, etc., may not indicate a gesture and may not trigger other UI actions.

[0063] Additionally, gesture errors can cause UI actions not to be performed. Alternatively, UI actions not performed can also cause gesture errors. In some implementations, gesture errors may include detecting incorrect eye gaze, detecting incorrect head movement, and / or detecting that an interaction or gesture has been missed. In example implementations, the user can re-perform the gesture for correction. Corrective movement is distinctly different from regular movement and can also be categorized as a pointing vector. For example, a pointing vector can be multi-dimensional. For example, a pointing vector can move away from a UI element and return towards it. For example, a pointing vector moving away from a UI element can be in a predefined direction (e.g., up, down, left, right, etc.) indicating the re-execution of a previous gesture.

[0064] In the example implementation, either head movement or eye movement can be used to determine a gesture. For example, if the eye gaze module 205 fails to generate eye movement data or the head movement module 210 fails to generate head movement data, another (eye movement data or head movement data) can be used to determine the gesture. For example, either eye movement data or head movement data can be used to generate a pointing vector, and the gesture can be determined based on this pointing vector. Furthermore, in the example implementation, the gesture module 215 can be configured to switch between using eye movement data, head movement data, or both eye movement data and head movement data to determine the gesture. In other words, since both head tracking and eye tracking are motion-oriented, if either eye movement data or head movement data is unavailable, the gesture module 215 can be configured to smoothly switch between eye movement data and / or head movement data without affecting the user's interaction with the UI 120.

[0065] Figure 5 This illustrates a method for generating gesture data based on an example implementation. For example... Figure 5 As shown, in step S505, an eye gaze is generated. For example, the method can be implemented in a wearable device. The wearable device may include gaze tracking elements, a visual positioning system (VPS), and / or gaze tracking functionality (e.g., an eye gaze module 205). For example, refer to the following... Figure 7The wearable device may include VPS data 752, including at least one of second position data 754, second orientation data 756, and second velocity data 758. Therefore, tracking eye gaze may include generating at least one of the second position data 754, second orientation data 756, and second velocity data 758.

[0066] In step S510, eye gaze characteristics are determined based on eye gaze. For example, eye gaze characteristics may include position data, orientation data, velocity, etc. In some implementations, eye gaze characteristics can be derived from position data, orientation data, velocity, etc. For example, eye gaze characteristics may include eye saccades, gaze points (e.g., positions on the UI), etc. In some implementations, eye gaze characteristics can be determined, generated, and calculated based on neural network data 742, gaze data 752, and / or VPS data 752. As an example, orientation can be calculated based on first position data 744 and second position data 754.

[0067] In step S515, head movement data is obtained. For example, the wearable device may include a head tracking element, an inertial measurement unit (IMU), and / or head tracking functionality (e.g., head movement module 210). For example, refer to the following... Figure 7 The wearable device may include IMU data 733. For example, IMU data 733 may include rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739 derived from measurements by a gyroscope and accelerometer associated with the IMU. In some implementations, head movement data may be derived from rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739. For example, head movement data may include a head motion vector derived from rotational velocity data 734, acceleration data 735, position data 737, orientation data 738, and velocity data 739. The head motion vector may include direction, velocity, acceleration, etc., associated with head movement.

[0068] In step S520, head movement is determined based on head movement data. For example, head movement may include a head motion vector, which includes direction, velocity, acceleration, etc. In an example implementation, position data 737, orientation data 738, and velocity data 739 may be used to calculate the direction of head movement (e.g., using two or more of sequential position data 737 and sequential orientation data 738) and the head movement velocity based on velocity data 739.

[0069] In step S525, at least one pointing vector is determined based on eye gaze characteristics and head movement. For example, the wearable device may have a UI that operates on a display of the wearable device. The UI may have an associated map including multiple pointing vectors associated with UI elements. The map may also include gaze points associated with UI elements. Therefore, the pointing vector may include gaze points on the UI and vectors (e.g., directions from one gaze point to another). In some implementations, determining the pointing vector may include selecting the pointing vector. For example, determining the pointing vector may include searching (e.g., filtering) within the map based on variables constituting the pointing vector based on UI elements, eye gaze characteristics, and / or head movement.

[0070] Augmented reality and virtual reality (AR / VR) systems can utilize wearable devices such as head-mounted displays (HMDs) and smart glasses. In some wearable devices, gestures for selecting UI elements can be based on eye gaze characteristics and user head movements. The gestures can be based on the correlation between eye gaze characteristics and user head movements. Figure 6A , Figure 6B and Figure 6C Smart glasses are described as an example of a wearable device configured to use the correlation between eye gaze characteristics and user head movement as gestures for selecting UI elements on the smart glasses' UI.

[0071] Figure 6A The illustration shows a user wearing example smart glasses 500, which includes display capabilities, eye / gaze tracking capabilities, and computing / processing capabilities. Figure 6B yes Figure 6A The example smart glasses 500 shown in the image are front views, and Figure 6C This is a rear view of the example smart glasses.

[0072] The example smart glasses 500 can be configured to track a user's eye gaze, determine the characteristics of the tracked eye gaze, determine the head movement of the user wearing the smart glasses 500, and generate gestures based on the eye gaze characteristics and the user's head movement. The example smart glasses 500 can be configured to display a user interface (UI) on the display of the smart glasses 500. The UI can be a head-locked UI. The generated gestures can be used to select UI elements of the head-locked UI. The selected UI element can be configured to trigger UI actions associated with the head-locked UI.

[0073] Example smart glasses 500 includes a frame 610. Frame 610 includes a front frame portion 620 and a pair of temple arm portions 630 rotatably connected to the front frame portion 620 via corresponding hinge portions 640. The front frame portion 620 includes a rim portion 623 surrounding a corresponding optical portion in the form of a lens 627, wherein a bridge portion 629 connects to the rim portion 623. The temple portions 630 are connected (e.g., pivotally or rotatably connected) to the front frame portion 620 at peripheral portions of the corresponding rim portions 623. In some examples, the lens 627 is a corrective / prescription lens. In some examples, the lens 627 is an optical material comprising glass and / or plastic portions that do not necessarily contain corrective / prescription parameters.

[0074] In some examples, smart glasses 500 include a display device 505 capable of outputting visual content that makes the visual content visible to a user. For discussion and illustration purposes only, Figure 6B and Figure 6C In the example shown, display device 505 is disposed in one of the two temple portions 630. Display device 505 can be disposed in each of the two temple portions 630 to provide binocular output of content. In some examples, display device 505 may be a see-through near-eye display. In some examples, display device 505 may be configured to project light from a display source onto a portion of teleprompter glass positioned at an angle (e.g., 30-45 degrees) as a beam splitter. The beam splitter may allow reflection and transmission values ​​that allow light from the display source to be partially reflected while the remaining light is transmitted through. Such optical designs allow a user to see both, for example, physical objects in the world through lens 627, and adjacent content output by display device 505 (e.g., digital images, user interface elements, virtual content, etc.). In some implementations, waveguide optics may be used to depict content on display device 505. Due to errors in head pose data, digital images may be rendered with a certain offset from physical objects in the world. Therefore, the example implementation can use a harmonic exponential filter in the head pose signal processing pipeline to correct errors in the head pose data.

[0075] As mentioned above, the display device 505 can output visual content, making the visual content visible to the user. In some implementations, the visual content may include (and / or serve as) a user interface. The UI may include at least one UI element. UI elements may include screens, windows, buttons, menus, toggle keys, icons, and / or other visual elements used by the user to interact with the smart glasses 500. Typically, a pointing device (e.g., a mouse) may be used to guide the cursor and make selections on a UI operating on a computing device. However, a pointing device may not be suitable for use on a wearable device. Therefore, gestures may be used to interact with the UI operating on the smart glasses 500. Gestures may be generated based on eye gaze characteristics associated with gaze tracking of the user's eyes and head movements of the user of the smart glasses 500.

[0076] In some implementations, the UI can be head-locked, where the UI remains in the same position on the display 505 as the user moves their head and / or the smart glasses 500 move. For example, the overlaid AR content can move with the user's head while remaining within the user's field of view (FOV). As an example, in a head-locked visual UI, the position and orientation of the overlaid AR content in the AR display depend solely on the position and orientation of the viewer's head (as if the AR object or content were securely attached to the viewer's head by a rigid rod), and are independent of any features of the real-world scene. For all types of viewer head movements (e.g., all amplitudes, frequencies, and directions, 6 degrees of freedom (6DoF) movements), the overlaid AR content can move immediately in response to the viewer's head movement onto the real-world scene in the AR display. For example, a circular or sinusoidal frequency head movement will cause the overlaid AR content to move onto the real-world scene at the same circular or sinusoidal frequency as the head movement. Furthermore, for example, tilting or rolling head movements will cause the covered AR content to tilt away from the vertical line of gravity at the same angle as the head movement or rolling.

[0077] In some examples, the sensing system 611 may include various sensing devices, and the control system 612 may include various control system devices, including one or more processors 614, such as those operatively coupled to components of the control system 612. In some examples, the control system 612 may include a communication module that provides communication and information exchange between the smart glasses 500 and other external devices. In some examples, the smart glasses 500 includes a gaze tracking device 615 for detecting and tracking eye gaze direction and movement. The gaze tracking device 615 may include sensors 617, 619 (e.g., a camera). Data captured by the gaze tracking device 615 can be processed to detect and track gaze direction and movement as user input. (The last sentence appears to be a separate, unrelated statement and is left untranslated.) Figure 6B and Figure 6C In the example shown, the gaze tracking device 615 is disposed in one of the two temple portions 630. Figure 6B and Figure 6C In the example arrangement shown, the gaze tracking device 615 and the display device 505 are disposed in the same temple portion 630, such that the user's eye gaze can be tracked not only with respect to objects in the physical environment but also with respect to the content output for display by the display device 505. In some examples, the gaze or gaze tracking device 615 may be disposed in each of the two temple portions 630 to provide gaze tracking for each of the user's two eyes. In some examples, the display device 505 may be disposed in each of the two temple portions 630 to provide binocular display of visual content.

[0078] In some examples, smart glasses 500 includes one or more of the following: an audio output device 506 (such as, for example, one or more speakers), an illumination device 508, a sensing system 611, a control system 612, at least one processor 614, and an externally oriented image sensor or a world-oriented camera 616. In example implementations, the externally oriented image sensor or world-oriented camera 616 may be used to generate image data for generating head pose data. The world-oriented camera 616 may include an inertial measurement unit (IMU). Alternatively (or otherwise), the IMU may be a separate module included in smart glasses 500. For example, the IMU may be included in one of the two temple portions 630. The IMU may include a set of gyroscopes configured to measure rotational speed and an accelerometer configured to measure the acceleration of the world-oriented camera 616 as the head and / or body or user moves. In example implementations, the world-oriented camera 616 and / or IMU may be used for head pose data generation operations (or processing pipelines).

[0079] Figure 7 This is a diagram illustrating an example of a processing circuit system 720. In an example implementation, the processing circuit system 720 may include circuitry configured to generate gaze tracking data (e.g., a signal processing pipeline). The gaze tracking data may be generated based on image data and / or IMU data.

[0080] In some implementations, one or more components of the processing circuitry system 720 may be or may include a processor (e.g., processing unit 724) configured to process instructions stored in memory 726. Figure 7Examples of such instructions depicted include an IMU manager 730, a neural network manager 740, a visual positioning system manager 750, and a gesture manager 760. Furthermore, such as Figure 7 As shown, memory 726 is configured to store various types of data, and its corresponding manager for using such data is described.

[0081] IMU manager 730 is configured to acquire IMU data 733. In some implementations, IMU manager 730 acquires IMU data 733 wirelessly (e.g., from an IMU associated with a world-facing camera 616). Figure 7 As shown, the IMU manager 730 includes an error compensation manager 731 and an integration manager 732.

[0082] Error compensation manager 731 is configured to receive IMU output (IMU data 733) from, for example, IMU manager 730, and use IMU intrinsic parameter values ​​to compensate the IMU output for error. Error compensation manager 731 is then configured to generate IMU data 733 after performing error compensation.

[0083] The integral manager 732 is configured to perform integration operations on the IMU data 733 (e.g., summing time-dependent values). Notably, the rotational velocity data 734 is integrated over time to produce orientation. Furthermore, the acceleration data 735 is integrated twice over time to produce position. Therefore, the integral manager 732 generates a 6DoF attitude (position and orientation) from the IMU output—namely, the rotational velocity data 734 and the acceleration data 735.

[0084] IMU data 733 represents gyroscope and accelerometer measurements in a world coordinate system (not a local coordinate system, i.e., the IMU's coordinate system) that has been compensated for for errors, as well as acceleration data 735. Furthermore, IMU data 733 includes 6DoF attitude and motion data, position data 737, orientation data 738, and velocity data 739 derived from gyroscope and accelerometer measurements. Finally, in some implementations, IMU data 733 also includes IMU temperature data 736; this can indicate further errors in the rotational velocity data 734 and acceleration data 735.

[0085] The neural network manager 740 is configured to acquire rotational velocity data 734 and acceleration data 735 as input and generate neural network data 742 including first position data 744, first orientation data 746, and first velocity data 748. In some implementations, the input rotational velocity data 734 and acceleration data 735 are generated by an error compensation manager 731 acting on the raw IMU output values—where errors are compensated by IMU intrinsic parameter values. For example... Figure 7As shown, the neural network manager 740 includes a neural network training manager 741.

[0086] The neural network training manager 741 is configured to take into account training data 749 and generate neural network data 742, including data about layers and cost functions and values. In some implementations, the training data 749 includes motion data obtained from, for example, measurements of a user wearing smart glasses 500 and moving their head and other parts of their body, as well as ground truth 6DoF pose data obtained from those measurements. In some implementations, the training data 749 includes measured rotational velocities and accelerations from the motion, paired with the measured 6DoF poses and velocities.

[0087] Furthermore, in some implementations, the neural network manager 740 uses historical data from the IMU to generate first position data 744, first orientation data 746, and first velocity data 748. For example, the historical data is used to augment the training data 749 with a mapping from previous rotational speeds, accelerations, and temperatures to their resulting 6DoF poses and movement outcomes, thereby further refining the neural network. In some implementations, the neural network represented by the neural network manager 740 is a convolutional neural network, where the layers are convolutional layers.

[0088] The Visual Positioning System (VPS) manager 750 is configured to acquire images as input and generate VPS data 752, including second position data 754 and second orientation data 756; in some implementations, the VPS data also includes second velocity data 758, i.e., image-based 6DoF pose. In some implementations, images are acquired using a world-facing camera (e.g., world-facing camera 616) on the frame of the smart glasses 500.

[0089] Furthermore, the processing circuitry system 720 may include a network interface 722, one or more processing units 724, and a non-transitory memory 726. The network interface 722 includes, for example, an Ethernet adapter, a token ring adapter, a Bluetooth adapter, a WiFi adapter, an NFC adapter, etc., for converting electronic and / or optical signals received from the network into electronic form for use by the processing circuitry system 720. The collection of processing units 724 includes one or more processing chips and / or assemblies. The memory 726 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid-state drives, and the like. The collection of processing units 724 and the memory 726 together form a processing circuitry system configured and arranged to perform the various methods and functions described herein. Therefore, the collection of processing units 724 and the memory 726 together form a processing circuitry system, and the signal processing pipeline is configured to use a harmonic exponential filter (e.g., included in the gesture manager 760) to generate corrected head posture data.

[0090] Example 1. Figure 8 This is a block diagram of the method for operating a wearable device based on the example implementation. For example... Figure 8 As shown, in step S805, eye gaze data is obtained (e.g., received). In step S810, eye gaze characteristics of the user of the wearable device are determined based on the eye gaze data. For example, the wearable device may include an eye tracking module configured to track eye movement using, for example, a rear camera, an inward-facing camera, or an eye-facing camera. The eye tracking module may be configured to generate, communicate, and / or store eye gaze characteristics. Eye gaze characteristics may include saccades, eye gaze vectors, blink vectors, blink points, timestamps associated with eye gaze vectors, and / or velocities associated with eye gaze vectors.

[0091] In step S815, head movement data is obtained (e.g., received). In step S820, the user's head movement is determined based on the head movement data. For example, the wearable device may include an IMU for tracking the head movement of a user wearing the wearable device. The IMU may be configured to generate, store, and / or communicate IMU data as head movement data including (or used to generate) head motion vectors. For example, the IMU data may include tracking (e.g., storing timestamped data) including the head's position, orientation, etc. The head motion vectors may be calculated based on the IMU data.

[0092] In step S825, a gesture is determined based on eye gaze characteristics, head movement, and the correlation between eye gaze characteristics and head movement. In an example implementation, gaze tracking can be used to determine a gesture by combining the current gaze position, head movement, and gaze movement. For example, a gesture can be determined by determining the current gaze position, determining the head movement, and determining the gaze movement, and combining these three into a gesture. For example, refer to... Figure 3E The gesture can be determined based on gaze position 305, head movement along line 320, and gaze movement along line 325. Furthermore, whenever the user's eye gaze changes, the head movement should follow the same vector as the eye gaze vector, thus confirming eye gaze. Movement along the same vector indicates the correlation between eye gaze characteristics and head movement. If not, eye gaze can be identified as a faulty measurement (e.g., incorrect eye gaze detection).

[0093] In step S830, a UI element of a head-locked user interface (UI) that is displayed, manipulated, processed, or performed on the wearable device is selected based on a gesture. In some implementations, the gesture may be used to identify a gaze point on the UI. In some implementations, the mapping may include a link between the gaze point and the UI element located at the gaze point. Therefore, determining a gesture may include identifying the gaze point and using the mapping to identify the UI element located at the gaze point within the UI. In step S835, a UI operation is triggered based on the UI element. The UI element may have an associated UI operation. For example, the UI operation may be configured to display an image. Therefore, interaction with the UI element located at the second gaze point may trigger the display of an image on the wearable device's display. Details associated with each of these steps can be further described above.

[0094] Example 2. The method as described in Example 1, wherein the eye gaze characteristic may include at least one of an eye gaze saccade and a gaze point. In some implementations, an eye gaze saccade may indicate whether the eye gaze is moving within a UI element or moving to, for example, another UI element. For example, a UI element may be a menu comprising multiple menu items. An associated eye gaze saccade may indicate whether the eye gaze is moving between menu items (e.g., a slow eye gaze saccade) or moving to a different UI element (e.g., a fast eye gaze saccade). In some implementations, if the eye gaze saccade is below a threshold value, the method of Example 1 may stop after step S810 because it is not necessary to generate a gesture based on the eye gaze. Alternatively or additionally, a UI element may include child elements (e.g., menu items). Therefore, a second gaze point may be associated with a child element within a UI element.

[0095] Example 3. As described in Example 1, head movement may include a head motion vector. For example, the wearable device may include an IMU for tracking the head movement of a user wearing the wearable device. The IMU may be configured to generate, store, and / or communicate IMU data as head motion data including (or used to generate) the head motion vector. For example, the IMU data may include tracking (e.g., storing timestamped data) including the head's position, orientation, etc. The head motion vector may be calculated based on the IMU data. Furthermore, whenever the user's eye gaze changes from one gaze point to another, and the head movement should follow the same vector as the eye gaze vector, an eye gaze is confirmed. If not, the eye gaze can be identified as a faulty measurement (e.g., erroneous eye gaze detection).

[0096] Example 4. As described in Example 1, wherein UI elements can be associated with an N×N block of UI, and UI operations cause changes to the properties of the UI elements.

[0097] Example 5. The method as described in Example 1, wherein the correlation between eye gaze characteristics and head movement can indicate that the user is looking outside the boundaries of the UI, and UI actions cause the UI to be hidden.

[0098] Example 6. The method described in Example 1 may further include identifying one of an incorrect eye gaze detection, head motion detection, or a missed UI interaction, and in response to identifying one of the incorrect eye gaze detection, head motion detection, or a missed UI interaction, causing the UI operation not to be performed.

[0099] Example 7. The method described in Example 1 may further include determining a second eye gaze characteristic, determining a second head movement, selecting a second UI element as an indicator correlation based on the correlation between the second eye gaze characteristic and the head movement characteristic, and triggering a user interface (UI) operation based on the selected second UI element.

[0100] Example 8. A method may include any combination of one or more examples from Example 1 to Example 7.

[0101] Example 9. A non-transitory computer-readable storage medium including instructions stored thereon, which, when executed by at least one processor, are configured to cause a computing system to perform a method as described in any one of Examples 1 to 8.

[0102] Example 10. An apparatus comprising a component for performing the method as described in any one of Examples 1 to 8.

[0103] Example 11. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one of Examples 1 to 8 via the at least one processor.

[0104] Generally, the eye gaze characteristics of a user selecting a UI element can be determined by relating them to the characteristics of the user's eye gaze that can be measured via a sensor device of a wearable device, such as a head-mounted device, and involve at least one eye movement and / or visual gaze point of the user. For example, eye gaze characteristics may include at least one of eye gaze saccades (i.e., determination that at least one eye gaze saccade has occurred) and gaze point.

[0105] Triggering UI actions based on the selected UI element can include triggering UI actions that are related to and therefore based on the selection of that UI element.

[0106] In some implementations, head movement may include head motion vectors; that is, the head movement used for gesture determination is determined based on head motion vectors.

[0107] Furthermore, UI elements can be associated with an N×N block of UI, and UI operations cause changes to the characteristics of the UI elements. An N×N block can be N columns by N rows. A block can be one or more pixels. For example, a UI operation can cause a change in the characteristics of the selected UI element (including the displayed characteristics of the selected UI element). In one embodiment, a UI operation can cause the selected UI element to be highlighted in an N×N block of UI by changing its color, removing the selected UI element from the UI, making the selected UI element 4D, or surrounding the selected UI element with a box on the UI. An N×N block can be an N column block by N row blocks. A block can be one or more pixels.

[0108] In some implementations, the correlation between eye gaze characteristics and head movement can indicate that the user is looking outside the boundaries of the UI, and UI actions cause the UI to be hidden.

[0109] Furthermore, during the determination of eye gaze characteristics, head movement, or gesture, one of the following can be detected: erroneous eye gaze detection, erroneous head movement detection, or missed UI interaction. Therefore, when attempting to determine a gesture based on eye gaze characteristics, head movement, and the correlation between eye gaze characteristics and head movement, a gesture error can be determined, where this gesture error causes the UI operation not to be performed. Thus, gesture error determination can include detecting erroneous eye gaze, detecting erroneous head movement, and / or detecting a missed interaction or gesture. In response to recognizing one of erroneous eye gaze detection, erroneous head movement detection, or missed UI interaction, the UI operation may be prevented from being performed.

[0110] Example implementations may include a non-transitory computer-readable storage medium including instructions stored thereon configured to cause a computing system to perform any of the methods described above when executed by at least one processor. Example implementations may include a device including components for performing any of the methods described above. Example implementations may include a device including at least one processor and at least one memory including computer program code configured to cause the device to perform at least any of the methods described above using the at least one processor.

[0111] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, and is coupled to receive data and instructions from and to the storage system, at least one input device, and at least one output device.

[0112] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0113] To provide interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user and a keyboard and pointing device (e.g., mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or middleware components (e.g., application servers), or front-end components (e.g., client computers with graphical user interfaces or web browsers through which users can interact with the implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0115] A computing system may include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is established by computer programs that execute on the respective computers and establish a client-server relationship between them.

[0116] Several implementation methods have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0117] Furthermore, the logical flow depicted in the figures does not require the desired result to be achieved in the specific order or sequence shown. Additionally, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system. Therefore, other implementations are within the scope of the appended claims.

[0118] While certain features of the described implementations have been described herein, those skilled in the art will now conceive of numerous modifications, substitutions, alterations, and equivalents. Therefore, it should be understood that the appended claims are intended to cover all such modifications and alterations falling within the scope of the implementations. It should be understood that they are presented by way of example only and not limitation, and various changes in form and detail are possible. Any part of the apparatus and / or method described herein can be combined in any combination, except for mutually exclusive combinations. The implementations described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

[0119] While the exemplary implementations may include various modifications and alternatives, their implementations are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the exemplary implementations are not intended to be limited to the specific forms disclosed, but rather, the exemplary implementations will cover all modifications, equivalents, and alternatives falling within the scope of the claims. Similar figures refer to the same elements in the description of the accompanying drawings.

[0120] Some of the above example implementations are described as processes and methods depicted as flowcharts. Although flowcharts describe operations as sequential processes, many operations can be executed in parallel, concurrently, or simultaneously. Furthermore, the order of operations can be rearranged. Processes can terminate upon completion of their operations, but may also have additional steps not included in the diagram. Processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0121] The methods discussed above (some of which are illustrated by flowcharts) can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments used to perform the necessary tasks can be stored in a machine or computer-readable medium, such as a storage medium. One or more processors can perform the necessary tasks.

[0122] The specific structural and functional details disclosed in this article are representative only for the purpose of describing the example implementation. However, the example implementation is embodied in many alternative forms and should not be construed as being limited to the implementation described in this article.

[0123] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the example implementation, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, terms and / or include any and all combinations of one or more of the associated listed items.

[0124] It should be understood that when an element is mentioned as being connected to or coupled to another element, that element may be directly connected to or coupled to that other element, or there may be an intermediary element. Conversely, when an element is mentioned as being directly connected to or directly coupled to another element, there is no intermediary element. Other terms used to describe the relationship between elements should be interpreted in a similar manner (e.g., between vs. directly between, adjacent vs. directly adjacent, etc.).

[0125] The terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the example implementations. As used herein, the singular forms "a," "an," and "this" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms include, contain, contain, and / or have specify the presence of the stated features, integers, steps, operations, elements, and / or components as used herein, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0126] It should also be noted that in some alternative implementations, the indicated functions / actions may not occur in the order shown in the diagram. For example, two diagrams shown consecutively may actually be executed simultaneously or sometimes in reverse order, depending on the functionality / behavior involved.

[0127] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the example implementations pertain. It should be further understood that terms defined, for example, in commonly used dictionaries, shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0128] The above example implementations, along with corresponding detailed descriptions, are presented in relation to the symbolic representation of software or algorithms and operations on data bits within computer memory. These descriptions and representations are intended to effectively communicate the essence of their work to others of ordinary skill in the art. An algorithm, as used herein and as it is commonly used, is considered a self-consistent sequence of steps that leads to a desired result. A step is a step that requires physical manipulation of physical quantities. Typically, although not always necessary, these quantities take the form of optical, electrical, or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. Primarily for general reasons, these signals are referred to as bits, values, elements, symbols, characters, items, numbers, or other terms that have sometimes proven convenient.

[0129] In the above exemplary implementations, references to symbolic representations (e.g., in the form of flowcharts) of actions and operations that can be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types, and can be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computers, etc.

[0130] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specifically indicated or as apparent from the discussion, terms such as processing or calculation or computation or determination, display or other terms refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical electronic quantities in the registers and memories of the computer system and converts that data into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission or display devices.

[0131] It should also be noted that the software implementation aspects of the example implementations are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium can be magnetic (e.g., floppy disk or hard disk) or optical (e.g., optical disc read-only memory or CD ROM), and can be read-only or random access. Similarly, the transmission medium can be twisted-pair cables, coaxial cables, optical fibers, or some other suitable transmission medium known in the art. The example implementations are not limited to these aspects of any given implementation.

[0132] Finally, it should be noted that although the appended claims set forth a particular combination of features described herein, the scope of this disclosure is not limited to the particular combination claimed herein, but extends to any combination covering the features or implementations disclosed herein, regardless of whether such particular combination has been specifically enumerated in the appended claims hereto.

Claims

1. A non-transitory computer-readable storage medium comprising instructions stored thereon, the instructions being configured, when executed by at least one processor, to cause a computing system to: Determine the eye gaze characteristics of users of wearable devices; Determine the user's head movement; Gestures are determined based on the eye gaze characteristics, the head movement, and the correlation between the eye gaze characteristics and the head movement. The gesture is used to select UI elements displayed on the wearable device's head-locked user interface. as well as UI actions are triggered based on the selected UI element.

2. The non-transitory computer-readable storage medium as claimed in claim 1, wherein, The eye gaze characteristics include at least one of eye gaze saccades and gaze points.

3. The non-transitory computer-readable storage medium as described in claim 1 or claim 2, wherein, The head movement includes head motion vectors.

4. The non-transitory computer-readable storage medium according to any one of claims 1 to 3, wherein The UI element is associated with an N×N block of the UI, and The UI operation causes changes to the properties of UI elements.

5. The non-transitory computer-readable storage medium according to any one of claims 1 to 4, wherein... The correlation between the eye gaze characteristics and the head movement indicates that the user is looking outside the boundaries of the UI, and The UI operation caused the UI to be hidden.

6. The non-transitory computer-readable storage medium of any one of claims 1 to 5, wherein the instructions further cause the computing system to: Identify one of the following: incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction. In response to the detection of one of incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction, the UI operation is not performed.

7. The non-transitory computer-readable storage medium of claim 6, wherein the instructions further cause the computing system to: Determine the second eye's fixation characteristics; Determine the movement of the second head; The second UI element is selected as the indicator relevance based on the correlation between the second eye gaze characteristic and the head movement characteristic; and UI actions are triggered based on the selected secondary user interface (UI) element.

8. A method comprising: Determine the eye gaze characteristics of users of wearable devices; Determine the user's head movement; Whether the user of the wearable device intends to perform a gesture is determined based on at least one of the eye gaze characteristics and the head movement; as well as Perform a gesture in response to determining the user's intent on the wearable device; Gestures are determined based on the eye gaze characteristics, the head movement, and the correlation between the eye gaze characteristics and the head movement. The gesture is used to select UI elements displayed on the wearable device's head-locked user interface. as well as UI actions are triggered based on the selected UI element.

9. The method of claim 8, wherein, The eye gaze characteristics include at least one of eye gaze saccades and gaze points.

10. The method of claim 8 or claim 9, wherein, The head movement includes head motion vectors.

11. The method according to any one of claims 8 to 10, wherein The UI element is associated with an N×N block of the UI, and The UI operation causes changes to the properties of UI elements.

12. The method according to any one of claims 8 to 11, wherein The correlation between the eye gaze characteristics and the head movement indicates that the user is looking outside the boundaries of the UI, and The UI operation caused the UI to be hidden.

13. The method of any one of claims 8 to 12, further comprising: Identify one of the following: incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction. In response to the detection of one of incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction, the UI operation is not performed.

14. The method of claim 13, further comprising: Determine the second eye's fixation characteristics; Determine the movement of the second head; The second UI element is selected as the indicator relevance based on the correlation between the second eye gaze characteristic and the head movement characteristic; as well as UI actions are triggered based on the selected secondary user interface (UI) element.

15. An apparatus comprising at least one processor and at least one memory including computer program code, said at least one memory and said computer program code being configured to cause the apparatus using said at least one processor: Determine the eye gaze characteristics of users of wearable devices; Determine the user's head movement; Gestures are determined based on the eye gaze characteristics, the head movement, and the correlation between the eye gaze characteristics and the head movement. The gesture is used to select UI elements displayed on the wearable device's head-locked user interface. as well as UI actions are triggered based on the selected UI element.

16. The device as claimed in claim 15, wherein, The eye gaze characteristics include at least one of eye gaze saccades and gaze points.

17. The device as claimed in claim 15 or claim 16, wherein, The head movement includes head motion vectors.

18. The device as claimed in any one of claims 15 to 17, wherein The UI element is associated with an N×N block of the UI, and The UI operation causes changes to the properties of UI elements.

19. The device according to any one of claims 15 to 18, wherein the computer program code further causes the computing system to: Identify one of the following: incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction. In response to the detection of one of incorrect eye gaze detection, incorrect head motion detection, or missed UI interaction, the UI operation is not performed.

20. The device of claim 19, wherein the computer program code further causes the computing system to: Determine the second eye's fixation characteristics; Determine the movement of the second head; The second UI element is selected as the indicator relevance based on the correlation between the second eye gaze characteristic and the head movement characteristic; and UI actions are triggered based on the selected secondary user interface (UI) element.

Images