Gaze tracking for user interface

Abstract

A gaze tracking for tracking a user's gaze within a user's viewing screen includes combining gaze tracking with head movement tracking wherein the head movement of the head movement tracking provides a coarse estimate of the user's gaze direction, while eye movement of gaze tracking provides fine tuning of the user's gaze direction within the user's viewing screen; dividing the user viewing screen into a plurality of gaze areas when the gaze area estimation is turned on; and combining the gaze tracking with the gaze area estimation to select a gaze area from a plurality of gaze areas as a gaze direction of the user.

Description

Background Art

[0001] The present disclosure relates to gaze tracking, and more particularly, to gaze tracking using eye movements as user input. Background Art

[0002] Sometimes human eyes may move or twitch in different directions without any specific purpose or reason. Therefore, detailed gaze or eye tracking is usually not used as input in games or other applications because it may generate high-fidelity noise. Summary of the invention

[0003] The present disclosure provides gaze tracking to track a user's gaze within a screen that the user is viewing.

[0004] In one implementation, a method for gaze tracking to track a user's gaze within a user viewing screen is disclosed. The method includes combining gaze tracking with head movement tracking, wherein head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and eye movement of the gaze tracking provides fine adjustment of the user's gaze direction within the user viewing screen. The method also includes dividing the user viewing screen into a plurality of gaze regions when gaze region estimation is turned on. The method also includes combining gaze tracking with gaze region estimation to select a gaze region from the plurality of gaze regions as the user's gaze direction.

[0005] In another implementation, a gaming device is disclosed. The gaming device includes a sensor and a processor. The sensor tracks gaze movement of a user. The processor is operable to (a) obtain information about the gaze movement from the sensor, and (b) execute an application to display a gaze icon and a notification icon based on the gaze information within a screen image of the application on a display. The notification icon is displayed at a predetermined position of the screen image.

[0006] In yet another implementation, a system for gaze tracking to track a user's gaze within a user viewing screen includes a gaze determiner, a processor, and a user interface. The processor combines gaze tracking with head movement tracking, wherein head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and eye movement of the gaze tracking provides fine adjustment of the user's gaze direction within the user viewing screen. The gaze determiner directs the processor to divide the user viewing screen into a plurality of gaze regions, and when the gaze determiner determines that the gaze region estimation is turned on, the gaze tracking is combined with the gaze region estimation to select a gaze region from the plurality of gaze regions as the user's gaze direction.

[0007] In a further implementation, a non-transitory computer-readable storage medium storing a computer program for tracking a user's gaze within a user viewing screen is disclosed. The computer program includes executable instructions that cause a computer to combine gaze tracking with head movement tracking, wherein head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and eye movement of the gaze tracking provides fine-tuning of the user's gaze direction within the user viewing screen. The computer program also includes executable instructions that cause a computer to divide the user viewing screen into a plurality of gaze regions when gaze region estimation is turned on. The computer program also includes executable instructions that cause a computer to combine gaze tracking with gaze region estimation to select a gaze region from a plurality of gaze regions as the user's gaze direction.

[0008] Other features and advantages will be apparent from the present description, which illustrates by way of example various aspects of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The details of the present disclosure, both as to its structure and operation, may be gleaned in part by examination of the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0010] Figure 1 is a flow chart of a process for gaze tracking of a user interface according to one implementation of the present disclosure;

[0011] Figure 2 is a block diagram of a gaze tracking system for gaze tracking of a user interface according to one implementation of the present disclosure;

[0012] Figure 3A is an example of a user viewing screen according to one implementation of the present disclosure;

[0013] Figure 3B showing a user moving a gaze icon toward a notification icon by moving the gaze toward the notification icon;

[0014] Figure 3C shows the gaze icon being moved to a decision area around the notification icon;

[0015] Figure 3D showing a viewing screen including a message panel which replaces a notification icon with details of a received message when a gaze icon moves into a decision area;

[0016] Figure 4A are representations of computer systems and users according to implementations of the present disclosure; and

[0017] Figure 4B is a functional block diagram illustrating a computer system hosting a gaze tracking application according to an implementation of the present disclosure. DETAILED DESCRIPTION

[0018] As mentioned above, human eyes can move or twitch in different directions without any particular purpose or reason. Therefore, detailed gaze or eye tracking is not usually used as input in games or interactive programs because it may generate high-fidelity noise. However, in some cases, gaze tracking can be used in conjunction or in combination with other methods and systems to enable gaze tracking to be used with user interfaces.

[0019] Certain implementations of the present disclosure provide methods and systems for gaze tracking of user interfaces. In one implementation, gaze tracking is combined with head movement tracking to provide fine-tuning. In another implementation, gaze area estimation is used to mitigate high-fidelity noise introduced in detailed eye tracking. Therefore, in the gaze area estimation, one or more gaze areas can be extended or highlighted in (one or more) overlay pictures or picture-in-picture (PIP) mode. In another implementation, gaze tracking is used to provide notifications to users so that users are minimally disturbed when, for example, playing games or experiencing other applications.

[0020] After reading the following description, it will become clear how to implement the present disclosure in various implementations and applications. Although various implementations of the present disclosure will be described herein, it should be understood that these implementations are presented only by way of example, not limitation. Therefore, the detailed description of various implementations should not be interpreted as limiting the scope or breadth of the present disclosure.

[0021] Figure 1 is a flow chart of a process 100 for gaze tracking of a user interface according to one implementation of the present disclosure. In one implementation, the process 100 implements gaze tracking for a user interface by combining gaze tracking with other methods and systems including head movement tracking and gaze region estimation. In one implementation, the process 100 is programmed into a hardware processor to perform gaze tracking within the combined system.

[0022] exist Figure 1 In the illustrated implementation, it is determined whether head movement tracking is turned on at step 110. If it is determined at step 110 that head movement tracking is turned on, gaze tracking is combined with head movement tracking to provide fine-tuning at step 120. For example, head movement tracking provides a rough estimate of the direction the user's head is facing, while gaze tracking provides a more detailed estimate of the direction of the user's head.

[0023] At step 130, it is determined whether gaze region estimation is on. In one implementation, at step 140, gaze region estimation divides the user viewing screen into a plurality of regions (e.g., rectangular boxes of preset size), and uses gaze tracking to estimate the direction of the user's gaze within the selected regions. If it is determined at step 130 that gaze region estimation is on, then at step 150, gaze tracking is combined with gaze region estimation to select a gaze region. In one implementation, one or more gaze regions (including the selected region) may be expanded or highlighted in an overlay picture or picture-in-picture (PIP) mode to show a detailed view of an object or person that the user is looking at.

[0024] Then, it is determined in step 160 whether a notification has been received. In one implementation, the notification includes a communication to the user while the user is playing or experiencing a game or other interactive application. If it is determined in step 160 that a notification has been received, a notification is provided on the user's viewing screen in step 170, indicating that the communication has been received. In one implementation, the communication includes messages from other users who play or experience the same game or interactive application as the user. In another implementation, the communication includes a message from a gaming system. In a further implementation, the notification is provided as a bubble icon on the user's viewing screen. In one implementation, the details of the message are displayed on the user's viewing screen when the user looks in the direction of the icon. The above-described notification method enables notifications to be provided to users playing, for example, interactive games with minimal interference.

[0025] In a specific implementation, a method for gaze tracking to track a user's gaze within a user viewing screen is disclosed. The method includes: combining the gaze tracking with head movement tracking, wherein the head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and the eye movement of the gaze tracking provides a fine adjustment of the user's gaze direction within the user viewing screen; when gaze area estimation is turned on, dividing the user viewing screen into a plurality of gaze areas; and combining the gaze tracking with the gaze area estimation to select a gaze area from the plurality of gaze areas as the user's gaze direction.

[0026] In one implementation, the method further includes displaying an icon on the user's viewing screen when gaze tracking is turned on. In one implementation, each of the multiple gaze areas is a rectangular box of a preset size. In one implementation, one or more gaze areas around the selected area are expanded or highlighted in an overlay picture or picture-in-picture mode to show a detailed view of an object or person within the user's gaze. In one implementation, the method further includes displaying a notification icon on the user's viewing screen when a notification is received. In one implementation, the notification includes a message from a gaming system. In one implementation, the notification includes a message from another user in the gaming system. In one implementation, the method further includes displaying a message on the user's viewing screen when it is determined that the user's gaze has moved to the notification icon.

[0027] Figure 2 FIG. 3 is a block diagram of a gaze tracking system 200 for gaze tracking of a user interface according to one implementation of the present disclosure. In the illustrated implementation of FIG. 3 , the system 200 includes a gaze determiner 220, a processor 230, and a user interface 240 coupled to the processor 230 and the gaming system 210. In one implementation, the blocks 220, 230, 240 of the system 200 are configured entirely in hardware, the hardware comprising one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. In one implementation, the system 200 combines gaze tracking with other methods including at least one of: head movement tracking, gaze region estimation, and message notification.

[0028] exist Figure 2 In the illustrated implementation, the gaze determiner 220 determines whether head movement tracking is turned on. If it is determined that head movement tracking is turned on, the gaze determiner 220 instructs the processor 230 to combine gaze tracking with head movement tracking to provide fine-tuning. For example, head movement tracking provides a rough estimate of the direction the user's head is facing, while gaze tracking provides a more detailed estimate of the direction of the user's head. In one implementation, head movement tracking is performed by (one or more) accelerometers and / or (one or more) gyroscopes disposed within the head-mounted device. In another implementation, gaze tracking is performed by (multiple) cameras disposed within the head-mounted device.

[0029] The gaze determiner 220 then determines whether gaze region estimation is on. In one implementation, the gaze region estimation divides the user viewing screen into a plurality of regions (e.g., rectangular boxes of a preset size) and uses gaze tracking to estimate the direction of the user's gaze within the selected regions. If it is determined that the gaze region estimation is on, the gaze determiner 220 instructs the processor 230 to combine gaze tracking with the gaze region estimation to select a gaze region. In one implementation, one or more gaze regions (including the selected region) may be expanded or highlighted in an overlay picture or picture-in-picture (PIP) mode to show a detailed view of an object or person that the user is looking at.

[0030] The gaze determiner 220 also determines whether a notification is received. In one implementation, the notification includes a communication to the user while the user is playing or experiencing a game or interactive application. If it is determined that a notification is received, the gaze determiner 220 instructs the processor 230 to display a notification indicating the receipt of the communication on the user's viewing screen. In one implementation, the communication includes messages from other users who play or experience the same game or interactive application as the user. In another implementation, the communication includes a message from a gaming system. In a further implementation, the notification is provided as a bubble icon on the user's viewing screen. In one implementation, when the user gazes in the direction of the icon, the details of the message are displayed on the user's viewing screen. Therefore, the gaze determiner 220 and the processor 230 provide gaze tracking for the user interface 240. The above notification method provides notifications to users playing, for example, interactive games with minimal interference.

[0031] In one implementation, the gaming system 210 includes a gaming console that provides interactive games or other applications. In another implementation, the gaming system 210 includes online interactive games or other applications. In yet another implementation, the gaming system 210 includes virtual reality and / or augmented reality interactive games or applications.

[0032] In one implementation, the gaze tracking feature is used by game developers / central commanders to customize games or experiences. For example, in a shooting game, the user's gaze can be used to switch weapons or weapon functions in real time.

[0033] Figure 3A is an example of the above-mentioned user viewing screen 300 according to one implementation of the present disclosure. Figure 3A The user viewing screen 300 includes two icons 310, 320. Icon 310 may appear at the start of an interactive game, for example, to indicate to the user that the gaze tracking mode is turned on. When a notification message is received, icon 320 appears on screen 300.

[0034] like Figure 3AAs shown in the example of FIG. 3 , the gaze icon 310 is displayed on the viewing screen 300 to indicate to the user that the gaze tracking mode is turned on. In one implementation, the gaze icon 310 is an eye-shaped icon. In another implementation, the gaze icon 310 is a cursor.

[0035] In one implementation, messages may be received while the user is playing a game. Thus, when a message is received, a notification icon 320 may be displayed in a predetermined area of ​​the viewing screen 300. The message that triggers the notification icon 320 may be related to the game currently being played (e.g., an announcement from a game provider), or it may be unrelated to the game (e.g., a text from a friend). When a message is received and the notification icon 320 is displayed, the gaze icon 310 may be moved by the user directing their gaze toward the notification icon 320.

[0036] Figure 3B The user is shown moving gaze icon 310 towards notification icon 320 by moving gaze 312 towards notification icon 320 .

[0037] Figure 3C It is shown that the gaze icon 310 is moved into the decision area 322 around the notification icon 320. Therefore, when the gaze icon 310 is moved into the decision area 322, additional information about the message may be displayed.

[0038] Figure 3D A viewing screen 350 including a message panel 330 is shown. When the gaze icon 310 is moved to the Figure 3C When the decision area 322 is shown, the message panel replaces the notification icon 320 with the details of the received message 332. Thus, Figures 3A to 3D The implementation enables the user to simply check the content of the message 332 corresponding to the notification icon 320, for example, without interfering with the game and without operating the game controller used by the user for the game. When the user wishes to perform any additional actions related to the message 332, the user can use the game controller to perform the action. For example, the user can check the full text of the message 332 by launching the message application or by replying to the message 332. In this example, the game application can then be paused to perform additional actions.

[0039] In one implementation, when the gaze icon 310 stays within the decision area 322 for a predetermined period of time, an additional action related to the message 332 may be performed. In another implementation, after the gaze icon 310 moves into the decision area 322, the additional action may be performed by a user input such as a game controller or a voice command. In one implementation, the decision area 322 is the same or different size as the gaze icon 310. In another implementation, when the gaze icon 310 moves into the decision area 322, the notification icon 320 changes size. In a further implementation, the notification icon 320 is highlighted to help the user easily understand the status of the message 332.

[0040] In one implementation, once the notification icon 320 is selected (e.g., by moving the gaze icon 310 into the decision area 322), the message 332 is displayed in the message panel 330 so that the user can check the message. The size of the message panel 330 can be larger than the notification icon 320 to accommodate the large-sized message 332. In one implementation, the notification icon 320 and the additional information (e.g., the message panel 330 and the message 332) disappear from the viewing screen 300 after a predetermined period of time (after it appears on the screen). Therefore, the above implementation enables the user to perform a gaze operation to check the message without interfering with the game screen or game control operation currently being played.

[0041] In a specific implementation, a gaming device is disclosed. The gaming device includes: a sensor for tracking gaze movement of a user; a processor operable to: obtain information about the gaze movement from the sensor; and execute an application to display a sight line icon and a notification icon based on the sight line information within a screen image of the application on a display, wherein the notification icon is displayed at a predetermined position of the screen image.

[0042] In one implementation, the application is a gaming application. In one implementation, the notification icon receives a notification to be displayed from an application other than the gaming application. In one implementation, the gaming device is a head mounted display (HMD).

[0043] In another specific implementation, a system for gaze tracking to track the gaze of a user within a user viewing screen is disclosed. The system includes: a processor, the processor is used to combine the gaze tracking with head movement tracking, wherein the head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and the eye movement of the gaze tracking provides a fine adjustment of the user's gaze direction within the user viewing screen; a gaze determiner, which is used to instruct the processor to divide the user viewing screen into a plurality of gaze areas, and when the gaze determiner determines that the gaze area estimation is turned on, the gaze tracking is combined with the gaze area estimation to select a gaze area from the plurality of gaze areas as the user's gaze direction.

[0044] In one implementation, head movement tracking is performed by at least one accelerometer and at least one gyroscope disposed within a head mounted device that communicates with the gaming system. In one implementation, gaze tracking is performed by at least one camera disposed within the head mounted device. In one implementation, the system further comprises a user interface coupled to the processor, the user interface being configured to display an icon on a user viewing screen when gaze tracking is turned on. In one implementation, the user interface further displays a notification icon on the user viewing screen when a notification is received. In one implementation, the notification comprises a message from the gaming system. In one implementation, the notification comprises a message from another user in the gaming system. In one implementation, the gaming system comprises a gaming console that provides an interactive game.

[0045] Figure 4A is a representation of a computer system 400 and a user 402 according to an implementation of the present disclosure. User 402 uses computer system 400 to implement Figure 1 The process of 100 and Figure 2 A gaze tracking application 490 for gaze tracking of a user interface of the system 200 .

[0046] Computer system 400 stores and executes Figure 4B In addition, the computer system 400 can communicate with a software program 404. The software program 404 can include software code for the gaze tracking application 490. The software program 404 can be loaded onto an external medium such as a CD, DVD, or storage drive, as will be explained further below.

[0047] In addition, the computer system 400 can be connected to a network 480. The network 480 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, the network 480 can communicate with a server 485 that coordinates engines and data used within the gaze tracking application 490. Moreover, the network can be different types of networks. For example, the network 480 can be the Internet, a local area network or a local area network, a wide area network, a metropolitan area network, an intranet or an extranet, or any variation of a wireless network.

[0048] Figure 4B 4 is a functional block diagram illustrating a computer system 400 hosting a gaze tracking application 490 according to an implementation of the present disclosure. The controller 410 is a programmable processor and controls the operation of the computer system 400 and its components. The controller 410 loads instructions (e.g., in the form of a computer program) from the memory 420 or an embedded controller memory (not shown) and executes these instructions to control the system, such as to provide data processing. In its execution, the controller 410 provides a software system to the gaze tracking application 490. Alternatively, the service may be implemented as a separate hardware component in the controller 410 or the computer system 400.

[0049] The memory 420 temporarily stores data for use by other components of the computer system 400. In one implementation, the memory 420 is implemented as RAM. In one implementation, the memory 420 also includes long-term or permanent storage, such as flash memory and / or ROM.

[0050] The memory 430 stores data temporarily or long term for use by other components of the computer system 400. For example, the storage device 430 stores data used by the gaze tracking application 490. In one implementation, the storage device 430 is a hard drive.

[0051] The media device 440 receives removable media and reads and / or writes data to the inserted media. For example, in one implementation, the media device 440 is an optical disk drive.

[0052] The user interface 450 includes components for accepting user input from a user of the computer system 400 and presenting information to the user 402. In one implementation, the user interface 450 includes a keyboard, a mouse, an audio speaker, and a display. In another implementation, the user interface 450 also includes a head mounted device worn by the user and used to collect eye movement as user input. The controller 410 uses the input from the user 402 to adjust the operation of the computer system 400.

[0053] The I / O interface 460 includes one or more I / O ports to connect to corresponding I / O devices, such as external storage devices or supplementary devices (e.g., printers or PDAs). In one implementation, the ports of the I / O interface 460 include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another implementation, the I / O interface 460 includes a wireless interface for wireless communication with external devices.

[0054] Network interface 470 includes a wired and / or wireless network connection, such as an RJ-45 or "Wi-Fi" interface supporting an Ethernet connection (including but not limited to 802.11).

[0055] Computer system 400 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), although for simplicity these components are not shown in the figure. Figure 4B In other implementations, different configurations of the computer system (eg, different bus or storage configurations or multi-processor configurations) may be used.

[0056] In a specific implementation, a non-transitory computer-readable storage medium storing a computer program is disclosed, the computer program being used to track a user's gaze within a user viewing screen. The computer program includes executable instructions that cause a computer to: combine the gaze tracking with head movement tracking, wherein the head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and the eye movement of the gaze tracking provides a fine adjustment of the user's gaze direction within the user viewing screen; when gaze area estimation is turned on, divide the user viewing screen into a plurality of gaze areas; and combine the gaze tracking with the gaze area estimation to select a gaze area from the plurality of gaze areas as the user's gaze direction.

[0057] In one implementation, the computer program also includes executable instructions that cause the computer to: display a notification icon on the user viewing screen when a notification is received; and display a message of the notification on the user viewing screen when it is determined that the user's gaze has moved to the notification icon.

[0058] The description of the disclosed implementations is provided herein to enable those skilled in the art to make or use the present disclosure. Those skilled in the art will readily appreciate many modifications to these implementations, and the principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the implementations shown herein, but should be given the widest scope consistent with the principles and novel features disclosed herein. Therefore, additional variations and implementations are also possible.

[0059] In a particular implementation of the present disclosure, not all features of each of the above examples are necessarily required. In addition, it should be understood that the description and drawings presented herein represent the broadly contemplated subject matter of the present disclosure. It should also be understood that the scope of the present disclosure fully encompasses other implementations that may become apparent to those skilled in the art, and the scope of the present disclosure is therefore limited only by the appended claims.

Claims

1. A method for gaze tracking to track a user's gaze within a screen viewed by the user, the method comprising: Combining gaze tracking with head movement tracking Wherein the head movement of the head movement tracking provides a rough estimate of the user's gaze direction, and the eye movement of the gaze tracking provides a fine adjustment of the user's gaze direction within the screen viewed by the user; When gaze area estimation is turned on, the user's viewing screen is divided into multiple gaze areas; as well as Gaze tracking is combined with gaze region estimation to select a gaze region from the plurality of gaze regions as a gaze direction of the user.

2. The method of claim 1, further comprising When gaze tracking is turned on, an icon is displayed on the screen the user is watching.

3. The method according to claim 1, wherein: Each of the plurality of gaze areas is a rectangular frame of a preset size.

4. The method of claim 1, wherein one or more gaze areas around the selected area are expanded or highlighted in an overlay picture or picture-in-picture mode to show a detailed view of an object or person within the user's gaze.

5. The method according to claim 1, further comprising When a notification is received, a notification icon is displayed on the user's viewing screen. The method of claim 5 , wherein the notification comprises a message from a gaming system. The method of claim 6 , wherein the notification comprises a message from another user of the gaming system.

8. The method according to claim 7, further comprising When it is determined that the user's gaze has moved to the notification icon, the message is displayed on the user viewing screen.

9. A gaming device comprising: Sensors to track the user's gaze movements; A processor operable to: obtaining information about gaze movement from the sensor; executing an application to display a gaze icon and a notification icon based on the gaze information within a screen image of the application on a display, Wherein, the notification icon is displayed at a predetermined position of the screen image.

10. The gaming device according to claim 9, wherein: The application is a game application.

11. The gaming device according to claim 10, wherein: The notification icon receives notifications to be displayed from an application other than the gaming application.

12. The gaming device according to claim 9, wherein: The gaming device is a head mounted display (HMD).

13. A system for gaze tracking to track a user's gaze within a screen viewed by the user, the system comprising: a processor for combining gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide a rough estimate of the user's gaze direction and eye movements of the gaze tracking provide fine adjustments to the user's gaze direction within a screen viewed by the user; A gaze determiner is configured to instruct the processor to divide the user viewing screen into a plurality of gaze areas, and when the gaze determiner determines that the gaze area estimation is turned on, combine the gaze tracking with the gaze area estimation to select a gaze area from the plurality of gaze areas as the user's gaze direction.

14. The system according to claim 13, wherein: The head movement tracking is performed by at least one accelerometer and at least one gyroscope disposed within a head mounted device that communicates with the gaming system.

15. The system of claim 14, wherein the gaze tracking is performed by at least one camera disposed within the head mounted device.

16. The system according to claim 13, further comprising A user interface coupled to the processor, the user interface being configured to display an icon on the user viewing screen when the gaze tracking is enabled.

17. The system of claim 16, wherein: When a notification is received, the user interface also displays a notification icon on the user viewing screen.

18. The system of claim 17, wherein the notification comprises a message from a gaming system.

19. The system of claim 17, wherein the notification comprises a message from another user of the gaming system.

20. The system of claim 19, wherein the gaming system comprises a gaming console that provides interactive gaming.

Images