Eye Tracking for User Interface

Abstract

Gaze tracking for tracking the user's gaze within the user view screen involves combining gaze tracking and head movement tracking. The head movement in the head movement tracking provides a rough estimate of the direction of the user's gaze, and the eye movement in the gaze tracking provides fine adjustment of the direction of the user's gaze within the user view screen. When the gaze zone estimation is active, the user view screen is divided into a plurality of gaze zones, and the gaze tracking and the gaze zone estimation are combined to select one gaze zone as the direction of the user's gaze from among the plurality of gaze zones.

Description

Technical Field

[0001] The present disclosure relates to eye tracking, and more particularly to eye tracking that uses eye movement as user input.

Background Art

[0002] The human eye may move or tremble in various directions without a specific purpose or reason. Therefore, detailed eye tracking or eyeball tracking may generate high-fidelity noise and is generally not used as input for games or other application programs.

Summary of the Invention

[0003] The present disclosure provides eye tracking for tracking a user's line of sight within a user view screen.

[0004] In one embodiment, an eye tracking method for tracking a user's line of sight within a user view screen is disclosed. The method includes combining eye tracking and head movement tracking. Here, the head movement in head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in eye tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen. Further, the method includes dividing the user view screen into a plurality of eye zones when the eye zone estimation is operative. Further, the method includes combining eye tracking and eye zone estimation to select one eye zone as the direction of the user's line of sight from among the plurality of eye zones.

[0005] In another embodiment, a game device is disclosed. The game device includes a sensor and a processor. The sensor tracks the movement of the user's line of sight. The processor is operable to (a) obtain information regarding the movement of the line of sight from the sensor and (b) execute an application program for displaying an eye icon and a notification icon based on the line of sight information within a screen image of the application program. The notification icon is displayed at a predetermined position on the screen image.

[0006] In yet another embodiment, a system for tracking a user's line of sight within a user view screen is disclosed. The system includes a line-of-sight determination unit, a processor, and a user interface. The processor combines line-of-sight tracking and head movement tracking. Here, the head movement in head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in line-of-sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen. When the line-of-sight determination unit determines that the line-of-sight zone estimation is in operation, it instructs the processor to divide the user view screen into a plurality of line-of-sight zones, and to combine line-of-sight tracking and line-of-sight zone estimation to select one line-of-sight zone as the direction of the user's line of sight from among the plurality of line-of-sight zones.

[0007] In yet another embodiment, a non-transitory computer-readable storage medium storing a computer program for tracking a user's line of sight within a user view screen is disclosed. The computer program includes executable instructions that cause a computer to combine line-of-sight tracking and head movement tracking, where the head movement in head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in line-of-sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen. The computer program further includes executable instructions that cause the computer to divide the user view screen into a plurality of line-of-sight zones when the line-of-sight zone estimation is in operation. The computer program further includes executable instructions that cause the computer to combine line-of-sight tracking and line-of-sight zone estimation to select one line-of-sight zone as the direction of the user's line of sight from among the plurality of line-of-sight zones.

[0008] Other features and effects will become apparent from the description of this specification that illustrates aspects of the present disclosure.

Brief Description of the Drawings

[0009] Details of the structure and operation of the present disclosure will become partially apparent by referring to the accompanying drawings. Note that the same reference numerals indicate the same parts. Hereinafter, each figure will be described.

[0010]

Figure 1

[0011]

Figure 2

[0012]

Figure 3A

[0013]

Figure 3B

[0014]

Figure 3C

[0015]

Figure 3D

[0016]

Figure 4A

[0017]

Figure 4B

MODE FOR CARRYING OUT THE INVENTION

[0018] As described above, the human eye may move or tremble in various directions without a specific purpose or reason. Therefore, detailed gaze tracking or eye tracking may generate high-fidelity noise and is generally not adopted as an input for games or interactive programs. However, in certain situations, gaze tracking can be made available in the user interface by combining it with other techniques or systems.

[0019] One embodiment of the present disclosure provides a gaze tracking method and a gaze tracking system for a user interface. In one embodiment, gaze tracking is combined with head movement tracking to achieve fine adjustment. In another embodiment, gaze zone estimation is used to reduce the high-fidelity noise caused by detailed eye tracking. Therefore, in gaze zone estimation, one or more gaze zones can be enlarged or highlighted in one or more overlay images or picture-in-picture (PIP) modes. Further, in another embodiment, interference to the user can be minimized by providing a notification to the user using gaze tracking, for example, during game play or the use of other application programs.

[0020] Reading the following description will clarify how to implement the present disclosure in various embodiments and applications. The various embodiments of the present disclosure are merely illustrative and not limiting. Therefore, the detailed description of each embodiment should not be construed as limiting the scope or application scope of the present disclosure.

[0021] FIG. 1 is a flowchart of a process 100 for gaze tracking for a user interface according to an embodiment of the present disclosure. In one embodiment, process 100 realizes gaze tracking for a user interface by combining gaze tracking with other techniques or systems such as head movement tracking and gaze zone estimation. In one embodiment, process 100 is programmed into a hardware processor to perform gaze tracking within a composite system.

[0022] In the embodiment shown in FIG. 1, at step 110, it is determined whether head movement tracking is operating. If it is determined at step 110 that head movement tracking is operating, then at step 120, gaze tracking is combined with head movement tracking and fine adjustment is performed. For example, head movement tracking provides a rough estimate of the direction in which the user's head is facing, while gaze tracking provides a more detailed estimate of the orientation of the user's head.

[0023] At step 130, it is determined whether gaze zone estimation is operating. In one embodiment, gaze zone estimation divides the user view screen into zones (e.g., rectangular boxes of a predetermined size) at step 140 and uses gaze tracking to estimate that the direction of the user's gaze is within the selected zone. If it is determined at step 130 that gaze zone estimation is operating, then at step 150, gaze tracking is combined with gaze zone estimation and a gaze zone is selected. In one embodiment, one or more gaze zones including the selected zone may be enlarged or highlighted to show a detailed view of an object or person within the user's line of sight in one or more overlay images or picture-in-picture (PIP) modes.

[0024] Next, at step 160, it is determined whether a notification has been received. In one embodiment, the notification includes communicating with the user while the user is playing or experiencing a game or other interactive application program. If it is determined at step 160 that a notification has been received, a notification indicating that communication has been received is displayed on the user view screen at step 170. In one embodiment, the communication includes one or more messages from other users who are playing or experiencing the same game or interactive application program. In another embodiment, the communication includes one or more messages from the game system. In yet another embodiment, the notification is displayed as a bubble icon on the user view screen. In one embodiment, when the user directs their line of sight towards the icon, the details of the message are displayed on the user view screen. By the above-described notification method, it is possible to provide a notification to a user, for example, while playing an interactive game, with minimal interference.

[0025] In a particular embodiment, a method of tracking the user's line of sight within the user view screen is disclosed. This method involves combining line of sight tracking and head movement tracking, where the head movement in head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in line of sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen, and dividing the user view screen into a plurality of line of sight zones when the line of sight zone estimation is active, and combining line of sight tracking and line of sight zone estimation to select one line of sight zone as the direction of the user's line of sight from among the plurality of line of sight zones.

[0026] In one embodiment, the method further includes displaying an icon on the user view screen when the eye tracking is active. In one embodiment, each of the plurality of eye zones is a rectangular box of a predetermined size. In one embodiment, one or more of the eye zones surrounding the selected eye zone are enlarged or highlighted in an overlay image or picture-in-picture (PIP) mode to show a detailed view of an object or person within the user's line of sight. In one embodiment, the method further includes displaying a notification icon on the user view screen when a notification is received. In one embodiment, the notification includes a message from the game system. In one embodiment, the notification includes a message from another user within the game system. In one embodiment, the method further includes displaying a message on the user view screen when it is determined that the user's line of sight has moved to the notification icon.

[0027] Figure 2 is a block diagram of an eye tracking system 200 for a user interface according to an embodiment of the present disclosure. In the embodiment shown in Figure 3, the system 200 includes an eye determination unit 220, a processor 230, and a user interface 240 connected to the processor 230 and the game system 210. In one embodiment, the blocks 220, 230, 240 of the system 200 are configured by hardware including one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. In one embodiment, the system 200 uses eye tracking in combination with other methods, which include at least one or more of head movement tracking, eye zone estimation, and message notification.

[0028] In the embodiment shown in FIG. 2, the gaze determination unit 220 determines whether the head movement tracking is operating. When it is determined that the head movement tracking is operating, the gaze determination unit 220 instructs the processor 230 to combine gaze tracking and head movement tracking to provide fine adjustment. For example, the head movement tracking roughly estimates the direction in which the user's head is facing, and the gaze tracking provides a more detailed estimate of the orientation of the user's head. In one embodiment, the head movement tracking is performed by one or more acceleration sensors and / or one or more gyroscopes disposed within the headset. In another embodiment, the gaze tracking is performed by one or more cameras disposed within the headset.

[0029] Next, the gaze determination unit 220 determines whether the gaze zone estimation is operating. In one embodiment, the gaze zone estimation divides the user view screen into zones (e.g., rectangular boxes of a predetermined size) and uses gaze tracking to estimate that the direction of the user's gaze is within the selected zone. When it is determined that the gaze zone estimation is operating, the gaze determination unit 220 instructs the processor 230 to combine gaze tracking and gaze zone estimation to select a gaze zone. In one embodiment, one or more gaze zones including the selected zone may be enlarged or highlighted to show a detailed display of an object or person within the user's line of sight in one or more overlay images or picture-in-picture (PIP) modes.

[0030] Furthermore, the line-of-sight determination unit 220 determines whether a notification has been received. In one embodiment, the notification includes communicating to the user while the user is playing or experiencing a game or an interactive application program. If it is determined that a notification has been received, the line-of-sight determination unit 220 instructs the processor 230 to display a notification indicating that communication has been received on the user view screen. In one embodiment, the communication includes a message from another user who is playing or experiencing the same game or application program. In another embodiment, the communication includes a message from the game system. In yet another embodiment, the notification is displayed as a bubble icon on the user view screen. In one embodiment, when the user directs their line of sight towards the icon, the details of the message are displayed on the user view screen. Thereby, the line-of-sight determination unit 220 and the processor 230 provide line-of-sight tracking for the user interface 240. By the above-described notification method, for example, a user playing an interactive game can be provided with a notification with minimal interference.

[0031] In one embodiment, the game system 210 includes a game console that provides an interactive game or other application program. In another embodiment, the game system 210 includes an online interactive game or other application program. In yet another embodiment, the game system 210 includes a virtual reality and / or augmented reality interactive game or application program.

[0032] In one embodiment, the line-of-sight tracking function is utilized by one or more game developers and one or more central directors for customizing games and experiences. For example, in a shooting game, the user's line of sight may be used to switch weapons or weapon functions in real time.

[0033] Figure 3A is an example of the above-described user view screen 300 according to an embodiment of the present disclosure. The user view screen 300 of Figure 3A includes two icons 310, 320. The icon 310 may be displayed at the start of an interactive game, for example, to indicate that the eye-tracking mode is active. The icon 320 is displayed on the screen 300 when a notification message is received.

[0034] As shown in the example of Figure 3A, the gaze icon 310 is displayed on the view screen 300 to indicate to the user that the eye-tracking mode is active. In one embodiment, the gaze icon 310 is an icon in the shape of an eye. In another embodiment, the gaze icon 310 is a cursor.

[0035] In one embodiment, the user may receive a message while playing a game. Therefore, when a message is received, the notification icon 320 may be displayed in a predetermined area on the view screen 300. The message that triggers the notification icon 320 may be related to the currently-played game (for example, a notification from the game provider), or may be unrelated to the game (for example, a text from a friend). When a message is received and the notification icon is displayed, the gaze icon 310 may move when the user directs their gaze in the direction of the notification icon 320.

[0036] Figure 3B shows the state where the gaze icon 310 is moved towards the notification icon 320 when the user moves their gaze 312 towards the notification icon 320.

[0037] Figure 3C shows the state where the gaze icon 310 moves into the decision area 322 around the notification icon 320. In this way, when the gaze icon 310 moves into the decision area 322, additional information regarding the message may be displayed.

[0038] FIG. 3D shows a screen 350 including a message panel 330 that replaces the notification icon 320 with the details of the received message 332 when the line-of-sight icon 310 moves into the decision region 322 shown in FIG. 3C. Thus, according to the embodiments from FIG. 3A to FIG. 3D, the user can easily check the content of the received message 332 corresponding to the notification icon 320 without, for example, interrupting the game or operating the game controller. If the user wants to perform additional operations regarding the message 332, the user can perform the operations using the game controller. For example, the user can check the full text of the message 332 by launching a message application or replying to the received message 332. In this example, the game application may be temporarily stopped to perform additional operations.

[0039] In one embodiment, when the line-of-sight icon 310 stays within the decision region 322 for a predetermined time, additional operations related to the message 332 may be executed. In another embodiment, after the line-of-sight icon 310 moves into the decision region 322, additional operations may be executed by user input such as a game controller or voice command. In one embodiment, the decision region 322 is the same size or a different size from the line-of-sight icon 310. In another embodiment, when the line-of-sight icon 310 moves into the decision region 322, the size of the notification icon 320 changes. In yet another embodiment, the notification icon 320 is highlighted so that the user can easily grasp the state of the received message 332.

[0040] In one embodiment, when the notification icon 320 is selected (e.g., by the gaze icon 310 moving to the decision area 322), a message 332 is displayed within the message panel 330, enabling the user to view the message. The size of the message panel 330 may be larger than the notification icon 320 in order to correspond to the size of the received message 332. In one embodiment, the notification icon 320 and additional information (e.g., the message panel 330 and the message 332) disappear from the view screen 300 after a predetermined time has elapsed since being displayed on the screen. Accordingly, according to the above-described embodiment, the user can view messages by means of a gaze operation without interfering with the operation of the game screen or the game controller.

[0041] In a particular embodiment, a game device is disclosed. This game device includes a sensor that tracks the movement of the user's gaze, and a processor operable to execute an application program that obtains information regarding the movement of the gaze from the sensor and displays a gaze icon and a notification icon within the screen image of the application program, the notification icon being displayed at a predetermined position on the screen image.

[0042] In one embodiment, the application program is a game application. In one embodiment, the notification icon receives a display notification from an application program other than the game application. In one embodiment, the game device is a head-mounted display (HMD).

[0043] In another specific embodiment, a gaze tracking system for tracking a user's gaze within a user view screen is disclosed. This system includes a processor that combines gaze tracking and head movement tracking, where the head movement in head movement tracking provides a rough estimate of the direction of the user's gaze, and the eye movement in gaze tracking provides a fine adjustment of the direction of the user's gaze within the user view screen, and a gaze determination unit that, when it determines that gaze zone estimation is active, divides the user view screen into a plurality of gaze zones for the processor, and combines gaze tracking and gaze zone estimation to instruct the processor to select one gaze zone as the direction of the user's gaze from among the plurality of gaze zones.

[0044] In one embodiment, head movement tracking is performed by at least one acceleration sensor and at least one gyroscope disposed within a headset that communicates with a game system. In one embodiment, gaze tracking is performed by at least one camera disposed within the headset. In one embodiment, the system further includes a user interface connected to the processor, and the user interface displays an icon on the user view screen when gaze tracking is active. In one embodiment, a notification icon is displayed when a notification is received. In one embodiment, the notification includes a message from the game system. In one embodiment, the notification includes a message from another user within the game system. In one embodiment, the game system includes a game console that provides an interactive game.

[0045] FIG. 4A is a schematic diagram of a computer system 400 and a user 402 according to an embodiment of the present disclosure. The user 402 uses the computer system 400 to execute a gaze tracking application 490 for gaze tracking for a user interface related to the process 100 of FIG. 1 and the system 200 of FIG. 2.

[0046] The computer system 400 stores and executes the gaze tracking application 490 of FIG. 4B. Further, the computer system 400 may communicate with a software program 404. The software program 404 may include software code for the gaze tracking application 490. The software program 404 may be loaded onto an external medium such as a CD, DVD, or storage drive, which will be described later.

[0047] Furthermore, the computer system 400 may be connected to a network 480. The network 480 can be connected in various architectures, such as, for example, a client-server architecture, a peer-to-peer network architecture, or other various architectures. For example, the network 480 can communicate with a server 485 that adjusts the engines and data used within the gaze tracking application 490. Also, the network may be various types of networks. For example, the network 480 may be the Internet, a local area network and its derivatives, a wide area network, a metropolitan area network, an intranet or extranet, or a wireless network.

[0048] FIG. 4B is a functional block diagram illustrating a computer system 400 equipped with a gaze tracking application 490 according to an embodiment of the present disclosure. The controller 410 is a programmable processor that controls the operation of the computer system 400 and its components. The controller 410 loads instructions (e.g., in the form of a computer program) from the memory 420 or an embedded controller memory (not shown) and executes these instructions to perform system control such as providing data processing. By these executions, the controller 410 provides the gaze tracking application 490 to the software system. Alternatively, this service may be implemented as an individual hardware component within the controller 410 or the computer system 400.

[0049] Memory 420 temporarily stores data for use by other components of computer system 400. In one embodiment, memory 420 is implemented as RAM. In one embodiment, memory 420 includes memory for long-term or permanent storage such as flash memory and / or ROM.

[0050] Storage 430 stores data for use by other components of computer system 400, either temporarily or over a long period of time. For example, storage 430 stores data used in the gaze tracking application 490. In one embodiment, storage 430 is a hard disk drive.

[0051] Media device 440 receives removable media and reads and / or writes data to the inserted media. In one embodiment, for example, media device 440 is an optical disk drive.

[0052] User interface 450 includes components that receive input from a user of computer system 400 and present information to user 402. In one embodiment, user interface 450 includes a keyboard, a mouse, audio speakers, and a display. In another embodiment, user interface 450 includes a headset worn by the user, and this headset collects eye movements as user input. Controller 410 uses input from user 402 to adjust the operation of computer system 400.

[0053] I / O interface 460 includes one or more I / O ports for connecting to corresponding I / O devices such as external storage and auxiliary devices (e.g., printers, PDA). In one embodiment, I / O interface 460 includes ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In other embodiments, I / O interface 460 includes a wireless interface for wireless communication with external devices.

[0054] The network interface 470 includes wired and / or wireless network connections, for example, an RJ-45 or Wi-Fi interface (including but not limited to 802.11, etc.) that supports an Ethernet connection.

[0055] The computer system 400 includes additional hardware and software (e.g., power supply, cooling, operating system) commonly included in a computer system, but these elements are not specifically shown in FIG. 4B for simplicity. In other embodiments, the computer system can use different configurations (e.g., different bus and storage configurations, or multiprocessor configurations).

[0056] In a particular embodiment, a non-transitory computer-readable storage medium is disclosed that records a computer program for tracking a user's line of sight within a user view screen. The computer program causes the computer to combine line of sight tracking and head movement tracking, where the head movement in head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in line of sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen, and when the line of sight zone estimation is active, divides the user view screen into a plurality of line of sight zones, and combines line of sight tracking and line of sight zone estimation to select one line of sight zone as the direction of the user's line of sight from among the plurality of line of sight zones, and includes executable instructions.

[0057] In one embodiment, the computer program further includes executable instructions that cause the computer to display a notification icon on the user view screen when a notification is received, and to display a notification message on the user view screen when it is determined that the user's line of sight has moved to the notification icon.

[0058] The description of each embodiment is provided to enable those skilled in the art to make or use the present disclosure. Many modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein are applicable to other embodiments without departing from the spirit or scope of the present disclosure. Accordingly, the present disclosure is not limited to each embodiment, but should be accorded the broadest scope consistent with the fundamental and novel features of the present disclosure. Therefore, additional variations and embodiments are possible.

[0059] Not all features of each of the above examples are necessarily required in a particular embodiment of the present disclosure. Further, the descriptions and drawings herein are to be understood as representative of the broad subject matter contemplated by the present disclosure. Also, the scope of the present disclosure fully encompasses other embodiments that may be apparent to those skilled in the art, and it is to be understood that the scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for tracking a user's line of sight within a user view screen, comprising: Combining line of sight tracking and head movement tracking, wherein the head movement in the head movement tracking provides a rough estimation of the direction of the user's line of sight, and the eye movement in the line of sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen; Dividing the user view screen into a plurality of line of sight zones when the line of sight zone estimation is active; Combining the line of sight tracking and the line of sight zone estimation to select one line of sight zone from the plurality of line of sight zones as the direction of the user's line of sight; A line of sight tracking method comprising the above.

2. Further comprising displaying an icon on the user view screen when the line of sight tracking is active; The line of sight tracking method according to Claim 1.

3. Each of the plurality of line of sight zones is a rectangular box of a predetermined size; The line of sight tracking method according to Claim 1.

4. One or more line of sight zones around the selected line of sight zone are enlarged or highlighted in an overlay image or a picture-in-picture mode to show a detailed display of an object or a person within the user's line of sight; The line of sight tracking method according to Claim 1.

5. Further comprising displaying a notification icon on the user view screen when a notification is received; The line of sight tracking method according to Claim 1.

6. The notification includes a message from a game system; The line of sight tracking method according to Claim 5.

7. The notification includes a message from another user within the game system; The line of sight tracking method according to Claim 6.

8. Further comprising displaying the message on the user view screen when it is determined that the user's line of sight has moved to the notification icon; The line of sight tracking method according to Claim 7.

9. A sensor for tracking the movement of a user's line of sight; A processor operable to obtain information regarding the movement of the line of sight from the sensor and to execute an application program for displaying a line of sight icon and a notification icon based on the line of sight information within a screen image of the application program; The notification icon is displayed at a predetermined position on the screen image; A game device.

10. The application program is a game application. The game device according to claim 9.

11. The notification icon receives a display notification from an application program other than the game application. The game device according to claim 10.

12. The game device is a head-mounted display (HMD). The game device according to claim 9.

13. A system for tracking a user's line of sight within a user view screen, A processor that combines line-of-sight tracking and head movement tracking, wherein the head movement in the head movement tracking provides a rough estimate of the direction of the user's line of sight, and the eye movement in the line-of-sight tracking provides a fine adjustment of the direction of the user's line of sight within the user view screen, a processor, When it is determined that the line-of-sight zone estimation is operating, the processor is instructed to divide the user view screen into a plurality of line-of-sight zones, and to combine the line-of-sight tracking and the line-of-sight zone estimation to select one line-of-sight zone as the direction of the user's line of sight from among the plurality of line-of-sight zones, a line-of-sight determination unit, A system comprising the same.

14. The system according to claim 13, wherein the head movement tracking is performed by at least one acceleration sensor and at least one gyroscope disposed within a headset that communicates with a game system.

15. The line-of-sight tracking is performed by at least one camera disposed within the headset. The system according to claim 14.

16. The system further comprises a user interface connected to the processor, and the user interface displays an icon on the user view screen when the line-of-sight tracking is operating. The system according to claim 13.

17. The user interface displays a notification icon on the user view screen when a notification is received. The system according to claim 16.

18. The notification includes a message from a game system. The system according to claim 17.

19. The notification includes a message from another user within the game system. The system according to claim 17.

20. The game system includes a game console that provides an interactive game. The system according to claim 19.

Images