Instruction input device and instruction input method using eye movement

Abstract

Instruction input device that uses eye movement, with: a control unit (50) which is intended for the following: Storing instructions, each corresponding to a frequency of a corresponding object; Displaying objects on a display unit (20) to show that each object moves at a corresponding frequency; Capturing a user's eye positions for any given time period; Capturing a frequency based on the user's captured eye positions for each time period; and Identifying an instruction that corresponds to the detected frequency, in relation to the stored instructions.

Description

BACKGROUND OF THE INVENTION Area of ​​the invention

[0001] The present invention relates to an instruction input device and an instruction input method that use an eye movement, and more precisely a technique that recognizes an input instruction based on a frequency detected by a mechanical eye movement of a user. Description of the state of the art

[0002] Recently, various applications utilizing eye-tracking technologies have been developed and implemented. Furthermore, with the development of various devices associated with gaze, eye-tracking schemes are shifting from two-dimensional to three-dimensional tracking of the eye, resulting in the development of various three-dimensional eye-tracking technologies.

[0003] Three-dimensional eye-tracking technologies are also referred to as depth direction (perspective) eye-tracking, because the tracking technologies detect the eye positions on a plane, such as a monitor, and also the distance of an object that a user is looking at, based on the eye, i.e., perspective, in order to represent the eye positions in a three-dimensional coordinate system instead of a two-dimensional coordinate system.

[0004] One method for controlling various devices that use such eye-tracking technologies involves capturing a point (e.g., position) on a screen that a user is looking at and performing an operation according to an instruction set to that point. However, the conventional method for controlling devices is complex to implement and, moreover, cannot guarantee the accuracy of device control, as eye-tracking technologies have been found to be prone to errors.

[0005] From DE 101 21 392 A1 and DE 10 2004 005 816 A1, instruction input devices are known which recognize selected instructions by means of gaze direction detection on displayed objects, and US 6 243 076 B1 discloses an instruction input device in which a gaze direction detection unit derives a movement pattern from the movement of the eyes that track a moving graphic and executes a function stored in this movement pattern. OVERVIEW

[0006] It is an object of the present invention to provide an instruction input device and an instruction input method that use eye movement, wherein an error rate of detecting a point on which a user is looking is reduced by storing frequencies of an object moving on a screen and instructions corresponding to them, comparing the frequency detected by the movement of the eye looking at the object with the frequency of the object and determining as input of an instruction if the compared value is contained within a predetermined range.

[0007] The problem is solved by an instruction input device having the features of claim 1 and an instruction input method having the features of claim 6. Advantageous further developments are found in the dependent claims.

[0008] In one aspect of the present invention, an instruction input device that uses eye movement can include a plurality of units controlled by a control unit. The plurality of units can include: an information storage unit designed to store instructions, each corresponding to a frequency of a corresponding object; a display unit designed to display objects such that each object moves at a corresponding frequency; an eye position detection unit designed to detect a user's eye positions for each time period; and a frequency detection unit designed to detect a frequency based on the user's eye positions detected by the eye position detection unit for each time period.Furthermore, the control unit may be designed to recognize an instruction corresponding to the frequency detected by the frequency detection unit in relation to the instructions stored in the information storage unit.

[0009] In another aspect of the present invention, an instruction input method that uses eye movement may include: storing instructions, each corresponding to a frequency of a corresponding object, by a control unit; displaying objects by the control unit to allow each object to move at a corresponding frequency; detecting the eye positions of a user for each time period by the control unit; detecting a frequency by the control unit based on the user's eye positions for each time period; and recognizing an instruction corresponding to the detected frequency with respect to the stored instructions by the control unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other problems, features and advantages of the present invention will become more apparent from the following detailed description, which was taken in conjunction with the accompanying drawings, in which: Fig. 1 is an exemplary representation illustrating an instruction input device that uses an eye movement according to an exemplary embodiment of the present invention; the Fig. Figures 2A to 2C are exemplary views showing ways in which an object has a certain frequency, according to an exemplary embodiment of the present invention; and Fig. 3 is an exemplary flowchart illustrating an instruction input method using eye movement according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0011] It is clear that the term "vehicle" or "vehicle-" or any other similar term used herein includes motor vehicles in general, such as passenger cars, including off-road vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft, including a variety of boats and ships, aircraft and the like, and hybrid vehicles, electric vehicles, combustion engine vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles and other vehicles using alternative fuels (e.g., fuels derived from raw materials other than petroleum).

[0012] Although an exemplary embodiment is described that uses a plurality of units to carry out the exemplary process, it is clear that the exemplary processes can also be carried out by a single module or a plurality of modules. Furthermore, it is clear that the term "control unit" refers to a hardware device containing memory and a processor. The memory is intended for storing the modules, and the processor is specifically intended for executing the modules to carry out one or more processes, which are described below.

[0013] Furthermore, the control logic of the present invention can be implemented as non-transient, computer-readable media on a computer-readable storage medium containing executable program instructions that are executed by a processor, a control unit, or the like. Examples of computer-readable storage media include, but are not limited to, read-only memory, random-access memory, compact disc read-only memory (CD-ROMs), magnetic tapes, floppy disks, memory sticks, smart cards, and optical data storage devices. The computer-readable storage medium can also be distributed across networked computer systems, allowing the computer-readable medium to be stored and executed in a distributed manner, for example, via a telematics server or a controller area network (CAN).

[0014] The terminology used herein serves only to describe certain embodiments and is not intended to limit the invention. As used herein, the singular forms "a" and "the" are intended to include the plural forms unless the context otherwise makes clear. It will also be clear that the expressions "includes" and / or "include," when used in this description, specify the presence of the aforementioned features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the expression "and / or" includes any or all combinations of one or more of the associated, listed elements.

[0015] Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0016] Fig. Figure 1 is an exemplary illustration demonstrating an instruction input device that uses eye movement according to an exemplary embodiment of the present invention. As shown in Fig. As shown in Figure 1, the instruction input device using eye movement can contain a plurality of units implemented in a control unit 50. The plurality of units can include an information storage unit 10, a display unit 20, an eye position detection unit 30, and a frequency detection unit 40.

[0017] In particular, the information storage unit 10 can be used to store instructions, each corresponding to a frequency of a specific object. Specifically, each object can be correlated with a preset frequency, and each frequency can be correlated with an instruction. The display unit 20 can be configured to show that each object moves at a corresponding frequency. Furthermore, the frequencies can range from 0.05 Hz to 5 Hz, which can be derived from eye movements.

[0018] Generally, when a user (e.g., a driver) looks at a mechanically (e.g., periodically) moving object, the eye moves in sync with the object. Therefore, once the frequency of an object is known, it may be possible to determine which object the user is looking at by detecting eye movement and obtaining the frequency. To ensure that an object moving on a screen has a specific frequency, the present invention uses the following three schemes, each described in the following sections. Fig. 2A to 2C are shown. In the Fig. In 2A-2C, at least one object moves on the screen.

[0019] 1) After the in Fig. In the scheme shown in Figure 2A, an object 210 can appear alternately and periodically on a first page and a second page 220. Consequently, a user's eye can alternately look at the object on the first page 210 and the second page 220 to perceive a frequency.

[0020] 2) After the in Fig. In the scheme shown in Figure 2B, an object can move continuously on a screen in a sinusoidal wave pattern.

[0021] 3) After the in Fig. In the scheme shown in 2C, an object can move continuously on a screen in a triangular wave pattern.

[0022] The eye position detection unit 30 can then be designed to detect a user's eye position for any duration, i.e., any eye movement of the user. Furthermore, the eye position detection unit 30 can be designed to detect the eye position using an "AdaBoost" algorithm.

[0023] In one embodiment, the eye position detection unit 30 can include a face area detector, a similarity calculator, and an eye position calculator. First, the face area detection unit can be configured to receive image data, detect a face area based on the image data, and transmit a face image corresponding to the face area to the similarity calculator. Second, the similarity calculator can be configured to calculate a similarity using the face image transmitted by the face area detector and an eye descriptor. Third, the similarity calculator can be configured to calculate a pixel corresponding to an eye position based on probability. Here, the eye descriptor can be stored in a database.

[0024] Subsequently, the eye position calculator can be designed to calculate a geometric eye position where the user's eye is actually located (e.g., a three-dimensional coordinate of the pupil) using a position containing a pixel that corresponds to the eye position calculated in the similarity calculator. In other words, the eye position calculator can be designed to calculate a geometric eye position using the angle between the two pupils and the distance between them. Furthermore, the eye position calculator can be designed to output calculated eye position data relative to an actual geometric eye position.

[0025] Furthermore, the frequency detection unit 40 can be configured to detect a frequency based on the eye positions of a user detected by the eye position detection unit 30 for any given time period. In other words, the frequency detection unit 40 can be configured to detect a maximum value between 0.05 Hz and 5 Hz, excluding any DC component, of an eye movement using a non-uniformly spaced fast Fourier transform (FFT). The control unit 50 can be configured to actuate the display unit 20 to allow an object moving on the screen to exhibit a specific frequency. Furthermore, the control unit 50 can be configured to execute the frequency detection unit 40 based on the eye positions of a user detected by the eye position detection unit 30 for any given time period.Furthermore, the control unit 50 can be designed to recognize an instruction corresponding to the frequency detected by the frequency detection unit 40, based on instructions stored in the information storage unit 50 according to the respective frequencies. In other words, the control unit 50 can be designed to detect the frequency of an eye movement in order to determine an input instruction corresponding to that frequency.

[0026] The present invention can be applied to devices that require input devices, and the instruction in the present invention can be easily entered by a frequency detection process according to eye movement.

[0027] Fig.Figure 3 is an exemplary flowchart illustrating an eye-movement-based instruction input method according to an embodiment of the present invention. The method may include: storing (301) instructions, each corresponding to a frequency of a corresponding object, by a control unit, and displaying (302) by the control unit that each object is moving at the corresponding frequency. A user can look at the objects on a screen, and one eye can move according to the object's movement.The procedure may further include the detection (303) of the user's eye positions for each time period by the control unit; detection (304) of a frequency by the control unit based on the user's eye positions detected by the eye position detection unit 30 for each time period; and recognition (305) of an instruction corresponding to the detected frequency by the control unit based on the instructions corresponding to the stored respective frequencies.

[0028] As mentioned above, the present invention provides an instruction input device and an instruction input method that use eye movement, wherein the error rate of detecting a point on which a user is looking can be reduced by storing frequencies of an object moving on a screen and instructions corresponding to them, comparing the frequency detected by the movement of the eye looking at the object with the frequency of the object and determining as input of an instruction if the compared value is contained within a predetermined range. SYMBOL OF EACH ELEMENT IN THE FIGURES 10 INFORMATION STORAGE UNIT 20 DISPLAY UNIT 30 EYE POSITION DETECTION UNIT 40 FREQUENCY SURVEILLANCE UNIT 50 CONTROL UNIT 301 STORAGE INSTRUCTIONS, EACH CORRESPONDING TO THE FREQUENCY OF EACH OBJECT 302 INDICATE THAT EACH OBJECT MOVES AT A CORRESPONDING FREQUENCY 303 Capture eye positions for any duration 304 DETECT FREQUENCY BASED ON EYE POSITIONS RECORDED FOR EACH TIME DURATION 305 RECOGNIZE INSTRUCTION CORRESPONDING TO THE DETECTED FREQUENCY, WITH RESPECT TO STORED INSTRUCTIONS

Claims

[1] Instruction input device that uses eye movement, with: a control unit (50) which is intended for the following: Storing instructions, each corresponding to a frequency of a corresponding object; Displaying objects on a display unit (20) to show that each object moves at a corresponding frequency; Capturing a user's eye positions for any given time period; Capturing a frequency based on the user's captured eye positions for each time period; and Identifying an instruction that corresponds to the detected frequency, in relation to the stored instructions. [2] Instruction input device according to claim 1, wherein the control unit (50) is further provided for the following: Activating the display unit (20) to show that an object appears alternately on a first page and a second page of a screen. [3] Instruction input device according to claim 1, wherein the control unit (50) is further provided for the following: Activating the display unit (20) to show that an object is moving in a sine wave pattern on a screen. [4] Instruction input device according to claim 1, wherein the control unit (50) is further provided for the following: Activating the display unit (20) to show that an object is moving on a screen in a triangular wave pattern. [5] Instruction input device according to claim 1, wherein the control unit (50) is further provided for the following: Activating the display unit to show that at least one object on a screen has a corresponding frequency. [6] Instruction input method which uses eye movement, comprising: Storing instructions, each corresponding to a frequency of a corresponding object, by a control unit (50); Displaying objects on a display unit (20) by the control unit (50) to show that each object moves at a corresponding frequency; Detection of a user's eye positions for each time period by the control unit (50); Detect a frequency by the control unit (50) based on the user's eye positions for each time period; and recognize an instruction corresponding to the detected frequency, in relation to the stored instructions by the control unit (50). [7] Instruction input method according to claim 6, wherein the display includes the following: Display of an object by the control unit (50) to show that the object appears alternately on a first page and a second page of a screen. [8] Instruction input method according to claim 6, wherein the display includes the following: Display of an object by the control unit (50) to show that the object moves on a screen in a sine wave pattern. [9] Instruction input method according to claim 6, wherein the display includes the following: Display of an object by the control unit (50) to show that the object moves on a screen in a triangular wave pattern. [10] Instruction input method according to claim 6, wherein the display includes the following: Displaying the objects by the control unit (50) to show that at least one object on a screen has a corresponding frequency. [11] Non-transitory, computer-readable medium containing program instructions that are executed by a processor or control unit (50), wherein the computer-readable medium comprises: Program commands that store instructions, each corresponding to a frequency of a corresponding object; Program commands that display objects on a display unit (20) to show that each object moves at a corresponding frequency; Program commands that capture a user's eye positions for any given time period; Program commands that capture a frequency based on the user's eye positions for each time period; and Program commands that recognize an instruction corresponding to the detected frequency in relation to the stored instructions. [12] Non-transitory, computer-readable medium according to claim 11 further comprising: Program commands that display an object to show that the object appears alternately on a first page and a second page of a screen. [13] Non-transitory, computer-readable medium according to claim 11 further comprising: Program commands that display an object to show that the object moves on a screen in a sine wave pattern. [14] Non-transitory, computer-readable medium according to claim 11 further comprising: Program commands that display an object to show that the object moves on a screen in a triangular wave pattern. [15] Non-transitory, computer-readable medium according to claim 11 further comprising: Program commands that display the objects to show that at least one object on a screen has a corresponding frequency.

Images