Device and method for input control

Abstract

Device for input control of a computing device with a display element (6), comprising at least one control camera (5), at least one further camera which serves as an object detection unit (4) for detecting the elements on the display element (6), and at least one data connection from the at least one control camera (5) and the at least one object detection unit (4) to at least one computing unit (9) and / or the computing device, and at least one marking element (1, 2), designed to be placed in the area of ​​the face of the user of the computing device in question.

Description

[0001] The invention relates to a device and a method for input control of computing units using the user's eyes.

[0002] Up to now, the control of inputs, such as menu navigation, into a computing device (e.g. desktop, laptop, tablet or smartphone) is usually carried out by a user using the input elements keyboard, mouse and touchscreen.

[0003] A disadvantage of the methods used so far is that a human hand must always operate these input elements. This means that people with limited hand function, for example due to medical reasons or because their work requires them to use their hand differently, can only operate the computer to a limited extent. Furthermore, hygiene problems can arise if these input elements are used by several different people.

[0004] From US Patent 2009 / 0196460A1, a system and method for operating a computer by means of eye tracking are known. The system tracks the user's gaze direction and comprises a system that determines the position and orientation of the head, a system for recording the user's eye movements, a display device that presents the user with a calibration image, and a processor that calculates the position of the eye's axis of rotation and the position of the user's gaze direction based on the position and orientation of the head, the position and orientation of the eye's center point, and the position of several calibration points present on the calibration image. The system does not require the placement of any marking elements on the user's face.

[0005] The publication (MARDANBEGI, D.; HANSEN, DW: Mobile gaze-based screen interaction in 3D environments. In: Proceedings of the 1st conference on novel gaze-controlled applications. 2011. pp. 1-4. DOI: 10.1145 / 1983302.1983304) concerns gaze-controlled screen interaction in 3D environments using a head-worn device for recording eye movements (an "eye tracker"). Such devices have at least one camera for capturing eye movements and another camera for recording image scenes. In contrast, an external eye tracker has only a single camera, which records the user's eye.

[0006] The aim of the present invention was to overcome the disadvantages of the prior art and to provide a device and a method that enable input into a computing device without touching input elements with the hand.

[0007] This problem is solved by a device and a method according to the claims.

[0008] The device according to the invention for input control of a computing device with a display element comprises at least one control camera, at least one further camera which serves as an object detection unit for capturing the elements on the display element, as well as at least one data connection from the at least one control camera and the at least one object detection unit to at least one computing unit and / or the computing device, and at least one marking element designed to be placed in the area of ​​the user's face of the computing device in question.

[0009] The computing devices mentioned are typically computers, such as desktops, netbooks, notebooks, or tablet computers. However, smartphones, game consoles, or smart televisions can also be controlled with the device or method according to the invention. The display elements mentioned are generally the screens of the respective computing devices.

[0010] Suitable computing units are known to those skilled in the art. These can be processors or controllers, which are in particular equipped with their own working memory. It is preferred, especially if the control camera is part of the computing device, to use the computing unit(s) of the computing device itself as the computing unit(s). The computing unit is preferably part of the device.

[0011] The relevant data connection to the at least one computing unit and / or the computing device is necessary to enable control of the computing device in the first place. In particular, a data connection to the computing device always exists.

[0012] Suitable data connections are known to those skilled in the art, as is the fact that data from several components can be transmitted via one and the same data connection. Preferred data connections are internal buses of the computing device, USB, LAN, WLAN, other serial or parallel interfaces, or Bluetooth. If the device according to the invention operates completely independently of the computing device, there is not only a data connection to the relevant computing units (which are not the computing device itself), but also an additional data connection to the computing device, through which control signals are preferably transmitted. If the device according to the invention is at least partially integrated into the relevant computing device, image data is also transmitted via at least one of the data connections in the computing device.

[0013] The control camera is a camera that records the user of the computing device or is designed to record this user. Suitable cameras are known to those skilled in the art. External webcams and / or cameras integrated into the computing device are preferred as control cameras.

[0014] In the case that the device according to the invention operates completely without the aid of the computing device, the device according to the invention additionally comprises at least one computing unit operating independently of the computing device in combination with at least one storage unit, which is configured in such a way that it can determine at least the position of the relevant marking elements from the image data of the control camera, in particular also the position of at least one of the user's eyes.

[0015] In the case where the device according to the invention is at least partially integrated into the relevant computing device, the aforementioned tasks are preferably performed by the relevant computing device.

[0016] The object detection units in question serve to detect objects on the respective display units. Suitable object detection units are routines of the computing device that reproduce the position of elements on the display element or at least one additional camera (hereinafter referred to as the "screen element camera").

[0017] This screen element camera is specifically the camera of video glasses or a webcam.

[0018] The marking elements according to the invention are preferably designed such that they stand out from the user's face either due to their shape and / or their color. In this way, it is particularly easy to automatically recognize them from the images of the control camera and to determine their position.

[0019] Preferably, the marking elements in question are arranged at a distance of max. 10 cm from at least one of the eyes, preferably max. 3 cm, in particular between both eyes of the user.

[0020] The inventive method for input control of a computing device with a display element comprises the following steps: a) Recording the face of the user of the computing device with at least one control camera; b) Processing the recorded images in such a way as to determine the position of at least one marker element and the position of at least one of the user's eyes; c) Capture of the elements on the display unit by at least one additional camera; d) Calculation of the user's gaze direction relative to the display element from the aforementioned positions and the detected elements, wherein the position of the at least one eye is preferably the position of the iris, the pupil or another element which can provide information about the gaze direction, such as contact lenses; e) Checking whether a predetermined action of the eye has been performed, preferably a blink, and if this action has been performed, sending the equivalent of a 'mouse click' to the computing device; f) repeating steps a) to f).

[0021] Even though the sequence of steps is partly sequential, the order given above does not necessarily represent the exact chronological order. For example, the elements of the display unit can be processed at any point between the start of a run and the calculations for step d), or the check for the previously defined action can be performed at any point, provided that the user's viewing direction relative to the display element has been calculated at least once.

[0022] In a preferred embodiment, at least one of the object detection units is a camera worn on the user's body, in particular camera glasses, which preferably has at least one of the marking units according to the invention.

[0023] In a further preferred embodiment, at least one of the marking units is attached to the user's eyes by means of contact lenses, which have at least one of these markings on their surface or inside their interior. Fig. Figure 1 shows a sketch of a preferred embodiment. Fig. Figure 2 outlines the functional connections of the individual components of a preferred embodiment.

[0024] In Fig. Figure 1 shows a preferred embodiment of a device for controlling a computer using the user's eyes. The user wears markings 1 and 2 on their face, with markings 1 and 2 positioned near one of the user's eyes 3. The control camera 5, integrated into the computer's screen 6, captures the user's face, enabling the computer to translate the markings 1 and 2 and the user's eye 3 into machine-readable data. Instead of internally detecting the position of display elements, such as the menu bars 7 and 8, the computer uses an external camera 4 to capture the screen 6 and provide this data to the computer.

[0025] Fig.Figure 2 outlines the functional connections of the individual components of a preferred embodiment. The marking elements 1, 2, and at least one of the user's eyes 3 are captured by the control camera 5, which is connected to the computing unit 9 for data transmission. Additionally, the content of the display element is captured by the object detection unit 4 and also transmitted to the computing unit 9. The necessary calculations are performed in the computing unit 9, and the results are used to control the computing device.

Claims

[1] Device for input control for a computing device with a display element (6), comprising at least one control camera (5), at least one further camera which serves as an object detection unit (4) for detecting the elements on the display element (6), and at least one data connection from the at least one control camera (5) and the at least one object detection unit (4) to at least one computing unit (9) and / or the computing device, and at least one marking element (1, 2), designed to be placed in the area of ​​the face of the user of the computing device in question. [2] Device according to claim 1, characterized by , that at least one of the computing units (9) is a computing unit of the computing device. [3] Device according to one of the preceding claims, characterized by, that in the case that the device according to the invention operates completely without the aid of the computing device, not only a data connection to the relevant computing units (9) exists, but also an additional data connection to the computing device, through which control signals are preferably transmitted, and in the case in which the device according to the invention is at least partially integrated into the relevant computing device, at least image data is also transmitted via at least one of the data connections in the computing device. [4] Device according to one of the preceding claims, characterized by , that the at least one control camera (5) is suitable for recording the user of the computing device, and is preferably selected from the group of standalone webcams and cameras integrated into the computing device. [5] Device according to any of the preceding claims, characterized bythat at least one additional camera is the camera of video glasses or a webcam. [6] Device according to one of the preceding claims, characterized by , that the marking elements in question (1, 2) are designed in such a way that they stand out from the user's face either because of their shape and / or because of their coloring. [7] Device according to one of the preceding claims, characterized by , that the relevant marking elements (1, 2) are arranged at a distance to at least one of the eyes (3) of max. 10 cm, preferably max. 3 cm, in particular between both eyes (3) of the user. [8] Device according to any of the preceding claims, characterized by , that at least one of the object detection units (4) is a camera worn on the user's body, in particular camera glasses, which preferably also has at least one of the marking units (1, 2) according to the invention. [9] Device according to any of the preceding claims, characterized by , that at least one of the marking units (1, 2) is attached to the user's eyes (3) by means of contact lenses. [10] Method for input control with a device according to one of the preceding claims, comprising the steps: a) Recording the face of the user of the computing device with at least one control camera (5); b) Processing the recorded images in such a way that the position of at least one marker element (1, 2) and the position of at least one of the user's eyes (3) are determined; c) Detection of the elements on the display unit (6) by at least one additional camera; d) Calculation of the user's viewing direction relative to the display element (6) from the aforementioned positions and the detected elements; e) Checking whether a predefined action of the eye (3) has been performed, and if so, sending the equivalent of a 'mouse click' to the computing device; f) repeating steps a) to f).

Images