Method and system for detecting user inputs

Abstract

The invention relates to a method for detecting user inputs on a graphical user interface (2) displayed on a display device (1). The method comprises detecting a hand (10) of a user by means of a detection device (3) in a detection range (4) of the detection device (3), the detection range (4) being a three-dimensional spatial area, determining a first hand posture (11) at a first time in the detection range (4), determining a second hand posture (12, 13, 14, 15) at a second time in the detection range (4) and identifying a user input on the graphical user interface (2) from a relationship between the first hand posture (11) and the second hand posture (12, 13, 14, 15).

Description

[0001] Method and system for capturing user input

[0002] The present invention relates to a method and a system for detecting user inputs on a user interface, in particular on a display device in a vehicle. In particular, user inputs can be detected contactlessly using gesture control.

[0003] Modern vehicles usually have at least one control unit with a display, such as a screen, which the driver or other passengers can use to control various functions. These can be various vehicle or comfort functions, such as settings for the navigation system, the air conditioning, seat settings, lighting settings, and the like. Various functions of an infotainment system can also be controlled, such as playing music, making phone calls, and the like.

[0004] In vehicles, it is known that certain functions can be controlled contactlessly using gestures. To do this, a user performs certain predefined gestures in a defined spatial area, for example in an area of ​​the vehicle cabin above the center console or in front of the dashboard with a display, which are recorded by a corresponding detection device such as a camera or sensor. However, this requires calibration of the system in global coordinates of the vehicle, which specify the defined spatial area with respect to the display in which the gestures are to be performed, even with each repositioning of the display or the detection device. Also, a user typically does not receive immediate feedback, for example haptic feedback that indicates successful control of the function.

[0005] Other electronic devices with a display, such as computer monitors or smart TVs, can also have corresponding user interfaces through which a user can perform various actions, such as selecting channels, changing settings, selecting streaming services, or the like. For these types of electronic devices, at least one input device is required, such as a computer mouse or a remote control, with which a cursor can be controlled and various actions can be selected by clicking or pressing a button. If necessary, a touchscreen can also be provided, which is operated by appropriate touch. However, contactless input via gesture control can also be advantageous here, as no separate input device is required.

[0006] Augmented reality (AR) input devices are also known. These allow for touchless input, with controls being superimposed onto the real environment without a physical display. However, this often requires a special device, such as AR glasses, that must be worn by the user. Other systems require a special input device that the user must wear on their hand or wrist. Such systems are therefore less practical for individual inputs to an input device, for example, in a vehicle while driving.

[0007] The present invention is based on the object of improving the detection of user inputs on a user interface, particularly in a vehicle. In particular, contactless detection of user inputs is to be improved.

[0008] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0009] A first aspect of the invention relates to a method, in particular a computer-implemented method, for detecting user inputs on a graphical user interface displayed on a display device. In the method, a user's hand is detected by a detection device in a detection region of the detection device, wherein the detection region is a three-dimensional spatial region. A first hand posture is determined at a first time in the detection region and a second hand posture is determined at a second time in the detection region. A user input is then determined on the graphical user interface from a relationship between the first hand posture and the second hand posture.

[0010] The aforementioned method according to the first aspect is therefore based, in particular, on user inputs being captured without contact. In particular, a method is provided that allows, for example, a cursor in a graphical user interface to be controlled with regard to movement and selection, similar to a computer mouse. Gestures can be used that enable a user's contactless interaction with the graphical user interface.

[0011] The key here is that user input is determined based on a relationship between hand positions within the detection zone at two different points in time. By evaluating relative hand movements instead of absolute spatial positions, a method can be provided that allows user inputs to be recorded within a detection zone of a detection device without, for example, requiring knowledge of the spatial position of a display (e.g., in a vehicle's global coordinate system) relative to the detection device. This eliminates the need for calibration, both upon initial use and upon any repositioning of the display and / or detection device.

[0012] This also results in simplified and more convenient operation for the user, as they no longer have to place their hand in a defined area permanently assigned to the display device. They simply need to place their hand anywhere within the detection area (i.e., the field of view) of the detection device, which is usually relatively large and in which their hand may already be located. For example, a user does not have to place their hand directly in front of a display, which might require them to extend their arm or sit up from a reclined position.

[0013] The term "user interface" or "graphical user interface" used here refers in particular to a graphical representation of control elements that are linked to a specific function and allow a user to control the function. The user interface ("UI" or "graphical user interface" - GUI) can contain control elements such as input areas, buttons, symbols, buttons, icons, sliders, toolbars, selection menus and the like, which a user can activate, in particular within the meaning of the present invention, without touching them. The (graphical) user interface can also be referred to as a (graphic) user interface. The GUI can in particular be displayed on a display device such as a display, screen, monitor and the like.The term “detection device” used here refers in particular to a device which can detect objects in three-dimensional space and determine their position without contact. In particular, the detection device can detect a user’s hand. For example, optical methods can be used to detect a user’s hand in space. The detection device can consist of one part or several parts, which can depend on which detection area is to be covered. For example, a (2D) camera or a 3D sensor can be provided. The detection area is the area within which events or changes can be perceived by the detection device, i.e. in the context of this disclosure in particular the area (or more precisely the three-dimensional spatial area) in which a hand can be detected. In the case of cameras or other optical detection devices orSensors can also be referred to as the “field of view”.

[0014] The term "three-dimensional spatial area" used here refers specifically to an area that can be described by three-dimensional coordinates. A position in the three-dimensional spatial area has unique three-dimensional coordinates. A hand position is then recorded specifically in three-dimensional coordinates.

[0015] The term "hand posture" used here refers in particular to the position of the fingers of a hand relative to one another, for example whether individual fingers are bent or stretched. Examples of a hand posture are a flat hand, a clenched fist, an index pose in which only the index finger is stretched, whether individual fingers are touching, in particular the thumb touching one of the other four fingers, and the like. The hand posture can in particular also include the orientation of the hand in space, for example the direction in which the stretched index finger is pointing (also referred to here as the "pointing direction"), as well as other orientations, for example the direction in which the palm of the hand is facing, etc. A hand posture can also be referred to as a "gesture". The method according to the invention can thus also be referred to as "gesture control" of a user interface.

[0016] The term "user input" as used here refers specifically to a user's interaction with the graphical user interface. This can be a simple movement of a pointer (also called a "cursor") on the graphical user interface or controlling a function, such as selecting and activating a control (especially by clicking or double-clicking), navigating through the user interface (e.g., scrolling), or changing the appearance of objects or controls, including moving objects (especially by dragging and dropping).

[0017] The term “vehicle” as used herein refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as sedans, vans, buses, trucks, delivery vans and the like.

[0018] The terms "comprises," "includes," "includes," "has," "has," "with," or any other variation thereof, as used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such a method or apparatus.

[0019] Furthermore, unless explicitly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0020] The terms "a" or "an" as used herein are defined as "one or more." The terms "another" and "another," and any other variations thereof, are defined as "at least one other."

[0021] The term “plurality” or “several” as used herein shall mean “two or more”.

[0022] The term “configured” or “set up” to fulfil a specific function (and respective modifications thereof) is to be understood within the meaning of the invention that the corresponding device is already in a design or setting in which it can carry out the function or is at least adjustable - i.e. configurable - so that it can carry out the function after being set accordingly. The configuration can be carried out, for example, by appropriately setting parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations or operating modes.

[0023] Preferred embodiments of the method are described below, which can be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

[0024] In some embodiments, the detection of a user input is initiated when a user's hand is detected in the detection area and a predetermined initiating hand posture is determined. In other words, the user initiates an input by assuming a predetermined hand posture, so that unintentional operation can be avoided if the hand is in principle detected by the detection device but is in an undetermined hand posture.

[0025] The specified hand posture can, in particular, be an indexing pose, in which the index finger of the detected hand is extended. Here, "extended" does not necessarily mean that a finger is fully extended, but rather that the extension exceeds a predefined value; thus, the finger may still be slightly bent, but is essentially fully extended. An indexing pose is recognized in particular when at least the middle finger, ring finger, and little finger of the hand are bent simultaneously. The thumb can be in any position. Thus, by using an indexing pose, the user indicates that they wish to make an input on the user interface.

[0026] In some embodiments, initiating the user input comprises placing a pointer at a predetermined initial position of the user interface. This position is preferably the center of the user interface, since all areas of the user interface are easily accessible from here, for example, compared to an initial position of the pointer at the edge. In particular, the initial position of the pointer does not depend on the absolute position of the hand in space, but is predetermined and always the same position. In known systems, an initial position can be determined, for example, by orthogonal projection of the index finger onto the user interface, although this requires calibration and, above all, knowledge of the position of the display device in space with respect to the detection device, as already explained.It can be specified that the pointer is not displayed on the user interface until initiation, but only appears (e.g. in the center of the user interface) when a pointing pose is detected.

[0027] In some associated embodiments, the time of initiation is the first time and the predetermined initiating hand posture is the first hand posture, wherein the pointer on the user interface is moved from the initial position to a desired position in accordance with a change from the first hand posture to the second hand posture with the index finger extended, wherein the second hand posture in relation to the first hand posture is or includes a changed pointing direction of the extended index finger. For the operation of the user interface, in this case it is therefore particularly provided that in particular the orientation of the hand in space (or more precisely the pointing direction of the index finger) is relevant, but not necessarily a (translational) movement of the hand in space. If the pointing direction remains unchanged or is parallel, then it can be provided that no movement of the pointer takes place. The user must therefore turn orAt least change the pointing direction. This can simplify operation, however, since in the simplest case, only the index finger needs to be moved. It also reduces the risk of the hand moving too far out of the detection range, potentially interrupting the operation.

[0028] In some related embodiments, the change in hand posture is determined for a plurality of consecutive points in time to move the pointer from the initial position to the desired position. A sufficiently high temporal resolution, determined by a sampling rate of the detection device, is advantageous to provide smooth movement of the pointer on the user interface. This can be, for example, 30 fps (frames per second). The movement can then be compiled as the sum of the movements at successive points in time.

[0029] In some related embodiments, the end of the pointer's movement at the desired position is detected by a predetermined hand position or by the release of the index finger's extension. The pointer's movement thus continues as long as the user continues to move their extended index finger. The movement can then be stopped when the user bends or no longer extends their index finger. However, any other hand position can also be defined as the end of the movement. The movement also stops when the user's hand is no longer detected; in this case, the pointer can also disappear completely from the user interface.

[0030] In some embodiments, a function of the user interface is controlled when, with the index finger extended, the distance between the tip of the thumb and the middle finger of the detected hand falls below a predetermined value. The distance can be defined, for example, as a Euclidean distance. In particular, the thumb (or the tip of the thumb) can touch the middle finger (at any point). Through the touch, the user receives haptic feedback while controlling the user interface, which can greatly simplify operation and make it more intuitive, especially compared to known methods in which the user moves their hand in free space without any feedback.

[0031] In some related embodiments, the distance from the tip of the thumb to the middle finger is defined using a point on the tip of the thumb to a bone vector of the middle finger, which is determined using a point on the tip of the middle finger as well as the articulation points of the distal interphalangeal joint, the proximal interphalangeal joint, and the metacarpophalangeal joint of the middle finger. The metacarpophalangeal joint (also abbreviated "MCP") is the joint connection between the metacarpal bones (ossa metacarpalia) of the metacarpus and the proximal phalanges. The proximal interphalangeal joint (PIP) and distal interphalangeal joint (DIP) are the joints between the phalanges.These points can be determined using the detection device, which can be configured to create a skeletal model of the user, or at least their hand, in order to track the movement of the hand and its hand positions. A correspondingly specified distance between the thumb tip and the bone vector of the middle finger (or a shorter distance) can then indicate contact between the thumb and the middle finger.

[0032] In some embodiments, the function of the user interface is to select a control element of the user interface if a change in the pointing direction when the distance between the thumb and middle finger remains below a predetermined threshold. In other words, the user holds their extended index finger essentially in the same pointing direction while the thumb touches the middle finger. This can be referred to as a "click." It is understood that the "click" function is particularly executed when the user then lifts the thumb from the middle finger again, i.e. the distance increases again. The same applies to a "double-click." Here, the user can "tap" their middle finger twice with their thumb. With both a "click" and a "double-click," the user receives haptic feedback from the touch of their fingers.

[0033] In some further embodiments, the function of the user interface is to move a control element of the user interface if a change in the pointing direction between a time when the distance between thumb and index finger is below a certain threshold and a time when the distance is exceeded exceeds a predetermined threshold. In other words, the user moves their index finger (or changes the pointing direction) while holding their thumb on their middle finger. This can also be referred to as "drag and drop." Here, too, the user receives corresponding haptic feedback, since their thumb touches their middle finger while they are, for example, moving or dragging an object of the user interface.

[0034] In some embodiments, the first and second hand positions are determined in three-dimensional coordinates, which are converted into two-dimensional coordinates of the user interface to control the user input. The three-dimensional coordinates can in particular be spherical coordinates, wherein a radius of the spherical coordinates can be normalized to a uniform value for both hand positions (e.g. r = 1). In other words, only the change in angle, but not the displacement, is used to determine the user input. As already explained above, this means that for operating the user interface, only the orientation of the hand in space (or more precisely the pointing direction of the index finger) is relevant, but not any (translational) movement of the hand in space.

[0035] A second aspect of the invention relates to a data processing system comprising at least one processor configured to execute the method according to the first aspect of the invention. The system comprises at least one display device configured to display the graphical user interface and a detection device configured to detect a user's hand in its detection area and determine a hand posture.

[0036] In some embodiments of the system, the detection device comprises at least one image capture device, in particular a camera. Using a camera, the position of a user's hand in three-dimensional space can be easily determined. One or more cameras can be provided. The camera can be an infrared camera. Advantageously, the at least one camera is a time-of-flight (ToF) camera. By using such a 3D sensor device, the position of the hand in three-dimensional space and its movement can be directly detected. 2D sensors can also be combined to detect the position of the hand in three-dimensional space.

[0037] In some embodiments of the system, the display device comprises at least one display, a monitor, a screen, or the like, which is arranged in a vehicle, for example, as part of an infotainment system. However, a display device may also be provided outside a vehicle, for example, a television, in particular a smart TV, a computer monitor, or other display devices in private or public spaces.

[0038] A third aspect of the invention relates to a computer program comprising instructions which, when executed on a system according to the second aspect, cause the system to carry out the method according to the first aspect.

[0039] The computer program can, in particular, be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program can have a plurality of interacting individual program modules. The system according to the second aspect can accordingly have a program memory in which the computer program is stored.Alternatively, the system can also be configured to access an external computer program, for example on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it which are used during the execution of the method or computer program or which represent outputs of the computer program.

[0040] The features and advantages explained with respect to the first aspect of the invention also apply accordingly to the further aspects of the invention.

[0041] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the drawings.

[0042] It shows:

[0043] Fig. 1 schematically shows a system according to an embodiment of the invention with a display device and a detection device;

[0044] Fig. 2 a hand in an indexing pose with joint points using the example of the index finger;

[0045] Fig. 3 shows an initial pointing pose together with a display device with initial position of a cursor;

[0046] Fig. 4 two hand positions at two points in time with different pointing directions;

[0047] Fig. 5 Examples of hand positions in a “click” state; and

[0048] Fig. 6 a diagram of different hand positions (states) and associated actions on the user interface.

[0049] Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention. Fig. 1 shows a system for the contactless detection of user inputs on a graphical user interface. The system has a display device 1 on which a graphical user interface 2 with control elements 5 and a pointer (cursor) 6 is displayed. The control elements 5 can be manipulated by clicking or drag-and-drop actions. The display device 1 can be any display, for example in a vehicle, or also, for example, a television, in particular a smart TV. For detecting a hand 10 of a user, a detection device 3 with a detection area (field of view) 5 is provided. The detection device 3 can be a (2D) camera or a 3D sensor, such as a ToF camera.A 3D sensor is particularly suitable for detecting the 3D coordinates of the hand 10, in particular of certain key points of the hand 10, such as the joint positions of the thumb, index finger, and middle finger (see Fig. 2). The display device 1 is shown here within the detection area 4. However, it can also be located entirely or partially outside the detection area 4. In particular, a defined spatial relationship between the display device 1 and the detection device 3 is not necessary for the method of the present invention.

[0050] Fig. 2 shows the hand 10 of a user with marked joint points 22, 23, 24 as well as the fingertip 21 (“TIP”) using the index finger as an example. These joint points are equally applicable, in particular, to the middle finger, ring finger, and little finger. The thumb accordingly has one joint point less (no joint point 22). Based on the points 21, 22, 23, 24, a bone vector 20 of the corresponding finger can be created, which simplifies the detection of the finger and its position. The joint points 22, 23, 24 are defined as a point 22 of the distal interphalangeal joint (“DIP”), a point 23 of the proximal interphalangeal joint (“PIP”), and a point 24 of the metacarpophalangeal joint (“MCP”).

[0051] With reference to Figs. 3 to 7, it is explained how user inputs are detected in the system shown in Fig. 1. For this purpose, it is first assumed that the user's hand 10 is located in the detection area 4.

[0052] Fig. 3 illustrates the initiation of a user input. In previously known systems, the position of the cursor could be calculated, for example, as an orthogonal projection of the index fingertip onto the user interface or as the intersection point of the extended pointing direction with the plane of the display device. However, this calculation requires knowledge of the position of the display device in the camera coordinate system. This in turn means that if the screen or camera is repositioned, the respective positions must be recalculated. According to the present invention, by detecting relative movements of the hand 10 in the detection area 4, control of the cursor 6 is made possible without requiring knowledge of the position of the display device 1 in space.

[0053] To prevent a user from inadvertently changing the position of the cursor 6, a previously defined event (e.g., the detection of the index finger or a click / double-click) in combination with a pointing pose is used as a trigger signal for the system. As soon as this event occurs, the cursor 6 is moved to an initial position, for example, the center of the display device 1, as shown in Fig. 3. Since an initial pointing direction is always assumed in subsequent steps, it should be noted at this point that although any event can be defined for initiation, the pointing pose should be present at the end of the event. This is shown as hand position 11 in Fig. 3. The index finger is extended, the position of the thumb is arbitrary, and all other fingers are bent.

[0054] Fig. 4 shows two hand positions 11 , 12 at two different times tn-i and tn (Fig. 4a and Fig. 4b) show how the cursor 6 can be moved on the user interface 2. After initialization (see Fig. 3), the user can move the cursor 6 on the user interface 2 by changing the orientation of their hand in space while maintaining the pointing pose. Therefore, both hand positions 11 and 12 are in different spatial orientations in the pointing pose.

[0055] A possible calculation of the cursor movement is described below. It is assumed that the user's pointing directions do, ..., c / r are detected by the system at different, consecutive times to, ..., fr. Here, do represents the initial pointing direction. Fig. 4 shows two example pointing directions d n -i and d n Time points t n -i and t nFurthermore, it can be helpful if the width w and height h of the display device 1 are known (see below). The pointing directions can be derived, for example, from 3D coordinates of at least two points on the index finger, with a line crossing both points being defined as the pointing direction. If more than two points are available, the pointing direction can be defined as the line that minimizes the sum of the Euclidean distances of all points to this line.

[0056] At all consecutive times t n -i, t n a movement of the cursor 6 is calculated and executed as tn). The total movement of the user can therefore be defined as the sum of all movements: m(t0, t T ) = .n=i m (. t nn> t n)-

[0057] The user's movements, ie the changes in the pointing direction, are now converted into a movement of the cursor 6. This means in particular that the change of two 3D vectors (ie pointing directions) must be transformed into a 2D movement (on the display device 1 or the displayed user interface 2). For this purpose, the direction vectors d n into spherical coordinates. The vectors can be normalized to length 1 (r = 1 ), so that direction vectors p n through two angles 0 n , p n can be represented. It therefore follows that n p n = (1 , 9 n , <p n ).

[0058] Finally, the 2D movement of the cursor can be defined by the angle changes as m(t n -i, t n ) = (Sh * (ß n -i - 0 n ), s, ( <p n -i- <p n)), where the two scaling values ​​Sw and Sh should be selected depending on the width w and the height Zi of the display device 1, respectively, as well as a defined maximum angle. This ensures that a change by a defined maximum angle also reaches the edge. Furthermore, the 2D position of the cursor 6 can be specified as c t The cursor position in two-dimensional x and y coordinates of the display device 1 or the user interface 2 is then referred to as (c t -i x + Sh * (ß n -i - 0 n ), c t -i y Sw * p n -i- <p n )). If the two pointing directions are parallel, the movement of the cursor can be defined as zero, so that c t = c t -i.

[0059] This definition of relative movement ignores the translation of the fingers and only reacts to changes in the pointing direction. It goes without saying that the translation vector can also be mapped into a relative cursor movement. However, ignoring the translation can be helpful in scenarios where a sensor with a limited detection range 4 is only intended to detect rotations of the hand for controlling a device. This minimizes the risk that the user does not leave the detection range 4 with their hand while interacting with the system. The end of the movement is reached either by a previously defined event or when the user's pointing pose is no longer detected. Fig. 5 shows various hand postures 13, 14, 15 with which a user can select ("click") controls or other objects on the user interface 2. A "click" is performed by touching the middle finger with the thumb.Since the index finger is not involved, the previously described requirements for the pointing pose can still be met, with “clicks” being performed during the movement of the cursor using the previously described method.

[0060] Fig. 5a) shows a hand position 13 in which the tip of the thumb touches the tip of the middle finger. In hand position 14 in Fig. 5b), however, the tip of the thumb touches the side of the middle finger. Hand position 15 in Fig. 5c) is considered equivalent to hand positions 13 and 14. Here, the palm faces upward and the thumb also touches the middle finger.

[0061] Suppose the position T ,p the thumb tip is recorded in 3D coordinates by the detection device 5, as well as the joint positions M 1 ' , / V °, l\ / P ip , M mcpof the middle finger (see Fig. 2) in 3D coordinates, which are used to define the bone vector of the middle finger in the world coordinate system. The "clicked" state is defined as the touch of the thumb and middle finger, which means that the Euclidean distance from T ipto the bone vector of the middle finger is smaller than a predetermined threshold value d (hand poses 13, 14, 15). The “released” state is defined conversely when the distance is greater than or equal to d (hand poses 11, 12). This gesture has the advantage of providing and allowing haptic feedback. It can be used to formulate the following main interactions. A “click” is defined as the sequence: “released” —> “clicked” —> “released”, a “double-click” as “released” —> “clicked” —> “released” —> “clicked” —> “released” and a “drag-and-drop” as “released” —> “clicked” (hold) —> “move” (i.e. “change pointing direction”) —> “released”.

[0062] Fig. 6 shows a diagram that summarizes the states and processes described above. The state "no detection" 100 is also defined here if, for example, the hand 10 is not in the detection area 5 or is at least partially obscured, or if none of the hand postures described above could be detected. Upon initiation, the pointing pose, i.e., the "released" state 100, is detected first (hand postures 11, 12). The state "clicked" 102 (hand postures 13, 14, 15) can then occur. This can be a single "click" 200, or a "double-click" 201 in the case of a double sequence 103. If, on the other hand, a movement 104 occurs while holding one of the hand postures 13, 14, 15, this is detected as a "drag and drop" 202.

[0063] The solution presented here assumes that the positions of important finger points are detected, although this is of course only one way to calculate the information necessary for control. In general, it is only necessary to detect the pointing direction and the touch of the thumb and middle finger. This can be achieved in various ways, including machine learning algorithms. In addition, there are several mathematical ways to calculate relative movement of the cursor, of which the presented transformation into spherical coordinates is just one. It enables contactless interaction with a device using gestures, while still providing haptic feedback to the user. A user does not have to wear sensors, which allows interaction with a device without prior preparation.

[0064] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide a guide to implementing at least one exemplary embodiment, with the understanding that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents.

[0065] LIST OF REFERENCE SYMBOLS

[0066] 1 display device

[0067] 2 User interface

[0068] 3 Recording device

[0069] 4 Detection range

[0070] 5 Control

[0071] 6 cursors

[0072] 10 hands

[0073] 11 Hand position

[0074] 12 Hand position

[0075] 13 Hand position

[0076] 14 Hand position

[0077] 15 Hand position

[0078] 20 bone vector

[0079] 21 Fingertip

[0080] 22 articulation point (DIP)

[0081] 23 pivot point (PIP)

[0082] 24 pivot point (MCP)

Claims

CLAIMS 1. A method for capturing user inputs on a graphical user interface (2) displayed on a display device (1), the method comprising: Detecting a hand (10) of a user by means of a detection device (3) in a detection area (4) of the detection device (3), wherein the detection area (4) is a three-dimensional spatial area; Determining a first hand posture (11) at a first time in the detection area (4); Determining a second hand position (12, 13, 14, 15) at a second time in the detection area (4); and Determining a user input on the graphical user interface (2) from a relation between the first hand position (11) and the second hand position (12, 13, 14, 15).

2. The method according to claim 1, wherein the determination of a user input is initiated when the hand (10) of a user is detected in the detection area (4) and a predetermined initiating hand posture (11) is determined.

3. The method according to claim 2, wherein the predetermined hand posture (11) is an index pose in which the index finger of the detected hand is extended.

4. The method according to claim 2 or 3, wherein initiating the user input comprises setting a pointer (6) to a predetermined initial position of the user interface (2).

5. The method according to claim 4, wherein the time of initiation is the first time and the predetermined initiating hand posture is the first hand posture (11), wherein the pointer (6) on the user interface (2) is moved from the initial position to a desired position in accordance with a change from the first hand posture (11) to the second hand posture (12) with the index finger extended, wherein the second hand posture (12) is or includes a changed pointing direction of the extended index finger in relation to the first hand posture (11).

6. The method according to claim 5, wherein the change in hand posture is determined for a plurality of consecutive times to move the pointer (6) from the initial position to the desired position.

7. The method according to claim 5 or 6, wherein an end of the movement of the pointer (6) at the desired position is detected by a predetermined hand position or a release of the extension of the index finger.

8. Method according to one of claims 3 to 7, wherein a function of the user interface (2) is controlled when, with the index finger extended, the distance between the tip of the thumb and the middle finger of the detected hand falls below a predetermined value.

9. The method according to claim 8, wherein the distance of the tip of the thumb to the middle finger is defined by means of a point on the fingertip of the thumb to a bone vector (20) of the middle finger, which is determined by means of a point (21) on the fingertip of the middle finger and the joint points (22, 23, 24) of the end joint, the middle joint and the base joint of the middle finger.

10. The method according to claim 8 or 9, wherein the function of the user interface (2) is to select a control element (5) of the user interface (2) if a change in the pointing direction when the distance between the thumb and middle finger falls below a predetermined threshold value.

11. Method according to one of claims 8 to 10, wherein the function of the user interface (2) is to move a control element (5) of the user interface (2) when a change in the pointing direction between a time when the distance between thumb and index finger falls below a certain threshold and a time when the distance exceeds a certain threshold.

12. Method according to one of the preceding claims, wherein the first and second hand positions (11, 12, 13, 14, 15) are determined in three-dimensional coordinates, which are converted into two-dimensional coordinates of the user interface (2) for controlling the user input.

13. The method according to claim 12, wherein the three-dimensional coordinates are spherical coordinates, wherein a radius of the spherical coordinates is normalized to a uniform value for both hand positions (11, 12, 13, 14, 15).

14. A data processing system comprising at least one processor configured to carry out the method according to one of the preceding claims, as well as at least one display device (1) configured to display the graphical user interface (2), and a detection device (3) configured to detect a user's hand (10) in its detection area (4) and to determine a hand posture (11, 12, 13, 14, 15).

15. A computer program with instructions which, when executed on a system according to claim 14, cause the system to carry out the method according to one of claims 1 to 13.

Images