Method for operating an input device, input device

Abstract

Method for operating an input device (1), in particular of a motor vehicle, which has a sensor (5) for non-contact detection of the position and / or change in position of at least one finger (F) of a hand (H) of a user above an input surface (2), wherein an input is detected and carried out depending on the position and / or change in position of the at least one finger (F), and wherein a distance (A) of the at least one finger (F) to the sensor (5) is determined, characterized in that, in order to detect the position and / or change in position depending on the distance (A), a dimension (B) of the finger is continuously determined, that the input surface (3) is monitored for contact of the finger by at least a second sensor (4), and that the distance (A) of the finger (F) to the sensor (5) is determined depending on the dimension (B) of the finger (F) detected during contact with the input surface (3).

Description

[0001] The invention relates to a method for operating an input device, in particular a motor vehicle, which has a sensor for non-contact detection of the position and / or change of position of at least one finger of a user's hand, wherein an input is detected and carried out depending on the position and / or change of position of the at least one finger, and wherein a distance of the at least one finger to the sensor is determined.

[0002] Furthermore, the invention relates to an input device, in particular for a motor vehicle, with a first sensor for non-contact detection of a position and / or change in position of at least one finger of a user over an input surface of the input device, wherein the input device is configured to recognize and perform an input depending on the position and / or change in position of the finger and to determine a distance of the finger to the sensor. State of the art

[0003] In modern motor vehicles, an input device is often provided as a combined display and input unit, for example, in the form of a touchscreen, so that operation and display of the (virtual) elements to be operated take place in the same location. It is also known to provide the input device separately from a display, for example, to enable ergonomically advantageous use. In particular, this can be achieved, for example, by placing the display unit in the upper area of ​​a control console, especially a dashboard, and the input device in an area of ​​an armrest, for example, near the gearshift, so that the driver does not have to take their eyes off the road too far while comfortably making the input.

[0004] While conventional touchscreens required the user to touch the input surface to input data, systems are now emerging that detect when a user's finger approaches the input surface and then adjust the display accordingly. For example, controls are only displayed when the user's hand approaches the screen. Otherwise, the space is used for other content. Typically, a sensor that operates without physical contact detects the position and / or changes in position of at least one finger of the user's hand. However, currently available systems only register that an approach is occurring.They provide no information about the precise location of the approach, meaning that the screen display can only be changed very generally depending on the finger's proximity, as the expected target position is unknown. However, it can be advantageous to configure different responses depending on the expected target position to further enhance user comfort. In principle, information about the position of a finger merely approaching the input surface could be determined using high-resolution depth sensors or stereo camera systems. However, such solutions are relatively expensive.

[0005] German patent application DE 10 2007 043 905 A1 also discloses a method for calibrating a camera device. This method involves calibrating the camera based on at least one object element identified within a captured image area.

[0006] German patent DE 10 2004 038 965 A1 describes a hand image detection device that uses finger width to recognize extended fingers. It determines finger width based on scan paths perpendicular to finger extension and ascertains whether a finger is protruding from the palm. This method enables precise recognition of hand gestures and reduces the processing effort required for hand image recognition.

[0007] US patent 2010 / 0194713A1 addresses a system for detecting and validating fingertip areas on an input surface. Key features include continuous measurement of finger width and verification against predefined limits to identify actual fingertips. The system thus enables precise interaction during input. Disclosure of the invention

[0008] The method according to the invention, with the features of claim 1, has the advantage that, despite the use of relatively inexpensive sensors, which are not capable of directly determining depth information, the approach of a finger to an input surface and the position of the approach can be determined reliably and quickly. According to the invention, it is provided that, in order to determine the distance of the finger from the sensor, the size of the finger is continuously determined, that the input surface is monitored for contact by the finger by at least a second sensor, and that the distance of the finger to the sensor is determined as a function of the detected size of the finger during contact with the input surface in order to determine the position and / or change in position as a function of the determined distance. Thus, according to the invention, two detection concepts are coupled together.On the one hand, the touch of the finger on the input surface is detected, and on the other hand, the finger's size is continuously determined without physical contact. It can be assumed that the size of a finger remains constant during operation. Depending on the finger's position relative to the non-contact sensor, which also determines the finger's size, the detected or measurable size of the finger changes. For example, if the finger is moved closer to the sensor, it appears to become larger. However, without knowing the actual size of the finger, the input device cannot determine the distance between the finger and the sensor, and therefore the finger's position above the input surface. The sensor can detect the distance of the finger from the input surface, but not its precise location above or on the input surface.According to the invention, by detecting the size of the finger at the moment it touches the input surface, the input device is calibrated using the second sensor. This is because, depending on the size detected during contact, subsequent finger measurements can be assigned to specific distances from the sensor. Preferably, changes in the size of the detected finger are evaluated, for example, using one or more characteristic curves or maps. A change in the perceived size is thus used to infer a change in the distance to the first sensor. Preferably, the second sensor detects the position of the finger on the input surface, so that the finger's position at the time of contact is known. The size of the finger or hand detected during contact is then assigned to this detected position on the input surface.Based on this, by detecting changes in the size of the finger or hand, it is not only possible to infer relative changes in distance, but also to determine the absolute position of the finger on or above the input surface using the first sensor as a function of these size changes. The position of the finger above the input surface here does not refer to the distance to the surface, but rather to the position of the finger relative to the input surface as seen from above.

[0009] According to an advantageous embodiment of the invention, the measurement taken is the width of the finger. The finger width is well-suited as the measurement because the shape of the finger can be easily determined using contour recognition methods, and thus the width can also be easily ascertained.

[0010] Furthermore, it is preferably provided that background modeling is performed to capture the user's finger or hand. Preferably, a static reference image is generated, which is subtracted from the current image at each point in an image sequence, so that only the pixels in the resulting difference image that differ from the previously captured or defined background image are marked. Preferably, a binary foreground mask is calculated by subsequent thresholding. Based on the foreground mask, the hand contour is determined, and optionally, the center of gravity of the hand contour is calculated.

[0011] Furthermore, it is preferably provided that at least one fingertip is determined from the hand contour depending on a curvature of the hand contour. Particularly preferably, fingertip detection is carried out by comparing the curvature of the contour with one or more predefinable threshold values. This allows the fingertip of the finger to be determined in a simple manner, whereby, preferably, closed contours are determined based on the fingertips thus identified, which are marked as a binary fingertip mask. These contours correspond, at least in general terms, to the shape of a fingertip or, for example, a fingerprint, with their centers of gravity preferably representing the positions of the fingertips.

[0012] According to an advantageous embodiment of the invention, the finger width is determined by the width of the measured fingertip contour or the generated closed contour, for example, as the diameter value of the closed contour. Preferably, the finger width is measured as a function of the distance between two fixed points on the contour or via the area of ​​the finger.

[0013] Preferably, the movement path of at least one finger, in particular the fingertip, is detected and stored. At least the movement path is stored for as long as the finger remains within the detection range of the first sensor, so that recalibration of the input device is not necessary during this time. Only when the finger leaves the detection range must the input device be calibrated as described above during the next use, by determining the size of the finger at the moment when the finger, in particular the fingertip, first touches the input surface.

[0014] The input device according to the invention, with the features of claim 8, is characterized in that it has a second sensor which detects contact of the input surface by the finger, and that it continuously detects the size of the finger by means of the first sensor and determines the distance of the finger to the sensor as a function of the detected size of the finger during contact with the input surface. This results in the advantages already mentioned above.

[0015] Preferably, the second sensor also detects the position of the fingertip on the input surface. This is particularly advantageous if the first sensor has a detection direction that does not correspond to the detection direction of the second sensor. If the first sensor is, for example, the aforementioned monocular camera, it is preferably arranged in the previously described method such that its main detection direction or axis is parallel or oblique to the plane of the input surface. If the second sensor now detects the position of the finger touching the input surface, the distance of the fingertip to the first sensor can be easily calculated due to the rigid arrangement between the first and second sensors.

[0016] It is particularly preferred that the second sensor is designed as a touch-sensitive sensor, in particular as a touchscreen or touch-sensitive display.

[0017] Furthermore, it is preferably provided that the first sensor is designed as a camera device, in particular as a monocular camera. The inventive design of the input device eliminates the need for a complex sensor device that also directly provides depth or distance information. Instead, it is advantageous to use a cost-effective monocular camera, which also delivers a smaller amount of data that is easier and faster to process.

[0018] The invention will now be explained in more detail using an exemplary embodiment. For this purpose, we will show... Fig. 1 an input device for a motor vehicle and Fig. 2. A distance measurement of a user's finger to the input device.

[0019] Fig. Figure 1 shows a simplified representation of an input device 1 for inputting control commands, for example, for the entertainment or navigation system of a motor vehicle. The input device 1 has a touch-sensitive screen 2 (touchscreen) which provides an input surface 3 on its screen surface. This surface receives input commands when the user touches the screen surface with one or more fingers. The screen surface is equipped with a sensor 4 that extends across the entire screen surface and thus covers the display of the screen 2. The sensor 4 detects, for example, capacitively, the touch of a finger on the screen and thereby records the touch position on the screen 2. The touch-sensitive screen 2 thus functions like conventional touchscreens.

[0020] Screen 2 is also assigned a sensor 5, which in this case is designed as a monocular camera 6. The monocular camera 6 has a main detection direction defined by an axis 7 in Fig. 1 is indicated. The detection field of camera 6 extends so far, as indicated by the dashed boundary lines 8, that camera 6 also completely captures the touch-sensitive screen 2 or the input surface 3.

[0021] The input device 1 is designed to detect user input by touching the screen 2 with one or more fingers F of the user's hand H, and also to enable input or responses when the finger F merely approaches the screen 2 but does not touch it. The input device 1 is designed such that it is possible to reliably detect the position of the finger relative to the screen 2, even if, during input, the finger F does not touch the screen 2 or the input surface 3 and is at a distance E from the input surface. This is achieved as follows: First, the background of the image captured by camera 6 is modeled to determine the static background and the moving foreground. The background consists, for example, of interior elements of the vehicle containing input device 1. The foreground is the hand H, which is moving as captured by camera 6. Both static and dynamic models can be used for background modeling. A subsequent thresholding process is used to calculate a binary foreground mask.

[0022] Subsequently, a hand contour is calculated based on the foreground mask, and preferably the center of gravity of the hand contour is determined.

[0023] Subsequently, a fingertip FS of finger F is determined based on the curvature of finger F as captured by camera 6, using the so-called "kcurvature" measure along the contour of finger F. Pixels whose "kcurvature" measure exceeds a certain threshold are marked as corresponding candidates in a binary mask. Closed contours, for example in the shape of a typical fingerprint, so-called "blobs," are calculated from this mask, with their centers of gravity representing the positions of the final fingertips FS. If several incorrect fingertips are detected through oversegmentation, they are preferentially discarded based on geometric heuristics or the distance to the center of gravity of the hand contour.

[0024] If one or more fingertips are detected and, if necessary, verified, the width B of the contour is determined for each detected fingertip. This is conveniently done using the distance between two fixed points on the contour or the area of ​​the finger. The detected fingertips FS are tracked for the entire time they are within the detection range of camera 6, and the recorded movement path of the fingertips FS is stored.

[0025] Assuming that the width of the respective finger F does not change, the relative change in distance of the finger to the camera 6 can be derived via a simple angular relationship, as described in Fig. 2 is shown.

[0026] Fig.Figure 2 shows camera 6 and fingertip FS at two different distances A1 and A2 from camera 6 along its principal axis 7. The width B of fingertip FS is, of course, the same in both positions. However, the angles α1 and α2, which are measured between principal axis 7 and a measuring axis 9 tangent to the outer contour of the respective fingertip, differ. The angles α1 and α2 are directly measured by camera 6, at least if camera 6 has already been calibrated. The following applies: tan(α1)⋅A1=tan(α2)⋅A2

[0027] This allows the ratio of the distances to be specified precisely.

[0028] In this case, the input device 1 detects the trajectory or movement path of the fingertip FS as soon as it is within the detection range of the camera 6. As soon as the user touches the input surface 3 with the fingertip FS, the position of the finger F on the input surface 3 is now also known in the formula mentioned above, and, due to the fixed arrangement of sensor 5 to screen 2, the distance of the fingertip FS to the sensor 5 is also known. Knowing the width B, which is detected by the camera 6 when the fingertip FS touches the input surface 3, the distances of the fingertip F in the other positions can then also be reliably determined based on the changes in size or angle.

[0029] The position tracking of the fingertip FS is therefore only precise from the first touch of the screen 2 of the input surface 3 and only as long as the user's finger F is captured in the image of the camera 6. If the finger F is moved out of the camera 6's detection range, the boundary condition of unchanged finger width cannot be guaranteed upon re-entry into the detection range, as the finger F could, for example, belong to a different user.

[0030] Preferably, the displayed interaction areas are selected such that small errors are negligible, or such interaction areas are used in such a way that the user is prompted to touch the touchscreen beforehand, particularly subconsciously. For example, a corresponding prompt to touch the screen / display can be shown. The prompt does not necessarily have to clearly indicate that this is for calibrating the input device 1. At the same time, assumptions can be made regarding the average width B of a finger and the average distance from the screen 2. These assumptions can also be adaptively adjusted to the user over the lifetime of the input device 1.

Claims

[1] Method for operating an input device (1), in particular of a motor vehicle, which has a sensor (5) for non-contact detection of the position and / or change in position of at least one finger (F) of a hand (H) of a user above an input surface (2), wherein an input is detected and carried out depending on the position and / or change in position of the at least one finger (F), and wherein a distance (A) of the at least one finger (F) to the sensor (5) is determined, characterized by , that to detect the position and / or change of position depending on the distance (A) a size (B) of the finger is continuously determined, that the input surface (3) is monitored for touching by the finger by at least a second sensor (4), and that the distance (A) of the finger (F) to the sensor (5) is determined depending on the detected size (B) of the finger (F) during the touching of the input surface (3). [2] Method according to claim 1, characterized by , that the size is measured as a finger width (B) of the finger (F). [3] Method according to any one of the preceding claims, characterized by , that the second sensor detects the position of the finger (F) on the input surface (3). [4] Method according to any one of the preceding claims, characterized by , that background modeling is performed to capture the user's finger (F) or hand (H) and a hand contour is determined based on the background modeling. [5] Method according to claim 1, characterized by , that at least one fingertip (FS) is determined from the hand contour depending on a curvature of the hand contour. [6] Method according to claim 1, characterized by , that the width (B) of the fingertip (FS) is determined as the finger width. [7] Method according to claim 1, characterized by, that the movement path of at least one finger (F), in particular the fingertip (FS) of the finger (F), is recorded and stored. [8] Input device (1), in particular for a motor vehicle, comprising a first sensor (5) for non-contact detection of a position and / or change in position of at least one finger (F) of a user over an input surface (3), wherein the input device (1) is configured to detect and execute an input depending on the position and / or change in position of the finger (F) and to determine a distance (A) of the finger (F) to the sensor (5), characterized by, that the input device (1) has a second sensor (4) which detects a touch of the input surface (3) by the finger (F), and that the input device (1) continuously detects the size of the finger by means of the first sensor (5) and determines the distance of the finger (F) to the sensor (5) as a function of the detected size (B) of the finger (F) during the touch of the input surface (3) in order to detect the position and / or change of position. [9] Input device according to claim 8, characterized by , that the second sensor (4) is designed as a touch-sensitive sensor (4), in particular as a touch-sensitive screen. [10] Input device according to any one of the preceding claims, characterized by , that the first sensor (5) is designed as a camera device, in particular as a monocular camera (6).

Images