Method and device for displaying a virtual hand of an operator of a vehicle control element

Abstract

Method for displaying a virtual hand (16, 18) of an operator of a control element (3) of a vehicle on a display device (4) of the vehicle, wherein a touch of a control surface of the control element (3) with a hand of the operator is detected and a position of a contact point is determined, an orientation of the operator's hand is determined by identifying the operator at least on the basis of seat occupancy sensors and / or on the basis of a closing of an electrical circuit via a seat (7) of the operator by the operator's body via the hand to the control surface, and on the basis of the orientation of the hand thus determined an image of the correspondingly oriented virtual hand (16, 18) is displayed on the display device (4) such that a tip of a finger corresponds to the position of the contact point.

Description

[0001] The invention relates to a method and a device for displaying a virtual hand of an operator of an operating element of a vehicle on a display device of the vehicle.

[0002] Vehicle controls must be located within the driver's reach; in particular, hand-operated controls must be easily accessible to the driver with one hand. On the other hand, displays should be located as close as possible to the driver's line of sight directed toward the road scene ahead, so that the time during which the driver's gaze is not directed toward the road scene in order to perceive the display is as short as possible. Furthermore, displays should not be located too close to the driver, so that the time the driver's eyes need to focus from infinity or from the road scene to the display is also as short as possible.

[0003] These requirements can lead to a conflict, in particular, if the control element is designed as a control surface via which different functions can be operated, for example as a touch-sensitive input surface in the form of a touchpad or a touch-sensitive screen (touchscreen). Such a control surface is usually arranged in the lower area of ​​the dashboard or in a center console so that the driver can easily reach it with one hand. However, to operate such a control surface or a touchscreen, a visual display is necessary in order to recognize the respective operating menu displayed with the associated virtual buttons and to carry out the necessary operating operations, for example to move displayed graphic elements by lightly touching them or to activate buttons by pressing them once or twice.Therefore, a display device, such as a screen, is often provided in a higher and forward-facing position to display menus and monitor operating operations, while the control surface is located in the lower part of the dashboard or in a center console. With such a separate arrangement of the control surface and the display device, it is desirable that the operating actions performed and the display on the display device are linked in the simplest and most intuitive way possible.

[0004] It has therefore been proposed that the hand of an operator of the control element be displayed on the display device in a manner that corresponds to the actual position of the hand relative to the operating surface. DE 10 2005 056 458 A1 discloses using a sensor unit to detect the hand of an operator operating a control element, to determine position data of the detected hand using image recognition, and to transmit this data to an optical display unit for displaying the detected operating hand.

[0005] According to DE 10 2009 057 739 A1, a camera captures one or more of a user's fingers used to operate a control element, and a video image of the user's hand is superimposed on an operating screen. This allows the user to operate the device in a manner similar to directly visually observing the operation of a touchscreen. However, due to the need for a camera and possibly image analysis to capture the hand or fingers in different situations and under different lighting conditions, this involves a relatively high level of effort and corresponding costs.

[0006] From DE 103 49 673 A1 a device and a method for data input in a motor vehicle are known, in which the driver makes written entries with the fingertip of one hand on an input field arranged within reach but not in the driver's field of vision, which are then displayed by text recognition on a display arranged in the driver's field of vision, wherein the current input position on the input field is marked on the display by means of a cursor.

[0007] From US 2009 / 0 309 848 A1 it is known that inputs on a camera-monitored input field within reach, but also not in the normal field of vision of the driver of a motor vehicle, are transferred to a display in the driver's field of vision, whereby an image analysis of the camera image is used to generate a virtual hand representation that can be displayed on the display.

[0008] DE 10 2008 000 192 A1 discloses transferring a hand position and orientation captured by a stereoscopic camera array on an input device in a vehicle to a display in the user's field of vision, whereby the display can display a stereoscopic image. By detecting whether the driver or the passenger is operating the input device, the optimal viewing angle for the stereoscopic image can be adjusted.

[0009] It is an object of the present invention to provide a method and a device for displaying a hand of an operator of an operating surface of a vehicle on a display device of the vehicle, wherein the above-mentioned disadvantages, in particular with regard to the need for a camera and possibly an image analysis, are avoided.

[0010] This object is achieved by a device and by a method as specified in the independent claims.

[0011] In a method according to the invention for displaying a hand of a person operating a vehicle control element with this hand, wherein the control element comprises or is designed as a touch-sensitive control surface, contact of the control surface with the operator's hand is detected and a position of a contact point of the hand with the control surface is determined. The contact is made, for example, with a finger of the hand, in particular with the tip of the finger, but can also be made with the ball of the hand and / or one or more fingers simultaneously. The contact point can be defined, in particular, as the center of gravity of a determined touch surface, which is approximately the size of a fingertip. The contact is detected in a manner known per se, for example by a capacitive and / or resistive sensor system of a touchpad or a touchscreen, as which the control surface can be designed.Capacitive touch detection has the advantage that multiple touch points can be detected, while resistive touch detection can detect the touch force. Therefore, it is advantageous to provide both capacitive and resistive touch detection.

[0012] According to the method according to the invention, the operator is identified, i.e. it is determined whether the person touching the operating surface of the operating element with one hand is the driver, the front passenger or another passenger of the vehicle. From this, it can be deduced as to the position of the hand when touching the operating surface. Furthermore, this usually determines whether the operator touches the operating surface with the right or the left hand. For example, in a left-hand drive vehicle, the driver will touch an operating surface arranged in the center console of the vehicle with the right hand, while the front passenger will use the left hand for this purpose. Similarly, if it is determined that the operator is sitting in the rear right seat, it can be assumed that they are using the left hand to touch the operating surface.By identifying the operator, an orientation of the hand operating the operating element relative to the operating surface is determined, wherein the term “orientation” in the context of the present invention includes both an angular position of the hand relative to the operating surface and the distinction between the right and the left hand under the assumption that the palm of the hand is directed towards the operating surface.

[0013] Based on the orientation of the hand determined in this way, an image of a hand is generated according to the invention and displayed on a display device with the determined orientation. In particular, the image of such a simulated or virtual hand is generated based on parameters that describe the angular position of the hand relative to the operating surface, as well as taking into account the knowledge of whether it is the operator's right or left hand. Furthermore, the image of the operator's hand is shown on the display device in such a way that the tip of a finger of the hand corresponds to the determined position of the contact point. In this case, it can be predetermined which finger of the hand corresponds to the contact point. This is generally the index finger, but can also be the middle finger, for example.

[0014] The display device, which may in particular be a screen or a projection device, is preferably arranged close to the driver's line of sight to the road, so that the driver only needs to briefly avert their gaze from the road ahead to view the display device's display. However, it may also be arranged for the display device to be viewed by the front passenger or other vehicle passengers, or for simultaneous viewing by several people. Furthermore, the display device can advantageously have multiple screens for displaying information for the driver, the front passenger, and / or other passengers.

[0015] By identifying the person touching the operating surface of the operating element with one hand, it is made particularly easy to determine the orientation of the hand used to operate the operating element relative to the operating surface and to display an image of a virtual hand in the determined orientation on the display device. In particular, this enables a sufficiently accurate representation of a simulated or virtual hand on the display device for operating the operating element, without the need for complex image capture and image analysis. By displaying the operator's hand in such a way that a fingertip corresponds to a touch point on the operating surface, a visual image can be created that largely corresponds to the direct observation of the input on the operating surface.For example, an operating menu can be displayed on a screen of the display device, from which menu items can be selected by touching or applying pressure to the displayed touch point. The image shown on the screen of the display device with the simulated or virtual hand of the operator can thus correspond to the visual impression when operating a touchscreen, for example. This enables particularly simple and intuitive operation of the operating element. For example, a transparent or semi-transparent representation of the hand can allow the displayed operating menu to remain visible even "below" the hand.

[0016] The operator can be identified based on a signal from seat occupancy sensors. Such seat occupancy sensors are present in many vehicles, for example to trigger a warning signal if a seat is occupied but the corresponding seatbelt is not fastened. For example, if the front passenger seat is unoccupied and the rear seats in a vehicle are also unoccupied, it can be concluded that it is the driver who is touching the control surface of the control element with one hand. On the other hand, if, for example, only the rear right seat is occupied, it can be assumed that the passenger sitting in this seat is the operator. In this way, the operator can be clearly identified in the majority of operating situations.

[0017] Alternatively or additionally, the electrical conductivity of the human body can be exploited to identify the operator. This exploits the fact that when the control surface is touched, an electrical circuit is closed via the operator's seat, through the operator's body, and then via the hand to the control surface. Whether electrical contact with the control surface is made via the driver's seat, the front passenger seat, or another seat within the vehicle determines whether the driver, the front passenger, or another passenger is touching the control surface. Furthermore, an identification code can be transmitted via the electrical circuit, which varies depending on the seat. This allows the operator to be identified in a particularly secure manner.

[0018] According to a preferred embodiment of the invention, the control surface is arranged in a center console of the vehicle so that it is accessible to both the driver and the front passenger, as well as any other passengers in the vehicle. At least one sensor is arranged to the side of the control surface for detecting a hand or arm of a person touching the control surface from their seat. The at least one sensor is designed in particular as a presence sensor and has an upward-facing detection area in which the presence of a body part of an operator can be detected. If, for example, such sensors are arranged on a left side of the control surface, a right side, and a rear side of the control surface as seen in the direction of travel of the vehicle, it is possible to detect the direction from which an operator is touching the control surface.This also makes it easy to determine whether the driver, the front passenger or another passenger is operating the control element.

[0019] Furthermore, it is preferred that a seating position of the identified operator be detected, which allows conclusions to be drawn, for example, about the operator's height and arm posture, in particular the angular position of the relevant arm and the operator's hand when operating the control element. Furthermore, the height determined from the seating position can be used to infer the size of the operator's hand. This enables a particularly accurate representation of the operator's virtual hand on the display device.

[0020] Preferably, the operator's seating position is detected by detecting the position of at least one adjustment element of the seat of the identified operator. For example, the operator's seating position is determined from the selected position of the seat in the longitudinal direction of the vehicle. This depends in particular on the operator's height. Furthermore, the positions of other adjustment elements of the seat of the identified operator, such as the seat back angle, the seat cushion angle, and / or the seat height, can be evaluated to detect the operator's seating position even more precisely. Adjustment positions of the mirrors and / or the steering wheel can also be evaluated to determine the operator's seating position if the operator is the driver. Appropriate displacement sensors can be provided to detect the positions of the adjustment elements.In vehicles with pre-determined seat adjustments for each individual, the preset seat adjustment parameters can be directly evaluated to determine the operator's seating position. By detecting the seating position using the position of at least one adjustment element, a particularly accurate determination of the operator's seating position is possible in a simple manner.

[0021] To determine the orientation of the operator's hand, a simplified model of the operator is preferably used. This model conveys a relationship between the determined seating position and parameters that indicate the orientation of the operator's hand touching the operating surface of the control element, such as the angular position of the hand relative to the operating surface. Alternatively, with the help of a predefinable and stored characteristic map or, if necessary, a multi-dimensional table (look-up table), the parameters that determine the orientation of the hand can be deduced from the determined seating position of the operator. In particular, the position of the operator's seat in the longitudinal direction of the vehicle allows a conclusion to be drawn about the operator's height, which in turn can be used to calculate an arm angle and an angular position of the hand.This makes it particularly easy to determine parameters that enable the display of a virtual hand of the operator on the display device.

[0022] Preferably, a hand angle and / or hand size of the operator are determined from the seating position of the identified operator. The hand size is determined in particular from the operator's height, which can be determined using the position of the operator's seat in the longitudinal direction of the vehicle. The hand angle is also primarily related to the longitudinal position of the operator's seat, but is also influenced to a lesser extent by other adjustment parameters of the seating position, such as the height and the seat backrest angle. Knowing the hand angle and / or hand size enables a particularly accurate representation of a virtual hand corresponding to the operator's hand on the display device.

[0023] According to a particularly preferred embodiment of the invention, the position of the touch point of the hand on the operating surface is displayed on the display device. The touch point is thus displayed on the display device together with the operator's virtual hand, with the tip of a finger of the hand corresponding to the displayed touch point. The image displayed on the screen of the display device with the simulated or virtual hand of the operator can thus correspond even better, for example, to the visual impression when operating a conventional touchscreen. This further improves the operability of the operating element.

[0024] Furthermore, it is preferred that the position of the point of contact of the hand on the operating surface is determined taking into account the determined orientation of the hand. This makes it possible for such a point of contact to be determined even in the case that the operator touches the operating surface not only at one point, but for example with a plurality of fingers and / or the ball of the hand, in which the operator touches the operating surface with the tip of a pre-determined finger. If, for example, the driver of a left-hand drive vehicle has been identified as the operator, then depending on the determined hand angle it can be assumed, for example, that the tip of the index finger, which is usually used to enter inputs into the operating surface, corresponds to the touch point located furthest in the left-front area if a plurality of touch points are detected.When detecting a flat touch that covers a larger area than a fingertip, the sensor is located in the front left area of ​​the touch surface. This allows for reliable identification of the fingertip position with which the operator typically operates the control element, even when the control surface is touched at more than one point, as can occur, for example, during fleeting operation or during bumpy driving.

[0025] A device according to the invention comprises an operating element that comprises an operating surface or is designed as such, wherein the operating surface is designed in particular as a touchpad or touchscreen and can be arranged in a center console or in a lower region of the dashboard of a vehicle. The device further comprises first sensor means for identifying an operator of the operating element and second sensor means for detecting a seating position of the operator. Such first sensor means can be, for example, seat occupancy sensors, electrical contact sensors for detecting the closure of an electrical circuit via the seat and the operator to the operating surface, and / or presence sensors for detecting an operator's arm reaching towards the operating surface; the presence sensors can be designed, for example, as optical or capacitive sensors.Second sensor means could, in particular, be displacement sensors coupled to a seat adjustment mechanism or sensor means for detecting an input for seat adjustment, which can also be used to set a person-dependent predetermined seat position. Furthermore, the device according to the invention comprises memory and processor means designed to determine the orientation of a hand with which the operator touches the operating surface of the operating element based on the signals from the sensor means and on the basis of a stored model of the operator or a stored table. The memory and processor means can form a control device or be assigned to such a control device. The device according to the invention further comprises a display device, which in particular comprises a screen that can be arranged close to a line of sight of the driver in the vehicle, preferably in the upper region of the dashboard.The display is configured to display an image of the hand in the determined position relative to the operating surface, i.e., in the determined orientation and in a position such that the tip of a predeterminable finger, such as the index finger, of the hand coincides with the contact point. The device according to the invention makes it particularly easy to display a simulated or virtual hand of the operator on the display device in relation to an operating menu in a manner that corresponds to the visual observation of the operator's hand on a touchpad or touchscreen. This improves the operability of the operating element.

[0026] Preferably, the display device has the same aspect ratio as the control surface and, in particular, the same size. This further improves the intuitive usability of the control element.

[0027] The device according to the invention is designed in particular to carry out the method described above.

[0028] The invention is explained in more detail below with reference to the drawings. They show: Fig. 1 shows a cockpit of a passenger car equipped with a device according to the invention and a schematic diagram of such a device; Fig. 2 a simplified flow diagram of a method according to the invention; Fig. 3 a simplified representation of an operator identification; Fig. 4 a simplified flowchart of the operation of an operating menu; and Fig. 5 the representation of a touch point and a simulated hand of an operator on a screen.

[0029] As in the right part of Fig. 1, a plurality of control elements are arranged on a dashboard 1 and on a center console 2 of a motor vehicle, in particular a passenger car. According to one exemplary embodiment, a touchpad 3 is arranged on the center console in such a way that the touch-sensitive surface of the touchpad 3 can be comfortably touched by the driver with the right hand. Furthermore, a screen 4 is arranged in the upper region of the dashboard, which screen can be perceived by the driver with a relatively slight change in his line of sight from the direction of the road ahead. Alternatively, the screen can also be designed as a head-up display - not shown here - i.e. the content is projected onto the windshield in the driver's line of sight.

[0030] The touchpad 3 can also be easily reached by the passenger, using the left hand. Fig. In the arrangement shown in Figure 1, the screen 4 is also visible to the passenger. Furthermore, a further touchpad 6 is located in the area of ​​the steering wheel 5, which can be operated by the driver without taking their hand off the steering wheel.

[0031] The touchpad 3 can be touched with a finger to select and activate certain menu items within an operating menu, for example, to operate the vehicle's radio system. As shown in the left part of Fig. 1, the inputs and actuations of the touchpad 3 initiated by the operator are evaluated by a control device which controls the screen 4 to display a virtual hand corresponding to the operator's hand and an operating menu. The additional touchpad 6 on the steering wheel 5, which can also be used to control some or all of the functions that can be operated with the touchpad 3, has priority over the touchpad 3 in the center console. Independently of the actuation of the touchpad 3 in the center console 2, the driver can thus control the relevant operating functions on the touchpad 6 in the area of ​​the steering wheel 5. In the following description of the method according to the invention, it is assumed that the additional touchpad 6 is not actuated.

[0032] While the touchpad 3 is easily accessible with the driver's right hand, it is not immediately clear to the driver at which position on the touchpad 3 an input is being made. To do so, the driver would have to significantly change their line of sight in order to see the touchpad 3 and the hand used to operate the touchpad 3; moreover, focusing at a relatively short distance would be necessary. To facilitate the operation of the touchpad 3, the operating menu is displayed on the screen 4, along with a simulated hand of the operator in a position and orientation corresponding to that of the hand relative to the touchpad 3, together with a touch point at which an input can be made by applying pressure with the finger.

[0033] For this purpose, as in Fig. 2 schematically shown, according to an embodiment of the invention after detection of a touch of the touchpad 3 (see Fig. 1) first determine which of the possible occupants of the vehicle is operating the touchpad. Such a situation is shown schematically in Fig. 3. According to Fig. 3, an electrical circuit is closed to the touchpad 3 via a seat 7 and the body 8 as well as the arm 9 and the driver's hand (not shown in detail) via the skin, whereby it can be determined whether the driver is touching the touchpad 3. The weak current flow from the seat 7 to the touchpad 3, which is imperceptible to the driver, is in Fig. 3 is indicated by the dashed line. In an analogous manner, a corresponding circuit via the seat 10, the body 11 and the arm 12 of the passenger can determine whether the passenger is touching the touchpad 3. In the Fig. In the situation shown in Figure 3, the passenger is not touching touchpad 3, so the corresponding circuit is not closed. In addition, for particularly secure identification of the operator, a code specific to the operator can be transmitted via the respective circuit.

[0034] If it is now determined which of the vehicle occupants is operating the touchpad 3, the Fig. 2 illustrated embodiment of the method according to the invention, the position of the seat longitudinal adjustment of the seat of the identified operator, who is in the Fig. 3 is the driver. From this, a model is used to determine the probable height of the operator and the probable orientation of the arm with which the operator operates the touchpad; these parameters can also be read from a table, for example. The corresponding calculations are carried out in real time. Furthermore, a predetermined probable operating gesture is used to determine the position of the hand with which the operator operates the touchpad 3 relative to a touch point that has been determined from the touchpad signal. For example, if it has been previously determined that the driver of the vehicle makes the inputs with the index finger of the right hand, the fingertip of the index finger of the right hand can be identified with the detected touch point on the touchpad 3.Together with the hand size, which is estimated from the determined body height, and the previously determined position of the arm with which the operator operates the touchpad 3, all parameters are known to display the operator's hand relative to the touch point on the touchpad in the correct size and position. Such a display of a simulated or virtual hand is shown on screen 4 (see ). Fig. 1), so that the operator can always follow the actions he has performed or is to perform in the operating menu on the touchpad 3.

[0035] The display corresponds to the one the operator would have when operating a touchscreen directly in front of them. Additionally, a transparent or semi-transparent display of the simulated or virtual hand can improve visibility of the underlying operating menu.

[0036] In Fig. Figure 4 schematically illustrates the operation sequence of touchpad 3. When a touch of the touchpad 3 is detected, it is first determined whether the force applied is below or above a threshold. If the force is lower, the touch point is displayed in a "passive" mode, while if the force is higher, it is displayed in an "input" mode. In the latter case, an additional check is performed to determine whether the position of the touch point corresponds to an input field that is active at the time in question. If this is the case, the action corresponding to the input field is performed, similar to the operation of a conventional touchscreen. Thus, a short press with the higher force selects menus or menu items, for example, while a long press calls up submenus, etc.

[0037] A possible display on screen 4 is shown in Fig. 5a to 5f. The figures in the upper row ( Fig. 5a to 5c) the passive mode, the images of the Fig. 5d to 5f the input mode. The representation of a conventional touchscreen, in which the operator has his hand in view, is shown in the left figures ( Fig. 5a and Fig. 5d). The touch point, ie an area in whose center the touch point is located or which represents the touch point, can be displayed differently depending on whether it is the passive or the input mode; in Fig. 5a is the contact point 14 in passive mode and in Fig. 5d shows the touch point 15 in input mode. In passive mode, for example, menu items can be moved by moving the finger, while in input mode, menu items can be activated. The displayed touch point 14, 15 on the screen 4 corresponds to the actual touch point on the touchpad 3. The operating menu is shown in the Fig. 5a to 5f not shown for the sake of clarity.

[0038] Since in the present case the operator, especially if it is the driver, does not have the control surface in his field of vision or does not have it constantly in his field of vision, a virtual hand 16, 18 of the operator is shown together with the menu to be operated on the display device (screen 4, see Fig. 1). Such a representation, if the driver has been identified as the operator, is shown in the Fig. 5b and Fig. 5e. The operator's hand is symbolically represented as a virtual hand 16, 18 based on the parameters determined as described above, with the tip of the index finger displayed at the position of the determined touch point. Fig. 5b, it is therefore not necessary to mark the contact point. In Fig. 5e shows that the touch point 17 is marked in input mode. Fig. 5b and Fig. 5e show the driver's virtual hand 16, ie, in a left-hand drive vehicle, the right hand, with the driver's inputs being made with the index finger. According to Fig. 5c and Fig. 5f, the passenger's hand is represented as a virtual hand 18 if the passenger has been identified as the operator. Together with the virtual hand 16, 18, a part of the operator's arm can also be represented, as in the Fig. 5b, Fig. 5c, Fig. 5e and Fig. 5f. The virtual hand 16, 18 and the operator's arm can be displayed transparently or partially transparently.

Claims

[1] Method for displaying a virtual hand (16, 18) of an operator of an operating element (3) of a vehicle on a display device (4) of the vehicle, wherein a contact of an operating surface of the operating element (3) with a hand of the operator is detected and a position of a touch point is determined, an orientation of the hand of the operator is determined by identifying the operator at least on the basis of seat occupancy sensors and / or by closing an electrical circuit via a seat (7) of the operator through the body of the operator via the hand to the operating surface, and based on the orientation of the hand determined in this way, an image of the correspondingly oriented virtual hand (16, 18) is displayed on the display device (4) in such a way that a tip of a finger corresponds to the position of the touch point. [2] Method according to claim 1, characterized bythat the operating surface is arranged in a center console (2) of the vehicle and the operator is identified by a sensor arranged to the side of the operating surface for detecting the hand and / or an arm (9) of the operator. [3] Method according to one of the preceding claims, characterized by that the operator's sitting position is detected. [4] Method according to the preceding claim, characterized by that the seating position is detected by detecting a position of at least one adjusting element of the operator's seat (7). [5] Method according to claim 3 or 4, characterized by that the orientation of the hand and / or a hand size is determined from the sitting position using a model of the operator and / or a table. [6] Method according to one of the preceding claims, characterized by that the contact point is displayed on the display device (4). [7] Method according to one of the preceding claims, characterized by that the position of the touch point is determined taking into account the determined orientation of the hand. [8] Device for displaying a virtual hand (16, 18) of an operator of an operating element (3) of a vehicle, characterized by that the device is designed to carry out a method according to one of the preceding claims.

Images