Method for operating a control device, control device and motor vehicle

Abstract

Method for operating an operating device (10) with a display device (14), wherein the display device (14) provides a graphical user interface (16) on which several graphical display elements (22) are displayed in one display plane, wherein the several graphical display elements (22) are each coupled with an operating function, characterized by the steps: - Selecting a graphic display element (22) from the multiple graphic display elements (22), wherein, for selection, a position of a hand (24) or a finger (26) of the hand of a user relative to the graphic display element (22) is detected by means of a detection device (20); - Non-contact detection of at least one further hand gesture of the user by means of the detection device (20), wherein the further hand gesture is performed by the hand (24) of the user; - Setting a display parameter of the selected graphic display element (22) depending on the detected further hand gesture of the user by means of a control device (18), wherein the display parameter of the selected graphic display element (22) is set independently of display parameters of the other graphic display elements (22).

Description

[0001] The invention relates to a method for operating an operating device. The invention also includes an operating device operable according to the method of the invention. Finally, the invention also encompasses a motor vehicle with the operating device of the invention.

[0002] It is generally known from the state of the art that a user can position an arrangement of graphic display elements, so-called icons, on a computer display according to their own ideas. To do this, the user can, for example, use a real control element, such as a computer mouse or a joystick.

[0003] Instead of using a dedicated control element, it is also possible for the user, if the display device has a touch-sensitive user interface, to reposition display content manually, for example by touching and moving his finger.

[0004] For example, DE 10 2008 051 756 A1 describes a system for controlling a vehicle information system. The system is operated using four different input modalities. Each of the four input modalities is assigned a user interface of the system. The first input modality is a turn or press, which is detected by a switch arranged on a touchscreen of the system. The second input modality is a gesture, which is detected by a capacitive sensor on the touchscreen. The third input modality is speech recognition, which is detected by a speech recognition module. The fourth input modality is gaze tracking, which is detected by an eye tracking module.Depending on the input modalities detected, a system's presentation manager decides on the type of output and the appropriate presentation method for the display content displayed on the touchscreen. Such a system is complex, and implementing it is accordingly particularly complex.

[0005] AU 2011 201 639 A1 describes a display system with a touch-sensitive user interface. An electronic document displayed on the display system is rotated or enlarged by a gesture from a user of the display system. The rotation or magnification of the document is performed by the gesture on or near the touch-sensitive user interface and is detected by the touch-sensitive user interface. The disadvantage of this solution is that the user must position their hand precisely so that their hand or gesture is detected by the capacitive sensor of the user interface.

[0006] DE 10 2006 028 046 A1 discloses a combined display and operating device for a motor vehicle. The device comprises a touch-sensitive screen for displaying and selecting operating functions. Furthermore, the device comprises a device for detecting an operating means. The screen is configured to switch from an information mode to an operating mode, in which a selection of operating functions is possible, when an operating means is detected in a predetermined area around the screen. Furthermore, the display and operating device comprises a manual operating element by means of which a selection of the operating functions and a change in the information displayed on the screen is possible. Each operating action, which is performed either on the manual operating element or the screen and is detected by the device, is assigned a specific function.A user must know exactly what to operate or where to position the control device to trigger a desired function. This makes the combined display and control device particularly complex to operate.

[0007] EP 2 741 171 A1 describes a controller for a human-machine interface in a vehicle that proposes contactless gesture instructions. The human-machine interface comprises visual display means, human-operable control devices, and a computer system. The human-machine interface comprises human-operated contactless control devices, referred to as instant location detection means, and is capable of recognizing gestures of a body part of a human user without the need to touch a corresponding human-operated control device in order to point / select / actuate control functions displayed on the corresponding visual display means. This contactless gesture control is typically used, for example, in vehicles or aircraft such as rotary-wing aircraft.

[0008] US 2015 / 0022432 A1 discloses a method comprising: receiving image inputs at a sensor of an information processing device; determining a predetermined gesture within the image input using one or more processors; detecting additional gesture inputs associated with the predetermined gesture; and processing an image displayed on a display device according to the additional gesture input associated with the predetermined gesture. Other embodiments are described and claimed.

[0009] US 2014 / 0105453 A1 describes a method for gesture recognition using natural images, comprising generating a series of variant images using two or more consecutive natural images, extracting an image feature from each of the variant images, and comparing the varying pattern of the image feature with a gesture definition to recognize a gesture. The method is inherently robust to blur in images and supports motion estimation in the X, Y, and Z axes without requiring the recognized object to maintain a fixed gesture.

[0010] The object of the present invention is therefore to provide an operating device and a method and a motor vehicle with such an operating device, by means of which a graphic display element can be manipulated in a particularly simple manner.

[0011] The problem is solved by the subject matter of the independent patent claims. Advantageous developments of the invention are disclosed by the features of the dependent patent claims, the following description, and the figures.

[0012] According to the invention, this object is achieved by a method for operating an operating device with a display device (GUI - Graphical User Interface), wherein the display device provides a graphical user interface on which a plurality of graphic display elements are displayed in a representation level, wherein the plurality of graphic display elements are each coupled to an operating function. In other words, each of the plurality of graphic display elements is assigned an operating function which can be triggered or activated by actuating the respective display element. A device can be operated by means of the operating function. The graphic display element thus preferably provides an operating interface for operating the device. Each display element is preferably an icon and / or a list entry in a menu.

[0013] In the method, a graphic display element is first selected from the plurality of graphic display elements. To select the graphic display element, the position of a hand or a finger of a user's hand is detected by a detection device. Subsequently, at least one further hand gesture of the user is detected contactlessly by a detection device, wherein the hand gesture is executed by the user's hand. Finally, a display parameter of the selected graphic display element is set by means of a control device as a function of the detected hand gesture of the user. The display parameter of the selected graphic display element is set independently of the display parameters of the other graphic display elements.

[0014] The method generates or causes an interaction between the graphic display element and a user hand located outside the display device. So that the user can specifically change a display parameter of the graphic display element, the user first selects one of several display elements displayed on the graphical user interface of the display device. "Changing the display parameter" here means that the graphical display element is moved from a reference position or a shape is changed. For example, depending on the hand gesture detected by the detection device, the control device can change a position of the graphical display element on the graphical user interface and / or change a size or extent and / or an orientation of the graphical display element on the graphical user interface.

[0015] To select the graphic display element, the position of the user's hand or finger is detected, either on the user interface of the display device or at a predetermined distance from the user interface relative to the graphic display element the user wishes to select. The hand or finger is positioned above the graphic display element to be selected. "Above" here means that the hand or finger is positioned in overlap with the display element to be selected. To detect the selection, the user interface can have a sensor matrix as part of the detection device. Selection of the corresponding display element can be detected by touching or capacitively detecting the user's hand or finger on the user interface.

[0016] However, when selecting the graphic display element, the user does not activate any operating function associated with the graphic display element. For example, it may be provided that another actuation of the graphic display element, for example, by repeatedly touching or tapping the selected graphic control element, allows the user to activate a function associated with the graphic display element, i.e., the operating function associated with the display element.

[0017] Once the graphic display element has been selected, the display parameters of the selected graphic display element can be changed. In other words, the selection can generate a coupling mode between the user's hand and the selected graphic display element. The graphic display element is therefore now coupled to the user's hand. To set the display parameters, at least one further hand gesture from the user is detected. The at least one further hand gesture is detected by the detection device. The hand gesture is performed by the user by moving their hand. When performing the hand gesture, the hand is free, i.e. contactless, in an area surrounding the display device.If the detection device detects the hand gesture, it transmits a detection signal to the control device, which evaluates the detection signal and then transmits a setting signal to the display device for setting the representation of the selected display element.

[0018] This offers the advantage that the user can interact with the graphical display elements displayed on the display device in a particularly simple manner. This allows the user to customize the display of the graphical display elements according to their preferences.

[0019] The invention includes optional further developments whose features provide additional advantages.

[0020] According to a further development, a rotary movement of the hand is recorded as the hand gesture. This rotates the selected graphic display element in the display plane of the graphic user interface corresponding to a direction of rotation of the rotary movement of the hand. In this case, the display parameter corresponds to a rotational position of the graphic display element. The control device thereby implements a rotation of the selected graphic display element. Rotary movement here means a rotational movement of the hand. In order for the recording device to accept the hand gesture, a user can first clench their hand into a fist or curl their hand with their fingers as if holding a wrench. In order for the recording device to register the rotary movement, the user can then turn their hand to the right or left.In other words, the user tilts their hand in a flowing movement at a predetermined angle from a start or initial position. During the rotational movement of the hand, the graphic display element can rotate around its own axis. The graphic display element can be rotated from its reference position until the hand stops or reaches a stop position. For example, the hand can be rotated 90 degrees from the start position. Since the movement of the graphic display element corresponds to the rotational movement of the hand, the graphic display element also rotates 90 degrees around its own axis. Axis here refers to an axis perpendicular to the display plane or user interface. Through the corresponding movement, the user can see exactly how the interaction between the executed hand gesture and the changes in the display of the graphic display element takes place.

[0021] A further embodiment of the invention provides that a combination gesture of the hand consisting of a rotational movement and a simultaneous movement of the hand perpendicular to the user interface is detected as a hand gesture. In other words, a simultaneous rotational and translational, i.e., linear, movement of the hand is performed as the combination gesture of the hand. As a result, the selected graphic display element is displayed enlarged or reduced in size in the display plane of the display corresponding to the combination gesture. In this case, the display parameter corresponds to an extension or size of the graphic display element. The control device implements an enlargement or reduction of the selected graphic display element based on the detected combination gesture.

[0022] A combination gesture is the superimposition of two different hand movements, which are recorded as a single hand gesture - the combination gesture. The first hand movement is preferably the previously described rotating movement of the closed hand. The second hand movement is a movement or a displacement of the hand perpendicular to the user interface of the display device. With the combination gesture, the user first clenches their hand into a fist, as already explained for the rotating movement, or curls their hand and fingers together as if holding a screwdriver. They then move their hand with a simultaneous rotating movement perpendicular to the user interface or away from the user interface. To enlarge the graphic display element, the user preferably moves their hand towards the user interface.To reduce the size of the graphic display element, the user prefers to move their hand away from the user interface. To change the display parameters of the graphic display element, the user performs a screwing motion with their hand. The user moves their hand as if using a screwdriver. This combination gesture implements a zoom effect, i.e., zooming in or out, in a particularly simple way.

[0023] Advantageously, the detection device detects the hand gesture after selection only after a predetermined displacement of the hand from a starting position. In other words, the detection device preferably only detects the hand gesture after a delay. If, for example, the rotational movement of the hand is detected as a hand gesture, an angle or a rotational distance or rotational position can be specified as the predetermined displacement, by which the hand must initially or at least be moved from the starting position or start position in order for the hand gesture to be detected. For example, the detection device can only detect the rotational movement from an angle of 20 degrees. In the case of a combination gesture, it can be provided, for example, that the combination gesture is only detected by the detection device when the user has moved their hand a distance of at least 5 mm perpendicular to the user interface.Particularly preferably, the detection device only detects the combination gesture after a displacement of 10 mm. This prevents a shaking of the hand, caused, for example, by the movement of the motor vehicle, from being mistakenly detected as a hand gesture. Such a delay can prevent incorrect operation of the operating device.

[0024] A further development of the invention provides that, after setting the display parameter of the selected graphic display element, the selected graphic display element is maintained in the set display if the user performs a confirmation action. If the user sets the display of the graphic display element and is satisfied with the set change, they can, for example, actuate a mechanical control element as a confirmation action, which transmits a confirmation signal to the control device. This keeps the selected graphic control element in the changed display. If the confirmation gesture is omitted, the selected graphic display element is automatically returned to a reference position after a predetermined time interval after setting the display parameter.For example, it can be provided that, after setting the display parameter of the selected graphic display element, the selected graphic display element returns to a reference position from which the user changed the display, if the detection device does not detect any confirmation action performed by the user after the predetermined time interval following setting the display. The predetermined time interval can be, for example, between 3 seconds and 10 seconds, in particular 5 seconds. The predetermined time interval can be stored in a timer of the operating device.

[0025] Advantageously, the graphic display element is highlighted when selected. In other words, the graphic display element is marked when selected. If the graphic display element is highlighted, it can, for example, be outlined in color or have a colored background. This provides the user with immediate visual feedback upon making their selection. This allows the user to see whether the desired graphic display element has been selected. Furthermore, the user knows that once the graphic display element has been selected and highlighted, they can manipulate it, i.e., change its display parameters.

[0026] In order to carry out the method according to the invention, the invention also provides an operating device. The operating device comprises a display device configured to provide a graphical user interface on which a plurality of graphical display elements are displayed, wherein the plurality of graphical display elements are each coupled to an operating function. The operating device is configured to detect one of the graphical display elements by detecting a position of a hand or a finger of a user's hand relative to the graphical display element. Furthermore, the operating device comprises a detection device configured to detect a hand gesture of the user. Furthermore, the detection device is configured to detect a hand gesture of the user without contact.Finally, the operating device also comprises a control device which is configured to set a display parameter of the selected graphic display element as a function of the detected hand gesture of the user, independently of display parameters of the other graphic display elements.

[0027] In order to reliably capture the hand gesture in an area surrounding the display device, a further development provides that the capture device captures the hand gesture using a camera. The camera here refers to an image capture device and an associated image processing device, which detects a hand in the image data of the image capture device and captures its position in space. The camera is preferably designed as a time-of-flight camera (TOF camera). This has the advantage that a relative position of the hand in space with respect to the camera can be captured directly based on the 3D image data of the time-of-flight camera.

[0028] Advantageously, the display device and a part of the detection device are designed as a touchscreen. In other words, the display device and a part of the detection device together form the touchscreen.

[0029] The operating device can be provided in the manner described in a portable, mobile terminal, for example, a smartphone or tablet computer. In particular, it is provided that the operating device is implemented in a motor vehicle. Accordingly, the invention also provides a motor vehicle with an embodiment of the operating device according to the invention. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car.

[0030] The advantages and preferred embodiments described for the method according to the invention also apply to the motor vehicle according to the invention and the operating device according to the invention and vice versa.

[0031] An exemplary embodiment of the invention is described below. It shows: Fig. 1 a schematic representation of an embodiment of the operating device according to the invention; Fig. 2 a schematic representation of the operating device of Fig. 1 when a user selects a graphic display element; Fig. 3 a schematic representation of the operating device with a selected graphic display element; Fig. 4 a schematic representation of a user's hand performing a hand gesture; Fig. 5 a schematic representation of the user’s hand when performing a combination gesture; and Fig. 6 a schematic representation of the user's hand when performing the combination gesture.

[0032] The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, the described components of the embodiment each represent individual, independently considered features of the invention, which also further develop the invention independently of one another and are thus also to be considered as components of the invention, either individually or in a combination other than that shown. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

[0033] In the figures, functionally identical elements are provided with the same reference numerals.

[0034] Fig. 1 shows a perspective view of an operating device 10. The operating device 10 is provided, for example, in a motor vehicle 12. Alternatively, the operating device 10 can also be provided in a mobile device, such as a smartphone.

[0035] The operating device 10 comprises a display device 14 with a graphical user interface 16, a control device 18 and a detection device 20. The display device 14 and the detection device 20 are signal-coupled to the control device 18.

[0036] The control device 18 is configured to control the display content on the graphical user interface 16. For example, graphical display elements 22 can be displayed on the graphical user interface 16. The graphical display elements 22 are, for example, an icon or a menu entry. A respective operating function is assigned to the respective graphical display element 22. The operating function can trigger a device function of a device (not shown in the figures) of the motor vehicle 12. In other words, the graphical display elements 22 serve as a user interface for operating a device of the motor vehicle 12. The display content in Fig. 1 can, for example, provide a standard menu layout for individual menu items in the form of graphic display elements 22. Each menu item can, for example, represent one of the operating functions for a corresponding device, such as a radio, a navigation device, or an air conditioning system.

[0037] The operating device 10 allows a user to select the graphic display elements 22 and to change or adjust their display parameters individually and independently of one another. The selection of one of the graphic display elements 22 is Fig. 2 and Fig. 3. When selecting the graphic display element 22, the user does not activate any device function. To recognize or detect the user's selection, the graphic user interface 16 can have a sensor matrix. The sensor matrix can be part of the detection device 20 of the operating device. By touching or capacitively detecting a hand 24 or a finger 26 of the user on the user interface 16, a selection of the corresponding display element 22 is detected. To do so, the user positions, as shown in Fig. 2, his hand 24, in particular a finger 26 of the hand, relative to the graphic display element 22 to be selected. If the user's selection is detected, the selected graphic display element 22, as shown in Fig. 3. The selected graphic display element 22 is highlighted, for example, in color.

[0038] After selecting one of the displayed graphic display elements 22, the operating device 10 enables the user to change or adjust the display parameters of the selected graphic display element 22. For this purpose, the operating device 10 uses the detection device 20. The detection device 20 can comprise, for example, a camera, in particular a TOF camera. The camera has a detection area 28. When a hand gesture is detected, the detection device 20 transmits a detection signal E to the control device 18. This, in turn, controls the display device 14 by transmitting a control signal S to the display device 14.

[0039] The user can change the display of the selected graphic display element 22 by performing a hand gesture in the detection area 28. For this purpose, the user's hand 24 is freely positioned in the vicinity of the display device 14 in the detection area 28. The display of the selected graphic display element 22 then changes in accordance with the hand movement.

[0040] In the Fig. 4 to 6 show different hand gestures of the user that can be interpreted by the operating device 10.

[0041] In Fig. Figure 4 shows how the user can rotate the selected graphic display element 22 around its own axis. When rotating the graphic display element 22, the graphic display element 22 is displaced from a reference position 30. To rotate the graphic display element 22, the user performs a rotational movement with their hand 24. The rotational movement of the hand 24 and the corresponding rotation of the graphic display element 22 are visualized by the respective arrow. Depending on the direction in which the user rotates or tilts their hand 24, the graphic display element 22 moves in the corresponding direction. Fig. 4, the user turns his hand 24 to the left, i.e., counterclockwise. The graphic display element 22 is also rotated to the left when the rotational movement is detected from a reference position 30.

[0042] When performing the hand gesture to rotate the graphic display element 22, the user first clenches their hand 24 into a fist or curls their fingers as if holding a wrench. They then rotate their hand 24 in a desired direction. Once they have reached the desired setting for the graphic display element 22 and want the graphic display element 22 to be held in the set position, the user can use the hand with which they are not performing a hand gesture to actuate an operating element (not shown in the figures), such as a knob or key. The user thereby performs a confirmation action. For this purpose, the operating element is also signal-coupled to the control device 18.

[0043] Instead of rotating the graphic display element 22, the user can also vary its size. A corresponding hand gesture for enlarging or reducing the graphic display element 22 is shown in the Fig. 5 and Fig. 6 shown.

[0044] To adjust the size of the graphic display element 22, a combination gesture of the hand 24 is performed as a hand gesture. This means that the user performs two different movements with their hand 24 simultaneously. The first movement is a rotational movement, as already described above for rotating the graphic display element 22. The second movement is a displacement of the hand 24 perpendicular to the graphic user interface 16. In other words, a combination gesture of the hand 24 simultaneously performs a rotational and translational, i.e., linear, movement of the hand 24.

[0045] With the combination gesture, the user, as already explained for the rotating movement, first clenches their hand 24 into a fist or curls their hand 24 and fingers together as if holding a screwdriver. They then move their hand 24 with a simultaneous rotating movement perpendicular to the user interface 16 or away from the user interface 16. To enlarge the graphic display element 22, the user preferably moves their hand 24 towards the graphic user interface 16. To reduce the size of the graphic display element 22, the user preferably moves their hand 24 away from the graphic user interface 16. In order to change the representation of the graphic display element 22, the user therefore performs a screwing movement with their hand 24. The user moves their hand 24 as if they were operating a screwdriver.

[0046] Overall, gesture control of rotation and zoom functions is thus realized.

[0047] A particularly preferred embodiment is described below.

[0048] Gesture operation on the touchscreen is recorded using a TOF camera (Time of Flight camera) to combine a zoom function and a rotation function for display elements. Initially, individual menu items, such as radio or navigation, are shown on the touchscreen in a menu arrangement in a standard orientation. One of the display elements is then selected by highlighting it. To mark it, the user positions their hand over the symbol or display element. Direct highlighting is also possible using touch. By highlighting, the symbol is now activated and is highlighted accordingly. The symbol can now be rotated using a screw movement of the hand. For example, the hand is held and then turned left or right. Alternatively or additionally, a multi-touch gesture can also be performed. A screw movement of the hand rotates the symbol.If the hand is moved in the X direction, i.e. perpendicular to the touchscreen user interface, in addition to the screw movement and the screw is being turned at the same time, the symbol can be made smaller or larger.

Claims

[1] Method for operating an operating device (10) with a display device (14), wherein the display device (14) provides a graphical user interface (16) on which a plurality of graphical display elements (22) are displayed in a display plane, wherein the plurality of graphical display elements (22) are each coupled to an operating function, characterized by the steps: - selecting a graphic display element (22) from the plurality of graphic display elements (22), wherein for the purpose of selection, a position of a hand (24) or a finger (26) of the hand of a user relative to the graphic display element (22) is detected by means of a detection device (20); - Contactless detection of at least one further hand gesture of the user by means of the detection device (20), wherein the further hand gesture is carried out by the hand (24) of the user; - Setting a display parameter of the selected graphic display element (22) as a function of the detected further hand gesture of the user by means of a control device (18), wherein the display parameter of the selected graphic display element (22) is set independently of display parameters of the remaining graphic display elements (22). [2] Method according to claim 1, characterized by that a rotational movement of the hand (24) is detected as a hand gesture and as a result the selected graphic display element (22) is rotated in the display plane of the graphical user interface (16) corresponding to a direction of rotation of the rotational movement of the hand (24). [3] Method according to claim 1 or 2, characterized bythat a combination gesture of the hand (24) consisting of a rotational movement and a simultaneous movement of the hand (24) perpendicular to the graphical user interface (16) is detected as a hand gesture and as a result the selected graphical display element (22) is displayed enlarged or reduced in size in the display plane of the graphical user interface (16) corresponding to the combination gesture. [4] Method according to one of the preceding claims, characterized by that the detection device (20) detects the hand gesture after selection only after a predetermined displacement of the hand (24) from an initial position. [5] Method according to one of the preceding claims, characterized bythat after setting the display parameter of the selected graphic display element (22), the selected graphic display element (22) is held in the set display if a confirmation action is detected by the detection device (20) and / or if the confirmation action is not performed, the selected graphic display element (22) is automatically returned to a reference position (30) after a predetermined time interval after setting the display parameter. [6] Method according to one of the preceding claims, characterized by that when the graphic display element (22) is selected, the graphic display element (22) is provided with a marking. [7] Operating device (10) comprising: - a display device (14) which is configured to provide a graphical user interface (16) on which a plurality of graphical display elements (22) are displayed, wherein the plurality of graphical display elements (22) are each coupled to an operating function; characterized by , - a detection device (20), wherein the detection device (20) is configured to detect a selection of one of the graphic display elements (22) by detecting a position of a hand (24) or a finger of the hand (24) of a user relative to the graphic display element (22), wherein - the detection device (20) is further configured to detect a hand gesture of the user without contact; and - a control device (18) which is designed to set a display parameter of the selected graphic display element (22) as a function of the detected hand gesture of the user, independently of display parameters of the other graphic display elements (22). [8] Operating device (10) according to claim 7, characterized by that the detection device (20) comprises a camera, in particular a TOF camera. [9] Operating device (10) according to claim 7 or 8, characterized by that the display device (14) and a part of the detection device are designed as a touchscreen. [10] Motor vehicle (12) with an operating device (10) according to one of claims 7 to 9.

Images