Gesture interaction with a vehicle's driver information system

Abstract

A control device for moving and / or zooming a display content, the control device comprising: a display unit (5) with a display content (7), at least one camera (12.1, 12.2) configured to record a detection area in front of the display unit (5), a gesture recognition unit (14) coupled to the at least one camera (12.1, 12.2) and configured to recognize a predetermined gesture (G) performed with a hand (H) and a current position (P) of the gesture (G) in the detection area, and a display content adaptation unit (16) configured to adapt the display content (7) according to a change in the current position (P) of the gesture (G),in particular, to shift the display content accordingly when the position (P) changes in a plane parallel to the display unit (5) and / or to enlarge or reduce the display content when the position (P) changes towards or away from the display unit (5), wherein the display content adaptation unit (16) is configured to enlarge or reduce the display content (7) when the position (P) changes towards or away from the display unit (5) by a factor, wherein the factor corresponds to a power of base 2 with a change in the distance between the position (P) of the gesture (G) and the display unit (5) as an exponent.

Description

[0001] The invention generally relates to the automatic recognition of gestures performed by humans using a computer (gesture recognition). In particular, the invention relates to gesture interaction with a driver information system of a vehicle. Background of the invention

[0002] In the context of human-computer interaction, Kurtenbach et al. ("Gestures in Human-Computer Communication" in "The Art of Human-Computer Interface Design", pp. 309-317, 1990) define a gesture as a movement of the body that conveys information. For example, waving goodbye is a gesture. Pressing a key on a keyboard, however, is not a gesture, since the movement of the finger on its way to the key is neither observed nor significant; only the key that was pressed is relevant.

[0003] In vehicles, it is already known to operate vehicle components by interpreting gestures. DE 42 01 934 A1 discloses a data processing system that recognizes gestures and translates them into commands for controlling the data processing system. EP 1 408 443 A1 also discloses a system in which function control is achieved via gesture recognition, which includes, for example, various gestures in the form of hand or finger positions captured by a 2D or 3D camera. Corresponding actions are assigned to the respective hand and finger positions. The interpretation uses both static gestures and specific movement sequences.

[0004] Another approach is gesture recognition using stereoscopy. EP 1 477 924 A2, for example, shows a gesture recognition device for recognizing a person's postures or gestures based on images of the person taken by two cameras.

[0005] Various user input methods using gesture recognition on screens or navigation devices are also disclosed in the patent applications DE 10 2013 000 066 A1, DE 10 2013 201 746 A1, and DE 10 2006 037 154 A1. Scrolling of displayed objects controlled by 3D movement is known from US patent 2009 / 0 158 203 A1.

[0006] Modern vehicles provide the driver with extensive information and control options, for example, via a central display screen. The operation of modern navigation systems, in particular, is a constant subject of research, with the aim of improving driver usability, especially while driving, so that the driver is as little distracted from actually driving the vehicle as possible and their attention to the surroundings and road traffic is minimized. However, the aforementioned control options using gesture recognition are often not very intuitive. Therefore, the desired simplification of operating a vehicle component is often not achieved, as the driver's gestures are not always translated into the intended inputs or are not recognized at all.The consequence of this is that the actual operating process takes longer than theoretically expected and the driver's attention is more distracted. Disclosure of the invention

[0007] A possible object of the present invention is to propose an improved control method, in particular for controlling a map section of a navigation system or screen content of a driver information system, for example an implemented internet browser, which overcomes the disadvantages of the prior art.

[0008] At least one of the problems is solved by the features of the independent claims. Further features and details of the invention will become apparent from the dependent claims, the description, and the drawings.

[0009] A key concept of the invention is to control a displayed map section of a navigation map from a navigation system or comparable movable content on a central display unit of a driver information system via a gesture, preferably a free-space gesture, for example, to move or zoom it. That is, with a special gesture, e.g., a pincer grip with two fingers, the displayed content on the central display can be adjusted according to the position of a virtually graspable and movable camera in three dimensions (3D) in free space.

[0010] The control device according to the invention provides the driver or operator with an intuitive operating experience through a specific correlation between hand movement and changes in the display content. This simplifies and enhances the safety of controlling the navigation system or driver information system, thereby reducing the driver's distraction from operating the vehicle.

[0011] The invention thus relates to a control device for shifting and / or zooming the display content of a display unit. The control device comprises: (i) a display unit with display content, at least one camera configured to capture a detection area in front of the display unit, (ii) a gesture recognition unit coupled to the at least one camera and configured to recognize a predetermined gesture performed with a hand and the current position of the gesture within the detection area, and (iii) a display content adjustment unit configured to adjust the display content according to a change in the current position of the gesture, in particular to shift the display content accordingly when the position changes in a plane parallel to the display unit, and / or to enlarge or reduce the display content when the position changes towards or away from the display unit.

[0012] The display content adjustment unit is configured to enlarge or reduce the displayed content by a factor when the user's position changes relative to or away from the display unit. This factor is a power of 2 with the change in distance between the gesture and the display unit as the exponent. This power of 2, with the change in distance as the exponent, ensures that a defined movement of the user's gesture or hand in the x-direction doubles or halves the displayed content. This allows switching between a wide range of zoom levels with a single hand movement. The exponent can also be further scaled using a sensitivity factor.

[0013] The control unit can have two cameras to record the detection area in front of the display unit. This allows the gesture recognition unit to determine the current position of the gesture within the detection area using stereoscopy.

[0014] At least one of the cameras can be an infrared camera and / or a time-of-flight camera and / or a structured-light camera.

[0015] The actual gesture recognition can be performed using technical image analysis based on a continuous sequence of images of a user's hand / finger positions. Two approaches are generally possible: The gesture recognition unit can contain a database of relevant gesture examples, compiled from a large number of videos (e.g., over 1,000) analyzing different variations of the gesture along a meridian. An optically captured gesture can then be compared to the database using image recognition algorithms, and the relevant gesture can be identified based on this comparison to trigger the corresponding operating function. Alternatively, a so-called skeletal recognition method can be used, in which the hand and / or fingers are detected in the image data, and predefined gestures are inferred using a simplified skeletal model.

[0016] In the actual gesture recognition process, the image information is processed by established image processing algorithms that analyze the raw data and ultimately recognize the gestures. Pattern recognition algorithms can be used for this purpose. To remove noise from the input data and reduce data size, the image data can be preprocessed in a first step. Subsequently, features are extracted from the image data to recognize the gesture through classification. Hidden Markov models, artificial neural networks, and other established techniques from artificial intelligence research can be used for this.

[0017] The gesture recognition unit and / or the display content adjustment unit is / are preferably configured to move and / or zoom the display content as long as the predetermined gesture is recognized. The detection area is preferably a predetermined space in front of the display unit.

[0018] The display content adjustment unit can be configured to shift the display content when the gesture's position changes in a plane parallel to the display unit, such that the length of the movement of the display content corresponds to the distance of the gesture's movement in the plane parallel to the display unit. That is, a hand movement of 10 cm to the right in a plane parallel to the display unit results in a corresponding movement of the display content by 10 cm to the right. This provides a perfect mental model for the user, making the controls intuitive.

[0019] The display content adaptation unit can be further configured, once the coordinates of the last gesture position have been detected, to continue displaying the content according to the last detected position or the last detected speed at which the gesture position changed, preferably slowing down continuously until it comes to a standstill. This control behavior gives the user the feeling of being able to initiate or nudge the display content with the predefined gesture.

[0020] The displayed content is preferably a map section, such as a navigation map from a navigation system. The map data, and thus the map itself, can be stored, for example, in a database of the navigation system. The display content adjustment unit then serves to adjust the map section according to changes in the current position indicated by the gesture.

[0021] The invention further relates to a vehicle, in particular a motor vehicle, with a driver information system which has at least one display unit for displaying graphical data, for example from a navigation system or an internet browser, and a control device according to the invention, as described above, for moving and / or zooming the display content.

[0022] The invention is particularly well suited for controlling the display content of a driver information system in a vehicle. However, the invention can also be implemented on smartphones, tablet computers, or devices with a display whose content can be adjusted by a user according to the control method proposed here. Preferred embodiments

[0023] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. Likewise, the features mentioned above and those described further here can be used individually or in any combination. Functionally similar or identical parts or components are sometimes provided with the same reference numerals. The terms "left," "right," "top," and "bottom" used in the description of the exemplary embodiments refer to the drawings in an orientation with normally legible figure labels or reference numerals.The embodiments shown and described are not to be understood as exhaustive, but rather serve as examples to illustrate the invention. The detailed description is intended to inform those skilled in the art; therefore, known circuits, structures, and methods are not shown or explained in detail in order to avoid complicating the understanding of the present description. Fig. Figure 1 illustrates the basic principle of map section control. Fig. Figure 2 schematically shows the components of a navigation system with stereoscopic gesture recognition and gesture position recognition for adjusting a map section according to the principle of Fig. 1.

[0024] The following description explains the proposed control of the display content of a display unit using the example of controlling a displayed map section of a navigation system. Naturally, the control described here can, in principle, be applied analogously to other display content of a driver information system. As a further example, consider an internet browser integrated into the driver information system, where – as with the map section – small content can be focused with a short and quick gesture.

[0025] Fig. Figure 1 illustrates the basic principle of the map section control proposed here. A specific map section 7 from a (navigation) map 1 containing graphically processed geographic and road data is to be displayed on a display 5. The map section 7 is captured by a camera 3, which is aligned with the map 1. The image data captured by the camera 3 from a position P defines the map section 7. The outer boundary of the map section 7 is essentially determined by the distance of the camera 3 to the map 1, the opening angle 3.1 of the camera 3, and the format of the image sensor (not shown) of the camera 3. The image data is transmitted from the camera 3 to the display unit 5 and displayed there.

[0026] By changing the position P of camera 3 in the y and / or z direction, map section 7 can be moved accordingly. By changing the position P of camera 3 in the x direction, map section 7 can be zoomed in and out. That is, if position P is moved towards map 1 in the x direction, map section 7 is reduced in size. Since the display area of ​​the unit remains the same, this means that a smaller portion of map 1 is displayed, effectively zooming in on map 1. If position P is moved away from map 1 in the x direction, map section 7 is enlarged accordingly. Since the display area of ​​the unit remains the same, this means that a larger portion of map 1 is now displayed, effectively zooming out on map 1.

[0027] The above basic principle is, according to the first aspect, transferred by means of gesture control in free space to control a currently displayed map section 7 on the display unit 5 of a navigation system. The position of a predetermined gesture G in the form of a specific hand / finger position is recognized and translated into a position P of the virtual camera 3. Fig. 1 converted. That is, an adjustment of the map section 7 is made depending on the movement of position P of a recognized gesture G according to the Fig. The principle explained in point 1 was carried out.

[0028] Fig. Figure 2 schematically shows the components of a navigation system with gesture recognition and stereoscopic gesture position recognition for adjusting the map section 7 according to the principle of Fig. 1. The basic components of a navigation system and their function are assumed to be known and will not be explained in detail here. The additional components described below, relating to gesture recognition and stereoscopic gesture position recognition, as well as image processing and map section adjustment achievable with software, can be added to a known navigation system.

[0029] Fig. Figure 2 shows the display unit 5 of the navigation system, on which a map section 7 is displayed. Two cameras 12.1 and 12.2 for gesture recognition are arranged at the two upper corners of the display unit 5. The two cameras 12.1 and 12.2 enable stereoscopic detection of the position P of a recognized gesture G, which is performed with a hand H. For the map section adjustment described here, a gesture G has been preset in which the thumb and index finger of one hand touch each other in a pincer-like position, i.e., perform a so-called pincer grasp.

[0030] Gesture recognition has the advantage that the user's intention is clearly recognizable and the gesture can be distinguished from other hand movements. As a result, other hand movements within the detection range of cameras 12.1 and 12.2 do not lead to any (unintentional) adjustment of the map section 7. That is, the map section adjustment unit 16 is only activated when the predetermined gesture G is recognized.

[0031] A gesture recognition unit 14 with cameras 12.1, 12.2 can be a separate hardware component of a control unit 10 of the navigation system, be connected to it, or be implemented by existing software routines of the navigation system. The gesture recognition unit 14 is configured for both gesture recognition and determining the current position P of the gesture G in the free space in front of the display unit 5. The gesture recognition is implemented in a known manner using appropriate image processing and pattern recognition methods. The determination of the current position P of the gesture G is essentially based on knowledge of the positions of cameras 12.1, 12.2 and the current solid angles of the lines of sight 12.1r, 12.2r from the respective camera 12.1, 12.2, starting from the position P of the gesture G.This allows the gesture recognition unit 14 to determine the coordinates of position P with sufficient accuracy using known calculations of stereoscopy.

[0032] The 3D coordinates of position P of gesture G are preferably acquired in short time intervals, e.g. every 20 ms, and made available to the map section adjustment unit 16 to ensure smooth adjustment of the map section 7 (shift and / or zoom).

[0033] Once the gesture recognition unit 14 has recognized the predetermined gesture G, the coordinates of the (gesture) position P are recorded by the gesture recognition unit 14, as long as the user maintains the predetermined hand / finger position for gesture G. The coordinates (G nx , G ny , G nz ) form the basis for the subsequent adjustment of map section 7 by the map section adjustment unit 16. The entire map 1 (see Fig. 1) is stored in a database 18 of the navigation system, whereby the map section adjustment unit 16 is linked to the database, i.e., it has access to the map data. Known navigation systems also have options for setting the current map section. That is, the improved map section adjustment unit 16 proposed here can be added at this point in a known navigation system.

[0034] The recorded position coordinates can be marked and stored with a running index n, where the index n ranges from 1 to k. The last coordinates (G kx , G ky , G kz ) correspond to the last position P k , in which hand H performed the predetermined gesture G or left the technically defined detection area in front of the display unit 5. That is, the gesture recognition unit 14 detects k position coordinates during an interaction: x coordinate: G1x, G2x, G3x, G4x, G5x,...,Gkx y coordinate: G1y, G2y, G3y, G4y, G5y,...,Gky z coordinate: G1z, G2z, G3z, G4z, G5z,...,Gkz

[0035] The detection area can be defined as a specific space above or in front of display unit 5. Coordinates G nxyz The position P is calculated by the gesture recognition unit 14 as control input to the map section adjustment unit 16 only if the defined gesture G is recognized and the current position P of the gesture G is within the recognition area.

[0036] If a position P is detected outside the detection area or reached during an active interaction, or if the defined hand position is abandoned, the interaction is terminated. This means that no further adjustment of map section 7 takes place.

[0037] To display a specific map section 7 on the display unit 5, the current position P of the gesture G is converted into a position of a virtual camera (corresponding to camera 3 in Fig. 1), which can be moved arbitrarily in three dimensions, is converted. The calculated position Pvc of the virtual camera and an opening angle assigned to the virtual camera then define the map section 7 currently displayed on the display unit 5.

[0038] From the currently recorded position P of the gesture G with the coordinates (G nx , G ry , G nz The gesture recognition unit 14 calculates the corresponding position (P). VCnx , P VCny , P VCnz ) the virtual camera preferentially according to the following formulaic relationships: PVCnx=PVC1x⋅2((Gnx−G1x) / fx) PVCny=PVC(n−1)y+PVCnx⋅(Gny−G(n−1)y)⋅fy PVCnz= PVC(n−1)z+PVCnx⋅(Gnz−G(n−1)z)⋅fz where fx , f y , f z Sensitivity parameters are and n is the index for the positions P captured during an interaction, where - as explained above - 1 < n ≤ k.

[0039] The respective x-coordinate corresponds to the current distance of gesture G to display unit 5, or the height of the virtual camera above map 1, from which map section 7 is displayed on display unit 5. The relationship between the x-coordinate of camera P VCnx and the x-coordinate of the gesture position G nx This is used to zoom in or out on the map. The power of 2 means that a defined movement of the G gesture or hand H in the x-direction doubles the displayed map size. This allows switching between many zoom levels with a single hand movement.

[0040] Furthermore, the y- and z-coordinates P VCny , PVCnz the virtual camera from the x-coordinate P VCnx depending on the appropriate adjustment of the sensitivity parameters f y and f z The length of a card movement on display unit 5 corresponds to the distance of the movement of gesture G or hand H in the y- and / or z-direction. This means, for example, that a hand movement of 10 cm to the right results in a card movement of 10 cm to the right. This creates a perfect mental model, making the controls intuitive for the user.

[0041] According to another advantageous aspect of the proposed display content control, the adjustment of the display content can take into account the movement dynamics of gesture G or hand H during input. For this purpose, the map section adjustment unit 16 is configured after the last coordinates G are entered. kThe gesture G, sent from position P, is to continue moving map section 7 across map 1 at the last recorded speed of hand H or gesture G, with the movement continuously slowing down until it comes to a standstill. This behavior gives the user the feeling that they can "nudge" the map. This behavior can be represented by the following formulaic relationships and settings: last_velxyz=(Gkxyz−G(k−1)xyz) / steplength snxyz=0.5⋅b⋅(steplength⋅(n−k))2+last_velxyz⋅steplength⋅(n−k);(last_velxyz>0) snxyz=−0.5⋅b⋅(steplength⋅(n−k))2+last_velxyz⋅steplength⋅(n−k);(last_velxyz<0) PVCnx=PVC1x⋅2((Gkx+snx) / fx) PVCny=PVC(n−1)y+PVCnx⋅(sny−s(n−1)y)⋅fy PVCnz=PVC(n−1)z+PVCnx⋅(snz−s(n−1)z)⋅fz where n > k means that the map section 7 continues to advance after the last recognized gesture position, b is a preset braking factor, e.g. - 0.3 m / s, "steplength" is a step size corresponding to the current frame rate of cameras 12.1, 12.2, e.g. 20 ms at 50 frames / s, and "last_vel xyz “The final velocity of the movement of gesture G or hand H is, for example (G kxyz - G (k-1)xyz) / steplength.

[0042] The described progression of the map section ends as soon as s nxyz a defined minimum is reached or a new series of coordinates of a position P of a recognized gesture G is sent.

Claims

[1] Control device for moving and / or zooming a display content, the control device comprising: a display unit (5) with a display content (7), at least one camera (12.1, 12.2) configured to capture a detection area in front of the display unit (5), a gesture recognition unit (14) coupled to the at least one camera (12.1, 12.2) and configured to recognize a predetermined gesture (G) performed with a hand (H) and a current position (P) of the gesture (G) in the detection area, and a display content adjustment unit (16) configured to adjust the display content (7) according to a change in the current position (P) of the gesture (G),in particular, when the position (P) changes in a plane parallel to the display unit (5), the display content is shifted accordingly, and / or when the position (P) changes towards or away from the display unit (5), the display content is enlarged or reduced, wherein the display content adjustment unit (16) is configured to enlarge or reduce the display content (7) when the position (P) changes towards or away from the display unit (5) by a factor, wherein the factor corresponds to a power of base 2 with a change in the distance between the position (P) of the gesture (G) and the display unit (5) as the exponent. [2] Control device according to claim 1, wherein the control device has two cameras (12.1, 12.2) for recording the detection area in front of the display unit (5) and the gesture recognition unit (14) determines the current position (P) of the gesture (G) in the detection area by means of stereoscopy. [3] Control device according to claim 1 or 2, wherein the at least one camera (12.1, 12.2) is an infrared camera and / or a time-of-flight camera and / or a structured-light camera. [4] Control device according to one of the preceding claims, wherein the gesture recognition unit (14) and / or the display content adjustment unit (16) is / are configured to move and / or zoom the display content as long as the predetermined gesture (G) is recognized. [5] Control device according to one of the preceding claims, wherein the detection area is a predetermined space in front of the display unit (5). [6] Control device according to one of the preceding claims, wherein the display content adjustment unit (16) is configured to shift the display content (7) when the position (P) changes in a plane parallel to the display unit (5) such that the length of a movement of the display content corresponds to a distance of the movement of the position (P) of the gesture (G) in the plane parallel to the display unit (5). [7] Control device according to one of the preceding claims, wherein the display content adjustment unit (16) is further configured, once a last position (P) of the gesture (G) has been detected, to continue displaying the content (7) according to a last detected speed at which the position (P) of the gesture (G) has changed and to continuously slow down until it comes to a standstill. [8] Control device according to one of the preceding claims, wherein the display content is a map section (7) of a map (1) which is stored, for example, in a database (18) of a navigation system, and the display content adjustment unit (16) adjusts the map section (7) according to the change in the current position (P) of the gesture (G). [9] Vehicle, in particular motor vehicle, with a driver information system comprising at least one display unit (5) for displaying graphical data, for example from a navigation system or an internet browser, and a control device according to one of the preceding claims for moving and / or zooming the display content (7).

Images