User interface of a vehicle and method for configuring and controlling the user interface

Abstract

The present invention relates to a user interface for a vehicle and a method for configuring and controlling the user interface, wherein a signal device of the user interface has a large number of ultrasonic speakers for generating a haptically perceptible area within an interaction space. In order to create a user interface for a vehicle and a method for configuring and controlling a user interface that is flexibly operable and has more degrees of freedom with respect to display aspects, the user interface according to the present invention has: a display device for generating a virtual image of an operating interface; an operating unit for operating the virtual operating interface, the operating unit having a sensor system for detecting and / or tracking an operating object; and a signal device that is arranged to generate a perceptible signal based on an operation of the operating interface. For controlling and configuring the user interface, it is provided that the module successively emits defined test signals, which are recorded by an ultrasonic microphone directly and / or after reflection on one or more reflectors, the ultrasonic microphone being arranged directly on the module and / or freely arranged in space, so that the relative position and orientation of the module and the ultrasonic microphone are determined based on the propagation time and / or intensity of the recorded test signals.

Description

Field of the Invention

[0001] The present invention relates to a user interface for a vehicle.

[0002] The present invention also relates to a method for configuring and controlling a user interface, the signal device of the user interface having a plurality of ultrasonic speakers for generating a haptically perceptible area within an interaction space. Background Art

[0003] In conventional vehicles, user interfaces for operating the vehicle's own peripheral devices (especially navigation devices, infotainment systems, air conditioning facilities, etc.) are all oriented in the driving direction, so that the driver sitting in the driving direction can easily reach the user interface. However, in the case of autonomous driving, especially in the current autonomous driving levels 4 and 5, it is no longer mandatory for the driver to be oriented in the driving direction, but the driver can also rotate his seat position 180° in the direction of the passenger compartment (rear) to communicate with other passengers there, for example. In this position, it is no longer easy to reach the conventional user interface oriented in the driving direction.

[0004] In addition, there are known user interfaces in the form of displays, which are arranged in the passenger compartment, for example on doors, windows or tables. Although these user interfaces can be reached when the driver changes orientation, these user interfaces are fixedly installed and only allow display and operation in a manner strictly restricted by the display and the position. Therefore, the degree of freedom regarding the display of the operation is limited because it is not possible to arrange the user interface in a freely selectable or operator-position-adapted manner. Summary of the Invention

[0005] Starting from this, the object of the present invention is to create a user interface for a vehicle and a method for configuring and controlling the user interface, which can be operated flexibly and has more degrees of freedom through position-independent display.

[0006] This object is first solved by a user interface according to claim 1. According to the present invention, the user interface has: a display device for generating a virtual image of an operation interface; an operation unit for operating the virtual operation interface, the operation unit having a sensor system for detecting and / or tracking an operation object; and a signal device, which is arranged to generate a perceptible signal according to the operation of the operation interface.

[0007] Thereby, a flexible user interface is proposed, which can in particular be arranged in the passenger compartment of a vehicle and projects a virtual image of an operating interface into an interaction space arranged in the passenger compartment. Thereby, the operating unit can be easily reached and operated from different positions, so that the operating unit allows position - independent display and operation in a manner following the operator's position.

[0008] The preferred design of the present invention, in particular the preferred design of its individual components, is described below and in the dependent claims.

[0009] According to a first preferred embodiment, the display device has a 3D image generation system for generating a real image of the operating interface and an optical device for projecting the real image of the operating interface into the interaction space, wherein the interaction space is within the operator's field of view. Such a 3D image generation system is preferably an autostereoscopic system, in particular a 3D display with lenticular lenses. Alternatively, the 3D image generation system can also be a light field display, a volumetric display or a 3D display with a parallax barrier. The optical device of the display device causes a display perception that is separated from the surface and floating and can herein be configured as a multi - mirror plate, in particular as a multi - mirror plate with dichroic mirrors.

[0010] To avoid unwanted light reflections, the user interface has at least one optical filtering element, which can in particular be configured as a polarization filter. Specifically, it is provided that the display device has such an optical filtering element, which prevents unwanted reflections from the system, such as those caused by the display device itself. The optical device also preferably has an optical filtering element, which prevents unwanted external light reflections, such as those caused by solar radiation or lights inside and / or outside the passenger compartment.

[0011] Within the scope of the display device, according to another preferred design of the present invention, a sensor system is provided for detecting the facial orientation or the line of sight direction of the operator relative to the virtual display operating interface, which can be designed, for example, based on a camera. Thereby, the virtual operating interface can be observed from different perspectives.

[0012] In order to be able to operate the operating interface holographically displayed by the display device, a sensor system is provided according to a preferred design embodiment of the present invention. The sensor system enables precise detection and tracking of operating objects within the interaction space. Depending on the application, for example, a hand, finger, or face can be defined as an operating object. According to a preferred design embodiment of the present invention, the sensor system has at least one time-of-flight sensor and / or at least one stereo camera system, which preferably operate in the infrared spectrum. Alternatively and / or additionally, the sensor system can also have an acoustic camera with ultrasonic speakers and ultrasonic microphones, which will be discussed in more detail later. Here, the combination of different sensors within the sensor system has proven particularly advantageous in order to detect a larger motion spectrum of these operating objects through a suitable sensor distribution. Otherwise, in the case of using only a single sensor, occluded areas can occur, which may lead to misinterpretation of the operation. By integrating such a sensor system for identifying and / or tracking operating objects in the interaction space, hand-based operation can be directly performed using the operating units of the holographic display. Thus, the operation of the user interface is not interface-bound, i.e., there is no direct contact with the user interface of the component.

[0013] The user interface has a signaling device that signals possible operations on the virtual operating interface. According to a particularly preferred design embodiment of the present invention, it is stipulated in this regard that the signaling device is a device for generating a haptically perceptible area within the interaction space of the operating interface. For this purpose, suitable devices can generate, for example, acoustic signals, in particular ultrasonic signals, airflows, or light pulses. Alternatively, it is also possible to form a plasma or temperature stimulation in the air.

[0014] However, according to a preferred design embodiment of the present invention, it is preferably stipulated that the device for generating a haptically perceptible area within the interaction space has a large number of ultrasonic speakers, and for this purpose, the ultrasonic signals of the ultrasonic speakers can be coordinated with respect to phase, frequency, and amplitude. The ultrasonic speakers form multiple modules individually or in groups, and the multiple modules can be positioned or arranged at the interaction space such that

[0015] a) at least one ultrasonic speaker of one module is angularly oriented with respect to at least one ultrasonic speaker of another module, and / or

[0016] b) at least one ultrasonic speaker of the module is spaced from the common plane of the other ultrasonic speakers, and / or

[0017] c) the modules have a selectable spacing from each other.

[0018] Due to the localizability of each module with an ultrasonic loudspeaker and thus the freely selectable positions, the device can be optimally adapted to the existing structural space, thereby avoiding conflicts between the preferred positions of the image-generating device and the haptic-generating device. Through arbitrary, especially angled, arrangements of the individual loudspeakers and / or modules with respect to one another, an optimal three-dimensional orientation of these modules in space is achieved, where these modules can be arranged at arbitrary positions outside the interaction space and oriented in a desired orientation. Thereby, a personalized design of the device for generating haptic feedback can be achieved while optimally making full use of the available structural space. In addition, a device with substantially freely positionable modules has minimal space requirements, especially since the intensity of the haptic effect can be highly personalized due to the adjustability of the orientation of each module.

[0019] According to another preferred design, the ultrasonic loudspeakers of the module are arranged on a common module plane. Preferably, the ultrasonic loudspeakers of the module have orientations parallel to one another here. Although the specific design of such a module is not pre-given, especially with regard to the number and arrangement of the ultrasonic loudspeakers and is basically arbitrary, in practice, a matrix arrangement of the ultrasonic loudspeakers has proven to be advantageous. In particular, for example, a square matrix arrangement of 4×4 ultrasonic loudspeakers is preferred.

[0020] According to an advantageous embodiment of the invention, the device has at least three modules, the module planes of which are oriented at an angle with respect to one another. Thereby, these modules can be arranged around the interaction space in an optimal manner according to the structural space and aligned with the interaction space.

[0021] In order to configure the device according to the unique positions and orientations of the modules and the ultrasonic loudspeakers arranged on the modules, a control unit and the software (firmware) and programming interface (API) included on the control unit are preferably provided. The control device calculates unique ultrasonic signals with respect to its own phase, frequency, and amplitude according to the unique module positions and module orientations in order to generate regions with haptic feedback at arbitrary points in the interaction space. Here, if such regions should be generated at multiple positions, appropriate groups are formed from the existing modules, which are assigned to specific points and generate haptic feedback at that position. Since the haptic regions are generated from three different directions using the current three-dimensional arrangement of these modules, the haptic regions can also be felt in the case of different hand orientations, and possible shadow regions within the interaction space can be effectively avoided.

[0022] The entire device can be controlled via the control unit and thus the individual modules can be controlled, so that in this case simple and thus inexpensive modules can be used. In this case, in order to ensure correct manipulation of all modules in terms of time, all line lengths and signal propagation times must be known or of the same length.

[0023] Alternatively, it is provided that each module has an ultrasonic speaker for configuring and / or manipulating the module and a separate microcontroller for communicating with the control unit. This results in optimal networking of the module with the control unit configured as the main control device, and the communication can be carried out via a bus system with real-time capabilities.

[0024] According to a particularly preferred design of the invention, a plurality of ultrasonic microphones are provided, which are arranged directly on the module or freely in space, in particular freely in the passenger compartment of a vehicle. With the ultrasonic microphones, the simple automatic configurability of the device can be determined, and the ultrasonic microphones can also be used as three-dimensional acoustic cameras, with which objects in the interaction space can be identified, labeled and tracked by continuously determining their positions. This can eliminate the otherwise required image detection device, such as an image detection device in the form of a stereo camera. This will be discussed in detail when describing the method according to the invention.

[0025] The described device for generating a haptically perceptible feedback can be integrated into different vehicle models with different internal dimensions due to the flexible arrangement of the ultrasonic speakers and the automatic configurability.

[0026] Instead of the described device for generating a haptically perceptible area in the interaction space of the operating interface, the signaling device can also be provided for generating acoustic and / or optical feedback when operating the virtual operating interface. In the simplest case, the acoustic feedback is achieved by a sound emitted by a speaker or the like. In contrast, in the simplest case, the optical feedback can be achieved by a signal lamp arranged directly on the user interface. It can also be provided that in the case of operating a holographic display operating interface, the operating interface itself changes its display and shows the operation that has been carried out, for example, by other colors of the operating interface and / or by a short flash.

[0027] As already explained at the beginning, the invention also relates to a method for configuring and controlling a user interface, the signaling device of which has a large number of ultrasonic speakers for generating a haptically perceptible area in the interaction space.

[0028] According to claim 18, as provided by the present invention, in order to configure the device, the module successively emits defined test signals, which are recorded by an ultrasonic microphone directly and / or after reflection on one or more reflectors. The ultrasonic microphone is directly arranged on the module and / or freely arranged in space, so that the relative position and orientation of the module and the ultrasonic microphone can be obtained based on the propagation time and / or intensity of the recorded test signals. Considering the so-determined and known positions and orientations of the module and the ultrasonic microphone arranged on the module, the ultrasonic signals of the module can be advantageously coordinated in terms of their own phase, frequency, and amplitude, so that a haptically perceptible area can be generated at any position within the interaction space. The device is self-calibrated by the ultrasonic microphone, thereby simplifying the configuration of the device, which can be basically automatically performed without manual pre-giving. Here, the accuracy and resolution of the configuration increase with the increase in the number of modules and / or ultrasonic microphones. Therefore, it is preferably provided that at least three ultrasonic microphones are arranged in a manner that can be basically freely distributed in space. When the ultrasonic microphone can be freely positioned, the following advantages are also obtained: the position is not limited to the module position, and thus the number of ultrasonic microphones used and different directions, that is, the angle between the ultrasonic microphone and the measuring body, can be arbitrarily increased, thereby increasing the recognition range and the accuracy of the method.

[0029] In addition to the configuration of the device, complementarily, the ultrasonic microphone can also be used as an acoustic camera, with which objects within the interaction space can be identified, marked, and tracked by continuously determining their positions, so that the acoustic camera can also become part of the aforementioned sensor system. For this purpose, it is preferably provided that the ultrasonic microphone records the reflections from stationary or moving objects within the interaction space, such as the reflections of a hand inserted into the interaction space, and determines the position, shape, and movement of the object based on the reflected signals. In order to identify and track objects within the interaction space, the ultrasonic frequencies originally emitted by the ultrasonic speakers for generating haptic effects can be used. Alternatively, other different frequencies can also be used.

[0030] Finally, according to a preferred design of the method, it is provided that the object can be determined by existing classification algorithms, where the classification algorithms compare the reflected signals with existing data sets related to the object and allow for a clear assignment of the object in the case of consistency. This also enables specific and targeted stimulation of objects within the interaction space. Description of the Drawings

[0031] The following explains the specific implementation manners of the present invention based on the drawings. In the drawings:

[0032] Figure 1 shows a user interface, and

[0033] Figure 2 shows a device for generating a haptically perceptible area within an interaction space. DETAILED DESCRIPTION

[0034] Figure 1 Shows a user interface 1 within a passenger compartment (not shown) of a vehicle. The user interface 1 has a display device 2 by means of which a virtual image 3 of an operating interface can be projected into an interaction space 4 in which the operating interface can be recognized holographically by an operator 5. For this purpose, the virtual image is projected into the interaction space 4 via an optical steering assembly 6, in particular a multi-mirror plate.

[0035] To avoid internal reflections within the user interface 1, the display device 2 has a polarization filter 7 covering the display device 2.

[0036] In a similar manner, the illustrated optical assembly 6 also has a polarization filter 8 which prevents unwanted external reflections, in particular due to solar radiation and / or external light sources.

[0037] Optionally, the user interface 1 has an adjustment mechanism 9 which manually or automatically adjusts the angles and their position and orientation relative to the operator 5 of the display device 2 and / or the optical steering assembly 6.

[0038] The user interface 1 also has an operating unit for operating the virtual operating interface, which operating unit includes a sensor system 10 for detecting and / or tracking an operating object 11. In the illustrated embodiment, the sensor system 10 is arranged below the interaction space 4 and recognizes a hand moving within the interaction space 4 which forms the operating object 11 in the present case. By means of current object recognition and object tracking, the user interface 1 recognizes whether an operation should be carried out using the user interface 1 and, if necessary, which function should be operated and forwards corresponding commands.

[0039] Furthermore, the user interface 1 has a further sensor system 12 for detecting the facial orientation or object orientation of the operator 5 relative to the holographic display of the operating interface, which holographic display of the operating interface can be designed as camera-based in the illustrated embodiment. In particular, by means of a corresponding link with the adjustment mechanism 9, the virtual operating interface can be observed and operated from different viewing angles.

[0040] To signal to the operator 5 an operation that has been carried out, various options are provided. In addition to the light signals already mentioned, the user interface 1 as Figure 1As shown, there may be a device 21 for generating haptic feedback, and for this purpose, the device has a large number of ultrasonic speakers. The device 21 is only shown exemplarily in Figure 1 below the interaction space 4. In principle, the ultrasonic speakers can be arranged and oriented substantially arbitrarily within the passenger compartment of the vehicle. Refer to Figure 2 for an explanation of its specific embodiments and related configuration and control methods.

[0041] The shown device 21 has a plurality of ultrasonic speakers 22, 22 represented in a cylindrical shape. In this embodiment, these ultrasonic speakers are grouped to form a total of 25 modules 23, 23', and each of these modules has 16 ultrasonic speakers 22, 22'. Overall, the shown device has 400 ultrasonic speakers 22, 22'. The ultrasonic speakers 22, 22' of each module 23, 23' are respectively arranged in the form of a matrix with 4×4 ultrasonic speakers 22, 22' on a common module board 24 and are oriented parallel to each other. The module board 24 is configured to be flat and forms a module plane. The modules 23, 23' are arranged annularly and at an angle relative to the horizontal line such that at least one ultrasonic speaker 22 of one module 23 is oriented at an angle with at least one ultrasonic speaker 22' of another module 23' and is spaced from the common plane of the other ultrasonic speakers. Thus, the modules 23, 23' and the ultrasonic speakers 22, 22' can be substantially freely positioned and, in the shown embodiment, are aligned with the center of the interaction space 4 arranged above them.

[0042] For configuring and controlling the device 21, the modules 23, 23' at least partially have ultrasonic microphones 25, and in this embodiment, only three ultrasonic microphones 25 are shown exemplarily. To improve the accuracy of the configuration and control method, preferably all modules 23, 23' have at least one ultrasonic microphone 25. For configuration, the ultrasonic microphones 25 receive test signals, where the test signals are successively emitted by the modules 23, 23' and may be reflected on a reflector (not shown). Through the recorded signals, in particular the frequency and amplitude of these signals, the position and orientation of each module 23, 23' can be clearly determined, so that these ultrasonic signals can be adjusted in terms of the phase, frequency, and amplitude of these ultrasonic signals such that a point with a haptically perceivable area can be generated at any position within the interaction space 4. In this figure, the interaction space 4 with possible haptic feedback is shown substantially in a partially spherical manner, where the shape of this interaction space 4 also depends on the specific and freely selectable arrangement of the modules 23, 23'.

[0043] In the interaction space 4 itself, haptically perceptible areas can be generated at any position, which can be sensed by a person with a hand 11 reaching into the interaction space 4. The ultrasonic microphone 25 records the reflection of the ultrasonic signal on the hand 11 or another object, so that the movement of the hand 11 in the interaction space 4 can be determined based on the reflected signal.

[0044] A control unit 28 and a programming interface 29 are provided for controlling and configuring the device 21. In the illustrated embodiment, the control unit 28 is connected to the device 21 such that these control signals are directly sent from the control unit 28 to the individual modules 23, 23'. Alternatively, the modules 23, 23' can also have separate microcontrollers (not shown), which are connected to the modules 23, 23' on the one hand for control and to the control unit 28 on the other hand.

[0045] For controlling and coordinating all components of the current user interface 1, the user interface has an additional computing unit 13 connected to all components to be controlled (see Figure 1 ).

[0046] List of reference numerals

[0047] 1 User interface

[0048] 2 Display device

[0049] 3 Virtual image of the operation interface

[0050] 4 Interaction space

[0051] 5 Operator

[0052] 6 Optical steering component

[0053] 7 Polarization filter

[0054] 8 Polarization filter

[0055] 9 Adjustment mechanism

[0056] 10 Sensor system

[0057] 11 Operating object

[0058] 12 Sensor system

[0059] 13 Computing unit

[0060] 21 Device for generating haptic feedback

[0061] 22, 22' Ultrasonic speakers

[0062] 23, 23' Modules

[0063] 24 Module board

[0064] 25 Ultrasonic microphone

[0065] 28 Control unit

[0066] 29 Programming interface

Claims

1. A user interface for a vehicle, wherein the user interface is arranged in the passenger compartment of the vehicle and projects a virtual image of an operation interface into an interaction space arranged in the passenger compartment, and wherein the user interface comprises the following: a) A display device (2) for generating a virtual image (3) of the operation interface, having an optical steering assembly (6), wherein the display device (2) has a 3D image generation system for generating a real image of the operation interface, and wherein the optical steering assembly (6) is designed to project the real image of the operation interface into the interaction space (4), which is in the field of view of an operator (5); An adjustment mechanism (9), wherein the adjustment mechanism manually or automatically adjusts the angle of the display device (2) and / or the optical steering assembly (6), as well as their position and orientation relative to the operator (5); b) An operation unit for operating the virtual operation interface, the operation unit having a sensor system (10) for detecting and / or tracking an operation object (11); And c) A signal device, which is arranged to generate a perceptible signal according to an operation on the operation interface.

2. The user interface according to claim 1, Characterized in that The 3D image generation system is an autostereoscopic system.

3. The user interface according to claim 1, Characterized in that The optical steering assembly (6) is a multi-mirror plate.

4. The user interface according to claim 1, Characterized in that The user interface (1) has at least one optical filtering element (7, 8) to avoid reflections.

5. The user interface according to claim 4, Characterized in that The optical steering assembly (6) and / or the display device (2) of the user interface (1) each have at least one optical filtering element (7, 8).

6. The user interface according to claim 1, Characterized in that The sensor system (10) has a time-of-flight sensor and / or a stereoscopic camera system.

7. The user interface according to any one of claims 1 to 6, Characterized in that The signal device is a device (21) for generating a haptically perceptible area within the interaction space (4) of the operation interface.

8. The user interface according to claim 7, Characterized in that The device (21) for generating a haptically perceptible area within the interaction space (4) has a large number of ultrasonic speakers, for which the ultrasonic signals of the ultrasonic speakers can be coordinated in terms of phase, frequency and amplitude, and wherein the ultrasonic speakers form multiple modules individually or in groups, and the multiple modules can be positioned or arranged at the interaction space (4) such that a) At least one ultrasonic speaker of one module is angularly oriented with respect to at least one ultrasonic speaker of another module, and / or b) At least one ultrasonic speaker of a module is spaced from the common plane of other ultrasonic speakers, and / or c) The modules have a selectable spacing from each other.

9. The user interface according to claim 8, Characterized in that The ultrasonic loudspeakers of the module are arranged in a common module plane.

10. The user interface according to claim 8, wherein, the ultrasonic loudspeakers of the module have orientations parallel to each other.

11. The user interface according to claim 8, wherein, the ultrasonic loudspeakers of the module are arranged as a matrix.

12. The user interface according to claim 8, wherein, the device (21) has at least three modules, and the module planes of the at least three modules are angularly oriented with respect to each other.

13. The user interface according to claim 8, wherein, a control unit (28) and software and programming interfaces included on the control unit are provided for configuring the device (21) according to the unique positions and orientations of the modules.

14. The user interface according to claim 13, wherein, each module has a separate microcontroller for configuring and / or controlling the ultrasonic loudspeakers of the module and for communicating with the control unit (28).

15. The user interface according to claim 8, wherein, a plurality of ultrasonic microphones (25) are provided, and the plurality of ultrasonic microphones are arranged directly on the module or freely arranged in space.

16. The user interface according to claim 8, wherein, the signal device is provided for generating acoustic and / or optical feedback when operating the virtual operation interface.

17. The user interface according to claim 11, wherein, the ultrasonic loudspeakers of the module are arranged as a square matrix.

18. The user interface according to claim 2, wherein, the 3D image generation system is a 3D display with lenticular lenses.

19. The user interface according to claim 4, wherein, the optical filter element is a polarization filter.

20. A method for configuring and controlling the user interface according to any one of claims 8 to 17, wherein, in order to configure the device (21), the modules successively emit defined test signals, and the test signals are recorded by ultrasonic microphones (25) directly and / or after reflection on one or more reflectors. The ultrasonic microphones (25) are arranged directly on the module and / or freely arranged in space, so that the relative positions and orientations of the module and the ultrasonic microphones (25) are obtained according to the propagation time and / or intensity of the recorded test signals. Wherein, when the positions and orientations of the module and the ultrasonic microphones (25) are known, the ultrasonic signals of the module are coordinated in terms of their own phase, frequency and amplitude, so that a tactilely perceptible area can be generated at any position within the interaction space (4).

21. The method according to claim 20, wherein, reflections from stationary or moving objects within the interaction space (4) are recorded by the ultrasonic microphones (25), and the position, shape and / or movement of the object are determined according to the reflected signals.

Images