USER INTERFACE OF A VEHICLE AND METHOD FOR CONFIGURATION AND CONTROL OF THE USER INTERFACE

Description

[0001] The present invention relates to a user interface of a vehicle.

[0002] The present invention further relates to a method for configuring and controlling a user interface, the signaling device of which comprises a plurality of ultrasonic loudspeakers for generating a haptically perceptible area within the interaction space.

[0003] In conventional vehicles, user interfaces for operating vehicle peripherals, particularly the navigation system, infotainment system, air conditioning, etc., are oriented forward of the vehicle, making them easily accessible to a driver seated facing forward. However, in autonomous driving, especially at current levels 4 and 5, it is no longer essential for the driver to face forward. Instead, they can rotate their seat 180° towards the rear passenger compartment, for example, to interact with other passengers. In this position, the conventional forward-facing user interfaces are no longer readily accessible.

[0004] Furthermore, user interfaces in the form of displays are known, located within the passenger compartment, for example, on the doors, windows, or table surfaces. While these displays are accessible even if the driver changes position, they are fixed in place and only allow for a strictly location- and display-dependent presentation and operation. The degrees of freedom regarding the display of the controls are therefore limited, as a freely selectable arrangement of the user interface or one adapted to the operator's position is not possible.

[0005] In DE 10 2016 210 213 A1, a user interface, a means of locomotion, and a method for interaction between the means of locomotion and an occupant of the means of locomotion are proposed. In particular, with regard to the interaction, it is disclosed that haptically perceptible ultrasound pulses are to be generated by means of several ultrasound transducers, the modulated individual signals of which constructively superimpose on a surface of a virtual object, such that at least in the area of ​​an occupant's body surface, a surface of the virtual object becomes haptically perceptible.

[0006] Further proposals for making the surfaces of virtual objects perceptible using ultrasonic transducers are known from the publications "Haptic In-Vehicle Gesture Controls" (ORESTIS GEORGIOU ET AL, PROCEEDINGS OF THE 9TH INTERNATIONAL CONFERENCE ON AUTOMOTIVE USER INTERFACES AND INTERACTIVE VEHICULAR APPLICATIONS ADJUNCT, AUTOMOTIVEUI '17, ISBN 978-1-4503-5151-5, pages 233 - 238) and "Designing stationary airborne ultrasonic 3D tactile object" (INOUE SEKI ET AL, 2014 IEEE / SICE INTERNATIONAL SYMPOSIUM ON SYSTEM INTEGRATION, IEEE, ISBN 978-1-4799-6942-5, pages 159 - 162).

[0007] In contrast, the publication "Three-dimensional display technologies" (JASON GENG, ADVANCES IN OPTICS AND PHOTONICS, (20131122), vol. 5, no. 4, doi:10.1364 / AOP.5.000456, pages 456 - 535) describes how virtual objects can be represented.

[0008] In contrast, US 2018 / 310111 A1 describes how microphones worn by users can be used to track the user's position relative to ultrasound arrays and to calibrate the device for generating perceptible haptic effects.

[0009] Starting from this, the object of the present invention is to create a user interface for a vehicle and a method for configuring and controlling the user interface that is flexible to operate and has more degrees of freedom through a location-independent display.

[0010] This task is initially solved by the user interface according to claim 1.

[0011] According to the invention, the user interface comprises a display device for generating a virtual image of a user interface, wherein the display device includes a 3D image generation system for generating a real image of a user interface and optics that project the real image of the user interface into an interaction space located within the operator's field of vision and arranged within a passenger compartment of the vehicle, an operating unit for operating the virtual user interface with a sensor system for detecting and / or tracking control objects, and a signaling device configured to generate perceptible signals depending on the operation of the user interface, wherein the signaling device is a device for generating a haptically perceptible area in the interaction space of the user interface.wherein the device for generating a haptically perceptible area within the interaction space comprises a plurality of ultrasonic loudspeakers whose ultrasonic signals are tunable with respect to phase, frequency and amplitude, wherein the ultrasonic loudspeakers individually or in groups form several modules which are arranged on the interaction space such that , a) at least one ultrasonic loudspeaker of a module is oriented at an angle to at least one ultrasonic loudspeaker of another module and / or b) at least one ultrasonic loudspeaker of a module is spaced apart from a common plane of other ultrasonic loudspeakers, wherein The device comprises a control unit with software and a programming interface for configuring the device depending on the individual positions and orientations of the modules, wherein several ultrasonic microphones are arranged directly on the modules, wherein the user interface is configured to perform self-calibration, wherein, for the configuration of the device, the modules successively emit defined test signals which are registered by the ultrasonic microphones immediately and / or after reflection from one or more reflective bodies, so that the relative positions and orientations of the modules and ultrasonic microphones can be determined from the transit times and / or the intensities of the registered test signals.

[0012] This proposes a flexible user interface located within the passenger compartment of a vehicle, projecting a virtual representation of the user interface into an interaction space situated within the passenger compartment. This allows the control unit to be easily accessed and operated from different positions, enabling location-independent display and operation that adapts to the operator's position.

[0013] Preferred embodiments of the present invention, in particular of the individual components thereof, are specified below and in the dependent claims.

[0014] In a first preferred embodiment, the 3D image generation system is an autostereoscopic system, in particular a 3D display with lenticular lenses. Alternatively, the 3D image generation system can also be a light field display, a volume display, or a 3D display with a parallax barrier. The optics of the display device create a surface-detached and floating perception of the display and can be designed as a multi-mirror plate, in particular as a multi-mirror plate with dihedral mirrors.

[0015] To prevent unwanted light reflections, the user interface includes at least one optical filter element, which may be designed as a polarizing filter. Specifically, the display device is provided with such an optical filter element that prevents unwanted reflections from the system, such as those caused by the display device itself. The optics also preferably include an optical filter element that prevents unwanted light reflections from external sources, such as sunlight or lamps inside and / or outside the passenger compartment.

[0016] According to a further preferred embodiment of the invention, the display device includes a sensor system for detecting the operator's facial orientation or viewing direction towards the virtually displayed user interface, which can, for example, be camera-based. This allows the virtual user interface to be viewed from different angles.

[0017] To enable operation of the user interface holographically displayed by the display device, a sensor system is provided according to a preferred embodiment of the invention. This sensor system allows for the precise detection and tracking of control objects within the interaction space. Depending on the application, control objects can be defined as, for example, hands, fingers, or even faces. According to a preferred embodiment of the invention, the sensor system comprises at least one time-of-flight sensor and / or at least one stereoscopic camera system, preferably operating in the infrared light spectrum. Alternatively and / or additionally, the sensor system can also include an acoustic camera with ultrasonic loudspeakers and ultrasonic microphones, which will be discussed in more detail later.In particular, combining different sensors within the sensor system has proven advantageous in order to capture a wider range of movement of the control objects through appropriate sensor distribution. Using only a single sensor would otherwise result in obscured areas, which could lead to misinterpretations of the user interface. By integrating such a sensor system to detect and / or track the control objects in the interaction space, hand-based operation can be performed directly with the holographically displayed control unit. The user interface is therefore surface-independent, meaning that no direct contact with the user interface of the components is required.

[0018] The user interface has a signaling device that indicates any interaction with the virtual user interface. In this context, the signaling device is designed to create a haptically perceptible area within the interaction space of the user interface.

[0019] It is provided that the device for generating a haptically perceptible area within the interaction space comprises a plurality of ultrasonic loudspeakers whose ultrasonic signals can be tuned for this purpose with respect to phase, frequency and amplitude, wherein the ultrasonic loudspeakers individually or in groups form several modules that are arranged on the interaction space in such a way that a) at least one ultrasonic loudspeaker of a module is oriented at an angle to at least one ultrasonic loudspeaker of another module and / or b) at least one ultrasonic loudspeaker of a module is spaced away from a common plane of other ultrasonic loudspeakers.

[0020] The positionability, and thus the freely selectable position, of the individual modules with ultrasonic loudspeakers allows the device to be optimally adapted to the available installation space, thereby avoiding collisions between the preferred positions of the image-generating and touch-generating devices. The arbitrary, and in particular angular, arrangement of individual loudspeakers and / or modules relative to each other enables optimal three-dimensional alignment of the modules in space. These modules can be positioned at any location outside the interaction area and oriented as desired. This allows for a customized design of the device for generating haptic feedback while making optimal use of the available installation space.Furthermore, the device, with its essentially freely positionable modules, has a minimal space requirement, and in particular, the adjustability of the orientation of individual modules allows for a high degree of customizability of the strength of the haptic effect.

[0021] In a further preferred embodiment, the ultrasonic loudspeakers of a module are arranged on a common module plane. Preferably, the ultrasonic loudspeakers of a module are oriented parallel to each other. While the specific design of such a module, particularly with regard to the number and arrangement of the ultrasonic loudspeakers, is not predetermined and is essentially arbitrary, a matrix arrangement of the ultrasonic loudspeakers has proven advantageous in practice. In particular, square matrix arrangements of, for example, 4x4 ultrasonic loudspeakers are preferred.

[0022] According to an advantageous embodiment of the invention, the device has at least three modules whose module planes are aligned at angles to each other. This allows the modules to be arranged around the interaction space in an optimal manner, taking into account the available installation space, and to be aligned with the interaction space.

[0023] To configure the device based on the individual position and orientation of the modules and the ultrasonic loudspeakers mounted on them, a control unit with integrated software (firmware) and an application programming interface (API) is provided. The control unit calculates the individual ultrasonic signals based on their phase, frequency, and amplitude, depending on the individual module positions and orientations, in order to generate an area with haptic feedback at any point in the interaction space. If such an area needs to be generated at multiple locations, suitable groups are formed from the available modules, assigned to a specific point, and generating the haptic feedback at that point.Because the haptic area is generated from three different directions with the present three-dimensional arrangement of the modules, the haptic area can be felt even with different hand orientations, and any shadow areas within the interaction space can be effectively avoided.

[0024] The entire device, and therefore its individual modules, can be controlled by the control unit, allowing for the use of simple and thus inexpensive modules. To ensure accurate timing of all modules, all cable lengths and signal propagation times must be known or identical.

[0025] Alternatively, each module is provided with a separate microcontroller for configuring and / or controlling the module's ultrasonic loudspeakers and for communicating with the control unit. This results in optimal networking of the modules with the control unit, which serves as the main control unit, and enables communication via a real-time bus system.

[0026] Several ultrasonic microphones are provided, arranged directly on the modules. These microphones enable simple, automatic configuration of the device and can also be used as a three-dimensional acoustic camera. This allows objects within the interaction space to be detected, identified, and tracked through continuous position determination. This eliminates the need for a separate image acquisition device, such as a stereo camera. This will be discussed in more detail in the description of the method according to the invention.

[0027] The described device for generating haptically perceptible feedback can be integrated into different vehicle models with different interior dimensions due to the flexible arrangement of the ultrasonic loudspeakers and the automatic configurability.

[0028] As an alternative to the described device for generating a haptically perceptible area in the interaction space of the user interface, the signaling device can also be configured to generate acoustic and / or visual feedback when the virtual user interface is operated. In the simplest case, acoustic feedback is provided by a sound emitted from a loudspeaker or similar device. Visual feedback, on the other hand, can be implemented in the simplest case by a signal light located directly at the user interface. It is also provided that, when the holographically displayed user interface is operated, the user interface itself changes its appearance and, for example, indicates the operation by changing the color of the user interface and / or by briefly illuminating it.

[0029] As already stated at the outset, the present invention further relates to a method for configuring and controlling a user interface, the signaling device of which comprises a plurality of ultrasonic loudspeakers for generating a haptically perceptible area within the interaction space.

[0030] According to the invention, the modules sequentially emit defined test signals to configure the device. These signals are registered immediately and / or after reflection from one or more reflectors by ultrasonic microphones arranged directly on the modules. The relative positions and orientations of the modules and ultrasonic microphones are then determined from the transit times and / or intensities of the registered test signals. Taking into account the positions and orientations of the modules and the ultrasonic microphones thus determined and known, the ultrasonic signals of the modules are advantageously tuned with respect to their phase, frequency, and amplitude so that haptically perceptible areas can be generated at any point within the interaction space.The ultrasonic microphones enable self-calibration of the device, simplifying its configuration, which can be performed essentially automatically without manual input. Accuracy and resolution of the configuration increase with an increasing number of modules and / or ultrasonic microphones.

[0031] In addition to configuring the device, the ultrasonic microphones can also be used as an acoustic camera, enabling objects within the interaction space to be detected, identified, and tracked through continuous position determination. Thus, the acoustic camera can also be part of the aforementioned sensor system. For this purpose, it is preferably provided that reflections from stationary or moving objects within the interaction space—such as a hand reaching into the interaction space—are registered by the ultrasonic microphones, and the object's position, shape, and movement are determined from the reflected signals. The ultrasonic frequencies emitted by the ultrasonic loudspeakers to generate the haptic effect can be used for object detection and tracking within the interaction space.Alternatively, other frequencies that differ from this can also be used.

[0032] Finally, a preferred embodiment of the method provides for the identification of objects using existing classification algorithms that compare the reflected signals with existing object-related datasets and, if a match is found, allow for the unambiguous assignment of the object. This also enables specific and targeted excitation of the object within the interaction space.

[0033] Specific embodiments of the present invention are explained below with reference to the figures. These show: Fig. 1 a user interface and Fig. 2 a device for generating a haptically perceptible area within an interaction space.

[0034] In Fig. 1A user interface 1 is depicted within a (not shown) passenger compartment of a vehicle. The user interface 1 has a display device 2 with which a virtual image 3 of a user interface can be projected into an interaction space 4, where the user interface is holographically recognizable by an operator 5. For this purpose, the virtual image is projected into the interaction space 4 via an optically deflecting component 6, in particular a multi-mirror plate.

[0035] To avoid internal reflections within the user interface 1, the display device 2 has a polarization filter 7 which covers the display device 2.

[0036] Similarly, the optical component 6 shown also has a polarization filter 8 that prevents unwanted external reflections, which are caused in particular by sunlight and / or external light sources.

[0037] Optionally, the user interface 1 has an adjustment mechanism 9 that manually or automatically adjusts the display device 2 and / or the optically deflecting components 6 in their angle as well as in their position and orientation to the operator 5.

[0038] The user interface 1 also has a control unit for operating the virtual user interface, which includes a sensor system 10 for detecting and / or tracking control objects 11. In the illustrated embodiment, the sensor system 10 is arranged below the interaction space 4 and detects a hand moving within the interaction space 4, which in this case constitutes the control object 11. Through this object detection and tracking, the user interface 1 recognizes whether and, if so, which function is to be operated via the user interface 1 and forwards the corresponding commands.

[0039] Furthermore, the user interface 1 has an additional sensor system 12 for detecting the facial orientation or object orientation of the operator 5 in relation to the holographically displayed user interface, which in the illustrated embodiment can be camera-based. In particular, by means of a corresponding link with the adjustment mechanism 9, the virtual user interface can be viewed and operated from different angles.

[0040] To signal successful operation to operator 5, various options are provided. In addition to the previously mentioned light signals, the user interface 1 – as described in Fig. 1 The device 21 shown comprises a device for generating haptic feedback, which for this purpose includes a plurality of ultrasonic loudspeakers. This device 21 is shown only as an example in Fig. 1below interaction space 4. In principle, the ultrasonic loudspeakers can be arranged and oriented virtually anywhere within the passenger compartment of the vehicle. A specific embodiment of this, as well as the corresponding configuration and control procedure, is described with reference to Fig. 2 explained.

[0041] The illustrated device 21 has several cylindrical ultrasonic loudspeakers 22, 22', which in the present embodiment form a total of 25 modules 23, 23', each containing 16 ultrasonic loudspeakers 22, 22'. The illustrated device has a total of 400 ultrasonic loudspeakers 22, 22'. The ultrasonic loudspeakers 22, 22' of the individual modules 23, 23' are arranged in a matrix of 4x4 ultrasonic loudspeakers 22, 22' on a common module plate 24 and are aligned parallel to each other. The module plates 24 are planar and form module planes. The modules 23, 23' are arranged in a ring shape and at an angle to a horizontal plane, such that at least one ultrasonic loudspeaker 22 of one module 23 is oriented at an angle to at least one ultrasonic loudspeaker 22' of another module 23' and is spaced apart from a common plane of other ultrasonic loudspeakers.This means that the modules 23, 23' and the ultrasonic loudspeakers 22, 22' can be positioned essentially freely and, in the illustrated embodiment, are aligned towards the center of the interaction space 4 arranged above them.

[0042] For the configuration and control of the device 21, the modules 23, 23' have at least partially ultrasonic microphones 25, with only three ultrasonic microphones 25 being shown as an example in the present embodiment. To increase the accuracy of the configuration and control method, preferably all modules 23, 23' have at least one ultrasonic microphone 25. For configuration, the ultrasonic microphones 25 receive test signals that are emitted sequentially by the modules 23, 23' and which may be reflected by a reflecting body (not shown). The positions and orientations of the individual modules 23, 23' can be uniquely determined from the received signals, in particular their frequencies and amplitudes, so that the ultrasonic signals can be adjusted with respect to their phase, frequency, and amplitude to generate points with haptically perceptible areas at any position within the interaction space.In the figure, the interaction space 4 with possible haptic feedback is essentially depicted as semi-spherical, whereby the shape of the interaction space 4 also depends on the specific and freely selectable arrangement of the modules 23, 23'.

[0043] Within the interaction space 4 itself, haptically perceptible areas can be created at any position, which can be perceived by a person with a hand 11 entering the interaction space 4. The reflections of the ultrasound signals from the hand 11 – or another object – are registered by the ultrasound microphones 25, so that the movement of the hand 11 within the interaction space 4 can be determined from the reflected signals.

[0044] A control unit 28 and a programming interface 29 are provided for controlling and configuring the device 21. In the illustrated embodiment, the control unit 28 is connected to the device 21 such that the control signals are sent directly from the control unit 28 to the individual modules 23, 23'. Alternatively, the modules 23, 23' can also have separate microcontrollers (not shown) which are connected to the modules 23, 23' on the one hand and to the control unit 28 on the other.

[0045] To control and coordinate all components of the present user interface 1, the user interface has an additional computing unit 13 (see Fig. 1 ), which is connected to all components to be controlled. Reference symbol list

[0046] 1 User interface 2 Display device 3 Virtual representation of a user interface 4 Interaction space 5 Operator 6 Optically deflecting component 7 Polarizing filter 8 Polarizing filter 9 Adjustment mechanism 10 Sensor system 11 Control object 12 Sensor system 13 Computing unit 21 Device for generating haptic feedback 22, 22' Ultrasonic loudspeaker 23, 23' Module 24 Module plate 25 Ultrasonic microphones 28 Control unit 29 Programming interface

Claims

1. User interface of a vehicle, comprising a) a display device (2) for generating a virtual image (3) of an operation interface, wherein the display device (2) has a 3D-image generation system for generating a real image of an operation interface, and optics (6) that project the real image of the operation interface into an interaction space (4) located in the field of vision of an operator (5) and arranged within a passenger compartment of the vehicle, and b) an operation unit for operating the virtual operation interface, comprising a sensor system (10) for detecting and / or tracking operation objects (11), and c) a signaling device set up to generate perceptible signals on the basis of the operation of the operation interface, wherein the signaling device is a device (21) for generating a haptically perceptible region in the interaction space (4) of the operation interface, wherein in order to generate a haptically perceptible region within the interaction space (4) the device (21) has a plurality of ultrasonic loudspeakers (22, 22'), the ultrasonic signals of which are tunable for this purpose with respect to phase, frequency and amplitude, wherein the ultrasonic loudspeakers (22, 22') individually or in groups form a plurality of modules (23, 23') which are arranged at the interaction space (4) in such a way that 1. at least one ultrasonic loudspeaker (22) of a module (23) is oriented at an angle to at least one ultrasonic loudspeaker (22') of another module (23') and / or 2. at least one ultrasonic loudspeaker (22) of a module (23) is spaced apart from a plane common to other ultrasonic loudspeakers (22'), wherein d) the device (21) has a control unit (28) with software contained therein and with a programming interface for configuring the device (21) on the basis of the individual positions and orientations of the modules (23, 23'), wherein a plurality of ultrasonic microphones (25) are each arranged directly on the modules (23, 23'), wherein e) the user interface is set up to perform a self-calibration operation, wherein in order to configure the device (21) the modules successively emit defined test signals which are registered by the ultrasonic microphones (25) immediately and / or after reflection on one or more reflection bodies, with the result that the relative positions and orientations of the modules (23, 23') and ultrasonic microphones (25) are determined from the transit times and / or the intensities of the registered test signals.

2. User interface according to claim 2, characterized in that the 3D-image generation system is an autostereoscopic system, in particular a 3D display with lenticular lenses.

3. User interface according to either of claims 1 and 2, characterized in that the optics (6) are a multi-mirror panel.

4. User interface according to any of claims 1 to 3, characterized in that in order to avoid reflections the user interface (1) has at least one optical filter element, in particular at least one polarization filter (7, 8).

5. User interface according to claim 4, characterized in that the display device (2) and / or the optics (6) of the user interface (1) each have at least one optical filter element (7, 8).

6. User interface according to any of claims 1 to 5, characterized in that the sensor system (10) has a time-of-flight sensor and / or a stereoscopic camera system.

7. User interface according to any of the preceding claims, characterized in that the ultrasonic loudspeakers (22, 22') of a module (23, 23') are arranged on a common module plane.

8. User interface according to any of the preceding claims, characterized in that the ultrasonic loudspeakers (22, 22') of a module (23, 23') have a mutually parallel orientation.

9. User interface according to any of the preceding claims, characterized in that the ultrasonic loudspeakers (22, 22') of a module (23, 23') are arranged as a matrix, in particular as a square matrix.

10. User interface according to any of the preceding claims, characterized in that the device (21) has at least three modules (23, 23'), the module planes of which are oriented at an angle to each other.

11. User interface according to any of the preceding claims, characterized in that each module (23, 23') has a separate microcontroller for configuring and / or controlling the ultrasonic loudspeakers (22, 22') of a module (23, 23') and for communicating with the control unit (28).

12. User interface according to any of the preceding claims, characterized in that the signaling device is set up to generate acoustic and / or visual feedback during operation of the virtual operation interface.

13. Method for configuring and controlling a user interface according to any of claims 1 to 12, characterized in that in order to configure the device (21) the modules (23, 23') successively emit defined test signals which are registered by ultrasonic microphones (25) immediately and / or after reflection on one or more reflection bodies, which ultrasonic microphones are arranged directly on the modules (23, 23'), with the result that the relative positions and orientations of the modules (23, 23') and ultrasonic microphones (25) are determined from the transit times and / or the intensities of the registered test signals, wherein, for known positions and orientations of the modules (23, 23') and ultrasonic microphones (25), the ultrasonic signals of the modules (23, 23') are tuned with respect to their phase, frequency and amplitude such that haptically perceptible regions can be generated at any point within the interaction space (4).

14. Method according to claim 13, characterized in that reflections from stationary or moving objects within the interaction space (4) are registered by the ultrasonic microphones (25) and the position, the shape and / or the movement of the object is ascertained from the reflected signals.