3D ultrasound display with personalized feedback

Abstract

The invention relates to a display system for the spatial representation of content in a vehicle, comprising one or more particles (1) and an ultrasonic actuator unit (3) for generating sound waves in the ultrasonic range, as well as a control unit for controlling the ultrasonic actuator unit (3), wherein the control unit is designed to control the ultrasonic actuator unit (3) according to a predetermined display content, so that the particles (1) reproduce the display content under the influence of the sound waves, characterized in that the display system has a sensor unit (5) for detecting a mood of an occupant of the vehicle, and the control unit is designed to adapt a display content depending on the detected mood.

Description

[0001] The invention relates to a display system for the spatial representation of contents in a vehicle.

[0002] In the state of the art, applications are known to lift particles such as balls against gravity using sound pressure in the ultrasonic range and to carry out position control of the particles by targeted control of ultrasonic actuators.

[0003] In this context, US 2017 / 0 171 536 A1 relates to a volumetric display system comprising: a volumetric display unit in which a plurality of voxel particles can be rearranged via acoustic pressure waves to assume controllable positions in three-dimensional ("3D") space to collectively assume a uniform 3D shape; a plurality of acoustic actuators arranged around the volumetric display unit to emit the acoustic pressure waves and establish a 3D sound field pattern within the volumetric display unit to physically manipulate the voxel particles;and a control system coupled to the acoustic actuators to manipulate the 3D sound field pattern, the control system comprising a volumetric imaging module configured to receive 3D image data describing the uniform 3D shape and calculate the 3D sound field pattern that physically arranges the voxel particles in the uniform 3D shape.;

[0004] Particularly in the luxury segment of modern passenger cars, the application of such a volumetric display system is suitable in order to display information not only on two-dimensional displays, which can also only spatially display three-dimensional content to a limited extent by a corresponding projection into the plane of the two-dimensional display.

[0005] The object of the invention is to provide a volumetric display system for a vehicle which can display highly dynamic complex image patterns while interacting with an occupant.

[0006] The invention is based on the features of the independent claim. Advantageous further developments and refinements are the subject of the dependent claims.

[0007] A first aspect of the invention relates to a display system for the spatial representation of content in a vehicle, comprising one or more particles and an ultrasonic actuator unit for generating sound waves in the ultrasonic range and a control unit for controlling the ultrasonic actuator unit, wherein the control unit is designed to control the ultrasonic actuator unit according to a predetermined display content so that the particles reproduce the display content under the influence of the sound waves, characterized in that the display system has a sensor unit for detecting a mood of an occupant of the vehicle and the control unit is designed to adapt a display content depending on the detected mood.

[0008] The particle(s) can have various shapes, in particular spheres. For low mass inertia and the associated ease of movement by the ultrasonic actuator unit, very light particles, for example made of Styrofoam, are preferably used. The ultrasonic actuator unit is controlled by the control unit in such a way that the particle(s) do not just remain suspended in a static position, but exhibit a high movement speed along virtual object edges, with the object edges being calculated from the predetermined display content. It is precisely this high movement speed that creates the impression of a curve rather than a discrete and approximately point-like particle, so that contours of three-dimensional objects can be represented by one or more individually controlled particles.By appropriate dynamic and non-stationary control of the ultrasonic actuator unit, moving display contents can also be shown, for example a moving spatial figure.

[0009] The ultrasonic actuator unit has one or more ultrasonic loudspeakers, preferably at least three, to enable controlled adjustment of the spatial movement of the particle(s). As long as the particle(s) are within the effective range of the ultrasonic loudspeakers, the amplitude and distribution of the ultrasound can be used as a control variable to adjust the desired trajectory of the respective particle.

[0010] An advantageous effect of the invention is that display content in the vehicle can be presented not only on two-dimensional displays, but also spatially, for example, as three-dimensional figures and representations. Furthermore, unlike conventional visualizations and digital assistants, these display contents are not static but react to the vehicle occupants who interact with the vehicle. The behavior of passengers, especially those in the back seat, can be specifically controlled based on their vital functions, e.g., to calm, activate, or relax them.

[0011] Possible use cases of the display system are one or more of the following: - To display topographical features of a selected route: In this way, topographical features along a selected, hidden route can be displayed in three dimensions and the route can be experienced; - To display movement patterns, including abstract movement patterns, to evoke a desired mood, particularly to calm, motivate, or warn in conjunction with biofeedback from the driver or front passenger(s) by, for example, measuring heart rate, eye movement, movement on the seat or conversations in the car through steering wheel sensors, interior camera, weight sensors in the seats or measured volume in the interior; - Objects on the road in front of and below the car can be displayed in three dimensions to make them easier to see, for example animals, etc. - so that they are not endangered if they are under the car, for example to protect themselves from the sun.

[0012] According to an advantageous embodiment, the sensor unit is designed to determine a mood of the occupant on the basis of a detected frequency of changes of gaze and / or a detected sweat on a hand of the occupant and / or a detected volume in the interior of the vehicle, and to assign it to one of a plurality of predetermined possible moods, such as cheerfulness, stress, or the like.

[0013] Movement patterns and shapes are assigned to the respective display content, which can either reflect the detected mood or be diametrically opposed to it. Thus, detected stress can correspond with fast, abrupt, frequently changing movements and patterns, or trigger correspondingly opposite, slowly changing shapes and movements on the display system. This ensures good entertainment for passengers, especially in the back seat, and simultaneously creates a feeling of company for drivers alone in the vehicle. At the same time, it represents a helpful means of controlling the mood in the vehicle, as the display content can influence the behavior of the occupants. Emotions expressed in the display can stimulate the previously existing mood to change in a certain direction.

[0014] According to a further advantageous embodiment, the control unit is designed to adapt movement patterns and / or shapes of the display content consistently with the mood, so that a positive mood results in a positive display content.

[0015] According to a further advantageous embodiment, the control unit is designed to adapt movement patterns and / or shapes of the display content opposite to the mood, so that a positive mood results in a negative display content and so that a negative mood results in a positive display content.

[0016] According to a further advantageous embodiment, the control unit is designed to detect a mood of an occupant detected by means of the sensor unit in response to an adaptation of the display content and to provide it for a learning model for learning a relationship between display content and the subsequent mood.

[0017] The reaction to the spatial visualizations is measured and used as historical data to train a machine learning model, preferably located on a cloud server, and to store a relationship between display content and a mood, in particular whether it is perceived as pleasant or unpleasant, and thus to iteratively select a personalized and context-dependent standard appearance over many journeys, which is likely to be rated as adequate by the occupants.

[0018] According to a further advantageous embodiment, the display system further comprises a light source with a radiation area that at least partially overlaps with an effective area of ​​the ultrasonic actuator unit, wherein the control unit is designed to control the light source in focus and / or radiation direction and / or luminous intensity for illuminating the at least one particle depending on the display content.

[0019] As the image becomes more complex, the light source can be used to specifically and selectively illuminate the moving particle(s) through focus, direction, and / or luminous intensity, so that they are visible to the human eye only at specific times and in specific spatial positions, creating the impression of a three-dimensional moving image. In this functional extension, the respective particle would serve as both a moving medium and a projection surface.

[0020] According to a further advantageous embodiment, the display system further comprises a camera unit with a detection range directed toward at least part of the effective range of the ultrasonic actuator unit, wherein the control unit is designed to perform trajectory control of the at least one moving particle using the camera unit as a feedback unit regarding a current trajectory of the at least one particle. The effective range of the ultrasonic actuator unit is the space in which the particles move under the influence of the ultrasonic actuator unit, since the ultrasonic actuator unit can emit sufficiently effective ultrasonic waves there.

[0021] When using multiple particles, the position of the particles in space is central to the control, which can be adjusted by comparing the camera unit with the individual desired particle position.

[0022] In a wide range of designs, the particles are colored in different shades of gray, allowing the control system to distinguish them using image recognition. Each particle position can be specified in real time in relation to the specified display content. This allows deviations in the individual particle trajectories to be detected in relation to the strength and calibration of the ultrasonic signals and corrected accordingly.

[0023] Controlling at least one particle may be complex and not feasible manually. However, rapid motif changes and complex motif selection can be advantageously achieved with the help of a machine learning model. Complex and / or highly dynamic image patterns can thus also be displayed.

[0024] According to a further advantageous embodiment, the trajectory control comprises a machine learning model, wherein the control unit is designed to adapt parameters of the machine learning model by reinforcement learning during operation of the display system by comparing a trajectory desired according to the display content with an actual trajectory of the at least one particle.

[0025] To enable precise control, the control of the individual particles is trained and successively improved using reinforcement learning, also known as reinforcement learning, using standard formations to be achieved. The resulting model serves as the starting point for implementation in the vehicle. The machine learning model can be an artificial neural network. The above-mentioned color differentiation can also be used for effective use of reinforcement learning when using multiple particles.

[0026] According to a further advantageous embodiment, the at least one particle remains in a storage container of the display system as long as the ultrasonic actuator unit is deactivated, and wherein the control unit is designed to open a closure mechanism upon activation of the ultrasonic actuator unit, so that the at least one particle is released into the effective range of the ultrasonic actuator unit, and wherein the display system has a suction device which sucks the at least one particle into the storage container during the transition from the active ultrasonic actuator unit to the deactivated ultrasonic actuator unit.

[0027] This embodiment is particularly, but not exclusively, suitable for use when the display system is arranged at the height of a seat backrest of a driver's seat or front passenger seat. In this case, the reservoir can be installed in at least one of these seats, and strips can be provided in the frame to house the ultrasonic sensors. The frame can thus be integrated into the seat backrest of the driver's seat and the front passenger seat, and the display content can be displayed between these seat backrests.

[0028] According to a further advantageous embodiment, the ultrasonic actuator unit has ultrasonic loudspeakers which are arranged on a frame, wherein a collecting container is arranged on an underside of the frame, in which container the at least one particle remains as long as the ultrasonic actuator unit is deactivated, wherein the frame has a guide in order to be sunk into a vehicle element as long as the ultrasonic actuator unit is deactivated.

[0029] This embodiment is particularly suitable for use when the display system is installed in a dashboard between the driver and front passenger in front of the windshield. The frame is preferably a cuboid or body of another shape, preferably made of transparent plastic. The frame itself forms a three-dimensional projection surface within which the at least one particle is moved. When the display system is not in operation, the lower part of the frame serves as a collecting container for the at least one particle and, in a preferred embodiment in particular, is retracted into the dashboard, i.e. not visible to the driver, so as not to unnecessarily block the view when the display system is arranged between the driver and front passenger. When activated, it is extended.

[0030] Further advantages, features and details emerge from the following description, in which - if necessary with reference to the drawing - at least one embodiment is described in detail.

[0031] They show: Fig. 1: A display system between the driver's seat and the passenger seat according to an embodiment of the invention. Fig. 2: A dashboard display system according to an embodiment of the invention.

[0032] The representations in the figures are schematic and not to scale.

[0033] Fig. 1 shows a display system for the spatial representation of content in a vehicle. Four opposing strips have ultrasonic loudspeakers of an ultrasonic actuator unit 3, one pair per strip, which are installed in the inner outer side of the driver's seat and front passenger seat. The at least one particle 1 is located, for example, in a vertical tube embedded in the driver's and front passenger seats as a storage container 7 and is held there by a locking mechanism until the display system, in particular the ultrasonic actuator unit 3, is activated. Upon activation, the locking mechanism, in particular a flap, opens and releases the at least one particle 1, which is preferably forced out due to gravity and further preferably due to the pressure exerted on it by other particles 1.At the same time, the ultrasonic waves of the ultrasonic actuator unit 3 are preferably directed towards the outlet of the tubes in order to capture the particles 1 through the sound waves and transport them together to a central starting position between the seats. The locking mechanism remains open until the display system is deactivated again. Upon deactivation, the particles 1 are again brought to the tubes in equal quantities by the sound waves and drawn in by a suction device 9 installed in the seats. The suction device 9 is connected to a wind tunnel to enable air to be released into the footwell of the rear seat during the suction process. Once all particles 1 have returned to the tubes, the locking mechanism is closed and the suction device 9 is deactivated. The display content is shown as follows: The driver or front passenger activates the display system via an input unit in the vehicle.The particles 1 are then released from a compartment in the driver's seat into the center point between the ultrasonic bars, activating all ultrasonic loudspeakers of the ultrasonic actuator unit 3. If no retrievable driver-specific configuration for a display content, i.e. an appearance, is available, a standard configuration for the basic three-dimensional appearance of the display content is automatically selected from a database in the vehicle. Otherwise, the personalized configuration is chosen. The driver can choose from various options via the display if the default setting is not desired. The display assumes the corresponding appearance. Optionally, an additional light source can be used to depict more complex shapes and, in addition to the spatial effect, to use the particles as the moving medium as a projection surface.Using image recognition on image data from a camera unit, deviations from the particle movement and the initialized movements in three-dimensional space are analyzed in real time and used for live calibration of the display system. This analysis and calibration are performed by a machine learning algorithm that, based on increasing and decreasing deviations in the trajectories of the particles 1, corrects the control towards the desired target state, while simultaneously improving the accuracy of the control unit in controlling the ultrasonic actuator unit 3. Analogously to the three-dimensional display content of the display system, a corresponding graphic is displayed on a screen with a flat display surface visible to the driver, so that this experience is perceptible for both the driver and passenger.In addition to the digital assistant's visualization, specific shapes and images can also be selected from a database and displayed on the display system. Based on the occupant's mood, which is determined by sensor unit 5 and recorded, for example, by the frequency of changes in gaze, sweat on the hands, noise level in the interior, etc., the control unit adjusts the dynamics of the display content. The improved model, training data, and driver feedback are stored on a cloud server for future journeys. Improved machine learning models are loaded into the vehicle via wireless updates to continuously improve display control.

[0034] Fig. 2 an alternative arrangement of the display system compared to that of the Fig.1. The ultrasonic actuator unit 3 has ultrasonic loudspeakers arranged on a frame 11. A collecting container is arranged on the underside of the frame 11, in which container the at least one particle 1 remains as long as the ultrasonic actuator unit 3 is deactivated. The frame 11 has a guide so that it can be sunk into a vehicle element as long as the ultrasonic actuator unit 3 is deactivated. The display system is installed in the dashboard in the middle in front of the windshield between the driver and front passenger. The ultrasonic loudspeakers are arranged in the edges of a frame 11 made of transparent plastic. The frame 11 itself forms a three-dimensional projection surface within which the particles 1 are moved.When the display system is deactivated, the lower part of frame 11 serves as a container for particles 1 and is retracted into the dashboard, thus out of sight of the driver, so as not to unnecessarily obstruct the view. When activated, it retracts.

[0035] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols 1 particle 3 Ultrasonic actuator unit 5 Sensor unit 7 storage containers 9 Suction device 11 frames

Claims

[1] Display system for the spatial representation of contents in a vehicle, comprising one or more particles (1) and an ultrasonic actuator unit (3) for generating sound waves in the ultrasonic range and a control unit for controlling the ultrasonic actuator unit (3), wherein the control unit is designed to control the ultrasonic actuator unit (3) according to a predetermined display content, so that the particles (1) reproduce the display content under the influence of the sound waves, characterized by that the display system has a sensor unit (5) for detecting a mood of an occupant of the vehicle and the control unit is designed to adapt a display content depending on the detected mood. [2] Display system according to claim 1, wherein the sensor unit (5) is designed to determine a mood of the occupant on the basis of a detected frequency of changes of gaze and / or a detected sweat on a hand of the occupant and / or a detected volume in the interior of the vehicle, and to assign one of a plurality of predetermined possible moods. [3] Display system according to one of claims 1 to 2, wherein the control unit is designed to adapt movement patterns and / or shapes of the display content consistent with the mood, so that a positive mood results in a positive display content. [4] Display system according to one of claims 1 to 2, wherein the control unit is designed to adapt movement patterns and / or shapes of the display content opposite to the mood, so that a positive mood results in a negative display content and so that a negative mood results in a positive display content. [5] Display system according to one of the preceding claims, wherein the control unit is designed to detect a mood of an occupant detected by means of the sensor unit (5) in response to an adaptation of the display content and to provide it for a learning model for learning a relationship between display content and subsequent mood. [6] Display system according to one of the preceding claims, further comprising a light source with a radiation area which at least partially overlaps with an effective area of ​​the ultrasonic actuator unit (3), wherein the control unit is designed to control the light source in focus and / or radiation direction and / or luminous intensity for illuminating the at least one particle (1) depending on the display content. [7] Display system according to one of the preceding claims, further comprising a camera unit with a detection range directed towards at least part of an effective range of the ultrasonic actuator unit (3), wherein the control unit is designed to carry out a trajectory control of the at least one moving particle (1) with the camera unit as a feedback unit via a current trajectory of the at least one particle (1). [8] Display system according to claim 7, wherein the trajectory control comprises a machine learning model, wherein the control unit is designed to adapt parameters of the machine learning model by reinforcement learning during operation of the display system by comparing a trajectory desired according to the display content with an actual trajectory of the at least one particle (1). [9] Display system according to one of claims 1 to 8, wherein the at least one particle (1) remains in a storage container (7) of the display system as long as the ultrasonic actuator unit (3) is deactivated, and wherein the control unit is designed to open a closure mechanism upon activation of the ultrasonic actuator unit (3) so that the at least one particle (1) is released into the effective range of the ultrasonic actuator unit (3), and wherein the display system has a suction device (9) which sucks the at least one particle (1) into the storage container (7) during the transition from the active ultrasonic actuator unit (3) to the deactivated ultrasonic actuator unit (3). [10] Display system according to one of claims 1 to 8, wherein the ultrasonic actuator unit (3) has ultrasonic loudspeakers which are arranged on a frame (11), wherein a collecting container is arranged on an underside of the frame (11), in which container the at least one particle (1) remains as long as the ultrasonic actuator unit (3) is deactivated, wherein the frame (11) has a guide in order to be sunk into a vehicle element as long as the ultrasonic actuator unit (3) is deactivated.

Images