3D ultrasound display having personalized feedback

Abstract

The invention relates to a display system for the three-dimensional representation of contents in a vehicle, comprising one or more particles (1) and an ultrasonic actuator unit (3) for generating sound waves in the ultrasonic range, and a control unit for controlling the ultrasonic actuator unit (3), wherein the control unit is designed to control the ultrasonic actuator unit (3) according to a specified display content so that the particles (1) reproduce the display content under the influence of the sound waves, characterized in that the display system comprises a sensor unit (5) for identifying the mood of a passenger in the vehicle, and the control device is designed to adjust the display contents depending on the mood identified.

Description

[0001] 2024P00531WG

[0002] 1

[0003] Mercedes-Benz Group AG

[0004] 3D ultrasound display with personalized feedback

[0005] The invention relates to a display system for the spatial representation of content in a vehicle.

[0006] In the prior art, applications are known to use sound pressure in the ultrasonic range to lift particles such as spheres against gravity and to perform position control of the particles by targeted control of ultrasonic actuators.

[0007] In this context, US 2017 171 536 A1 relates to a volumetric display system comprising: a volumetric display unit in which a multitude of voxel particles can be rearranged via acoustic pressure waves to assume controllable positions in three-dimensional ("3D") space in order to collectively assume a uniform 3D shape; a multitude of acoustic actuators arranged around the volumetric display unit to emit the acoustic pressure waves and establish a 3D sound field pattern within the volumetric display unit in order to physically manipulate the voxel particles;and a control system coupled to the acoustic actuators to manipulate the 3D sound field pattern, wherein the control system includes a volumetric imaging module configured to receive 3D image data describing the uniform 3D shape and to compute the 3D sound field pattern that physically arranges the voxel particles in the uniform 3D shape.

[0008] Particularly in the luxury segment of modern passenger cars, the application of such a volumetric display system is advantageous, as it allows for the presentation of information not only on two-dimensional displays, which can only represent three-dimensional content to a limited extent through projection onto the two-dimensional display plane. The object of the invention is to provide a volumetric display system for a vehicle that can display highly dynamic and complex image patterns in interaction with an occupant.

[0009] The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

[0010] A first aspect of the invention relates to a display system for the spatial representation of content in a vehicle, comprising one or more particles and an ultrasonic actuator unit for generating sound waves in the ultrasonic range, as well as a control unit for controlling the ultrasonic actuator unit, wherein the control unit is configured to control the ultrasonic actuator unit according to a predetermined display content, so that the particles reproduce the display content under the influence of the sound waves, characterized in that the display system comprises a sensor unit for detecting the mood of an occupant of the vehicle and the control unit is configured to adapt a display content depending on the detected mood.

[0011] The particle(s) can have various shapes, particularly spheres. For low inertia and thus easy movement by the ultrasonic actuator unit, very lightweight particles, for example made of polystyrene, are preferably used. The ultrasonic actuator unit is controlled by the control unit in such a way that the particle(s) do not merely remain suspended in a static position, but rather exhibit a high velocity along virtual object edges, which are calculated from the predefined display content. This high velocity creates the impression of a curve instead of a discrete and approximately point-like particle, allowing the contours of three-dimensional objects to be represented by one or more individually controlled particles.By using appropriate dynamic and non-stationary control of the ultrasonic actuator unit, moving display content can also be shown, for example, a moving three-dimensional figure. The ultrasonic actuator unit has one or more ultrasonic loudspeakers, preferably at least three, to allow for controlled spatial movement of the particle(s). As long as the particle(s) are within the effective range of the ultrasonic loudspeakers, the amplitude and distribution of the ultrasound can be used as control variables to set a desired movement path for the respective particle.

[0012] A key advantage of the invention is that display content in the vehicle can be shown not only on two-dimensional displays, but also spatially, for example as three-dimensional figures and representations. Furthermore, unlike conventional visualizations and digital assistants, this display content is not static, but reacts to the vehicle occupant's interaction with the vehicle. The behavior of passengers, particularly those in the back seat, can be specifically controlled based on their vital signs, for example, to calm, activate, or relax them.

[0013] Possible use cases for the display system include one or more of the following:

[0014] - To display topographical features of a selected route: This allows topographical features along a selected, non-visible route to be displayed three-dimensionally and the route to be experienced;

[0015] - To display movement patterns, including abstract movement patterns, to evoke a desired mood, especially for calming, motivation, or warning in conjunction with biofeedback from the driver or passenger(s) by, for example, measuring heart rate, eye movement, movement on the seat or conversations in the car by means of steering wheel sensors, interior camera, weight sensors in the seats or measured volume in the interior;

[0016] - Objects on the road in front of and under the car can be displayed in three dimensions to make them easier to see, for example animals, etc. - to avoid endangering them if they are under the car, for example to protect themselves from the sun.

[0017] According to an advantageous embodiment, the sensor unit is designed to determine the mood of the occupant based on a detected frequency of eye movements and / or detected perspiration on a hand of the occupant and / or a detected volume level in the interior of the vehicle, and to select from a plurality of 2024P00531WG

[0018] to assign 4 predefined possible moods, such as cheerfulness, stress, or similar.

[0019] Each display message is associated with movement patterns and shapes that can either reflect the detected mood or be diametrically opposed to it. For example, detected stress can correspond to rapid, abrupt, and frequently changing movements and patterns, or, conversely, trigger slowly changing shapes and movements on the display system. This provides good entertainment for passengers, especially those in the back seat, while simultaneously creating a sense of companionship for drivers alone in the vehicle. At the same time, it offers a helpful tool for controlling the mood in the vehicle, as the occupants' behavior can be influenced by the displayed content. Strong emotions associated with the content can stimulate a shift in the prevailing mood.

[0020] According to another advantageous embodiment, the control unit is designed to adapt movement patterns and / or shapes of the display content consistently with the mood, so that a positive mood results in positive display content.

[0021] According to a further advantageous embodiment, the control unit is designed to adapt movement patterns and / or shapes of the display content in opposition to the mood, so that a positive mood results in negative display content and so that a negative mood results in positive display content.

[0022] According to a further advantageous embodiment, the control unit is designed to detect an occupant's mood, as detected by the sensor unit, in response to an adjustment of the display content and to provide it to a learning model for teaching a relationship between the display content and the subsequent mood.

[0023] The response to the spatial visualizations is measured and used as historical data to train a machine learning model, preferably located on a cloud server, and to establish a relationship between 2024P00531WG

[0024] 5

[0025] to store display content and mood, especially whether perceived as pleasant or unpleasant, and thus iteratively select a personalized and context-dependent default appearance over many journeys that is expected to be rated as adequate by the occupants.

[0026] According to a further advantageous embodiment, the display system further comprises a light source with a beam area that overlaps at least partially with an effective area of ​​the ultrasonic actuator unit, wherein the control unit is designed to control the light source in focus and / or beam direction and / or luminous intensity for illuminating the at least one particle depending on the display content.

[0027] As image complexity increases, the light source can be used to selectively illuminate the moving particle(s) by adjusting focus, direction, and / or light intensity. This ensures that the particle(s) are only visible to the human eye at specific times and in certain spatial positions, creating the impression of a three-dimensional moving image. In this functional extension, the particle would serve as both the moving medium and the projection surface.

[0028] According to a further advantageous embodiment, the display system further comprises a camera unit with a detection range directed towards at least a portion of the effective range of the ultrasonic actuator unit, wherein the control unit is configured to perform path control of the at least one moving particle with the camera unit as a feedback unit regarding the current path of the at least one particle. The effective range of the ultrasonic actuator unit is the space in which the particles move under the influence of the ultrasonic actuator unit, since the ultrasonic actuator unit can emit sufficiently effective ultrasonic waves into this space.

[0029] When using multiple particles, the position of the particles in space is central for control, which can be adjusted by the camera unit by comparing it with the individually desired particle position.

[0030] In a wide embodiment, the particles are colored in different shades of grey so that they can be distinguished for control via image recognition and 2024P00531WG

[0031] 6. In real time, each particle position can be specified in relation to the predefined display content. This allows deviations of the individual particle trajectories in relation to the strength and calibration of the ultrasound signals to be detected and corrected accordingly.

[0032] Controlling at least one particle can be complex and may not be feasible manually. However, rapid image changes and complex image selection can be advantageously achieved using a machine learning model. Complex and / or highly dynamic image patterns can thus also be displayed.

[0033] According to a further advantageous embodiment, the path control comprises a machine learning model, wherein the control unit is configured to adapt parameters of the machine learning model by reinforcement learning during the operation of the display system by comparing a path desired according to the display content with an actual path of the at least one particle.

[0034] To enable precise control, the control of individual particles via the ultrasonic actuator unit is trained and successively improved using reinforcement learning based on standardized formations. The resulting model serves as the basis for implementation in the vehicle. The machine learning model can be an artificial neural network. The color-coded differentiation mentioned above can also be used for the effective application of reinforcement learning when multiple particles are involved.

[0035] According to a further advantageous embodiment, the at least one particle remains in a reservoir of the display system as long as the ultrasonic actuator unit is deactivated, and wherein the control unit is configured to open a closure mechanism when the ultrasonic actuator unit is activated, so that the at least one particle is released into the effective range of the ultrasonic actuator unit, and wherein the display system has a suction device which, during the transition from the active ultrasonic actuator unit to the deactivated ultrasonic actuator unit, sucks the at least one particle into the reservoir.

[0036] This embodiment is particularly, but not exclusively, suitable for use when the display system is located at the height of a driver's seat backrest or 2024P00531WQ

[0037] 7

[0038] The unit is positioned in the front passenger seat. The reservoir can then be installed in at least one of these seats, and strips can be provided in the frame to house the ultrasonic sensors. The frame can thus be integrated into the seatbacks of both the driver's and passenger's seats, and the display content can be shown between these seatbacks.

[0039] According to a further advantageous embodiment, the ultrasonic actuator unit has ultrasonic loudspeakers which are arranged on a frame, wherein a collection container is arranged on an underside of the frame in which the at least one particle remains as long as the ultrasonic actuator unit is deactivated, wherein the frame has a guide to be lowered into a vehicle element as long as the ultrasonic actuator unit is deactivated.

[0040] This embodiment is particularly suitable for use when the display system is installed in a dashboard between the driver and front passenger in front of the windshield. The frame is preferably a cuboid or a body of other shape, preferably made of transparent plastic. The frame itself forms a three-dimensional projection surface within which the at least one particle is moved. When the display system is inactive, the lower part of the frame serves as a collection container for the at least one particle and, in a preferred embodiment, is retracted into the dashboard, thus not visible to the driver, in order not to unnecessarily obstruct their view when the display system is positioned between the driver and front passenger. When activated, it is extended.

[0041] Further advantages, features and details will become apparent from the following description, in which - possibly with reference to the drawing - at least one embodiment is described in detail.

[0042] They show:

[0043] Fig. 1 : A display system between the driver's seat and the passenger's seat according to an embodiment of the invention.

[0044] Fig. 2: A display system on the dashboard according to an embodiment of the invention. The illustrations in the figures are schematic and not to scale.

[0045] Fig. 1 shows a display system for the spatial representation of content in a vehicle. Four opposing strips have ultrasonic loudspeakers of an ultrasonic actuator unit 3, one pair per strip, which are installed in the inner outer surface of the driver's and passenger's seats. The at least one particle 1 is located, for example, in a vertical tube embedded in the driver's and passenger's seats, serving as a reservoir 7, and is held there by a locking mechanism until the display system, in particular the ultrasonic actuator unit 3, is activated. Upon activation, the locking mechanism, in particular a flap, opens and releases the at least one particle 1, which is preferably forced out due to gravity and, more preferably, due to the pressure exerted on it by other particles 1.Simultaneously, the ultrasonic waves from the ultrasonic actuator unit 3 are preferentially directed towards the outlet of the tubes. These waves capture the particles 1 and transport them together to a central starting position between the seats. The locking mechanism remains open until the display system is deactivated. Upon deactivation, the particles 1 are again drawn towards the tubes in equal quantities by the sound waves and sucked in by a suction device 9 installed in the seats. The suction device 9 is connected to a wind tunnel to allow air to be discharged into the footwell of the rear seat during the suction process. Once all particles 1 have returned to the tubes, the locking mechanism is closed and the suction device 9 is deactivated. The display content is shown as follows: The driver or front passenger activates the display system via an input unit in the vehicle.The particles 1 are then released from a compartment in the driver's seat, activating all ultrasonic loudspeakers of the ultrasonic actuator unit 3, into the center point between the ultrasonic strips. If no driver-specific configuration is available for a display content, i.e., a specific appearance, a standard configuration for the basic three-dimensional appearance of the display content is automatically selected from a database in the vehicle. Otherwise, the personalized configuration is selected. The driver can choose from various options via the display if the standard setting is not desired. The display then adopts the corresponding appearance. Optionally, a light source can be used to display more complex shapes (2024P00531WG).

[0046] 9 and, in addition to the spatial dimension, to use the particles as the moving medium as a projection surface, either additionally or alternatively. Image recognition of image data from a camera unit analyzes deviations between particle movement and the initialized movements in three-dimensional space in real time and uses this information for live calibration of the display system. This analysis and calibration are performed by a machine learning algorithm that, based on increasing and decreasing deviations in the trajectories of the particles 1, corrects the control towards the desired target state, simultaneously improving the accuracy of the control unit when activating the ultrasonic actuator unit 3. Analogously, a corresponding graphic is displayed on a flat screen visible to the driver, mirroring the three-dimensional display content of the display system, so that this experience is accessible to both the driver and passenger.In addition to visualizing the digital assistant, shapes and images can also be specifically selected from a database and displayed on the screen. Based on the occupant's mood, determined by sensor unit 5 (e.g., frequency of eye movements, perspiration on hands, interior noise level), the control unit adjusts the dynamics of the displayed content. The improved model, training data, and driver feedback are stored on a cloud server for future journeys. Enhanced machine learning models are wirelessly updated to continuously improve the display's control.

[0047] Fig. 2 shows an alternative arrangement of the display system compared to that of Fig. 1. Here, the ultrasonic actuator unit 3 has ultrasonic loudspeakers arranged on a frame 11. A collection container is arranged on the underside of the frame 11, in which at least one particle 1 remains as long as the ultrasonic actuator unit 3 is deactivated. The frame 11 has a guide for being recessed into a vehicle component as long as the ultrasonic actuator unit 3 is deactivated. The display system is installed in the dashboard in the center, in front of the windshield, between the driver and front passenger. The ultrasonic loudspeakers are arranged in the edges of a frame 11 made of transparent plastic. The frame 11 itself forms a three-dimensional projection surface within which the particles 1 are moved.When the display system is out of operation, the lower part of the frame 11 serves as a collection container for the particles 1 and is retracted into the dashboard, thus not visible to the driver in order not to unnecessarily obstruct their view. When activated, it is extended.

[0048] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

2024P00531WG 11 Mercedes-Benz Group AG Patent claims 1. Display system for the spatial representation of content in a vehicle, comprising one or more particles (1) and an ultrasonic actuator unit (3) for generating sound waves in the ultrasonic range, and a control unit for controlling the ultrasonic actuator unit (3), wherein the control unit is configured to control the ultrasonic actuator unit (3) according to a predetermined display content, so that the particles (1) reproduce the display content under the influence of the sound waves, characterized in that the display system comprises a sensor unit (5) for detecting the mood of an occupant of the vehicle and the control unit is configured to adapt a display content depending on the detected mood.

2. Display system according to claim 1, wherein the sensor unit (5) is configured to determine the mood of the occupant based on a detected frequency of eye movements and / or detected perspiration on a hand of the occupant and / or detected volume in the interior of the vehicle, and to assign it to one of a multitude of predefined possible moods.

3. Display system according to one of claims 1 to 2, wherein the control unit is configured to adapt movement patterns and / or shapes of the display content consistently with the mood, such that a positive mood results in positive display content.

4. Display system according to one of claims 1 to 2, wherein the control unit is configured to adapt movement patterns and / or shapes of the display content opposite to the mood, so that a positive A positive mood leads to negative ad content, and conversely, a negative mood leads to positive ad content.

5. Display system according to one of the preceding claims, wherein the control unit is configured to detect an occupant's mood as a reaction to an adjustment of the display content by means of the sensor unit (5) and to provide it to a learning model for learning a relationship between display content and subsequent mood.

6. Display system according to one of the preceding claims, further comprising a light source with a beam area which overlaps at least partially with an effective area of ​​the ultrasonic actuator unit (3), wherein the control unit is configured to control the light source in focus and / or beam direction and / or luminous intensity for illuminating the at least one particle (1) depending on the display content.

7. Display system according to one of the preceding claims, further comprising a camera unit with a detection area directed towards at least a part of an effective area of ​​the ultrasonic actuator unit (3), wherein the control unit is configured to perform path control of the at least one moving particle (1) with the camera unit as a feedback unit about a current path of the at least one particle (1).

8. Display system according to claim 7, wherein the path control comprises a machine learning model, wherein the control unit is configured to adapt parameters of the machine learning model by reinforcement learning during the operation of the display system by comparing a path desired according to the display content with an actual path of the at least one particle (1).

9. Display system according to any one of claims 1 to 8, wherein the at least one particle (1) remains in a storage container (7) of the display system as long as the ultrasonic actuator unit (3) is deactivated, and wherein the control unit is configured to open a locking mechanism when the ultrasonic actuator unit (3) is activated, so that the at least a particle (1) is released into the effective area of ​​the ultrasonic actuator unit (3), and wherein the display system has a suction device (9) which, during the transition from the active ultrasonic actuator unit (3) to the deactivated ultrasonic actuator unit (3), sucks the at least one particle (1) into the storage container (7).

10. Display system according to any one of claims 1 to 8, wherein the ultrasonic actuator unit (3) comprises ultrasonic loudspeakers arranged on a frame (11), wherein a collection container is arranged on an underside of the frame (11) in which the at least one particle (1) remains while the ultrasonic actuator unit (3) is deactivated, wherein the frame (11) has a guide for being lowered into a vehicle element while the ultrasonic actuator unit (3) is deactivated.

Images