Multimedia system of a motor vehicle, method for operating a multimedia system, and motor vehicle

Abstract

The invention relates to a multimedia system of a motor vehicle, comprising an audio source with audio content, an acoustic playback assembly, and an optical reproduction assembly, wherein the optical reproduction assembly has a plurality of lighting elements which are spatially distributed in the motor vehicle interior, wherein the respective position of the lighting elements is defined relative to a vehicle coordinates system, wherein a data processor is provided which is configured to extract spatial information from the audio content, wherein the data processor controls the optical reproduction assembly on the basis of the audio content and the spatial information in order to generate spatial light effects in conjunction with the audio content.

Description

[0001] Mercedes-Benz Group AG

[0002] Multimedia system of a motor vehicle, method for operating a multimedia system and motor vehicle

[0003] The text describes a multimedia system of a motor vehicle, a method for operating a multimedia system, and a motor vehicle.

[0004] Multimedia systems for motor vehicles, methods for operating multimedia systems and motor vehicles of the type mentioned above are known in the prior art.

[0005] Listening to audio content, such as music, while traveling in a motor vehicle is one of the most common activities of motor vehicle passengers. This audio content can be in various formats, such as mono, stereo, or surround sound. Surround sound experiences, in particular, are very popular with passengers because they are immersive and make them feel like they are right in the middle of the sound experience.

[0006] Several surround sound formats are known and established, e.g. Dolby Surround or Dolby Atmos.

[0007] A Dolby Surround system uses multiple speakers positioned throughout the room. A typical setup includes front speakers (left, right, and center), surround speakers (left and right), and a subwoofer for low frequencies. A Dolby Surround audio signal is encoded in a special format that contains information about the placement of the sounds. A decoder analyzes this signal and assigns the different sound components to the corresponding speakers.

[0008] Dolby Atmos is an extension of Dolby Surround, designed to deliver an even more immersive and realistic sound experience. It expands the concept of multi-channel audio by incorporating not only horizontal but also vertical sound dimensions. Dolby Atmos utilizes object-based audio technology. Instead of distributing sound across fixed channels, as is the case with traditional surround systems, Dolby Atmos allows sound objects to be placed at any point in three-dimensional space, with each object assigned a vector in its metadata. Each sound object can be positioned and moved independently, enabling precise control of the soundscape.

[0009] A typical Dolby Atmos setup includes not only conventional speakers (front, center, surround) but also ceiling speakers or special Atmos-enabled speakers that project sound upwards. These additional speakers create the vertical dimension of the sound, allowing sounds from above, such as rain or a helicopter, to be reproduced realistically.

[0010] The Dolby Atmos decoder analyzes the audio signal and assigns the sound objects to the corresponding speakers, taking into account the position and movement of the objects in space. This makes it possible to create a dynamic and immersive sound experience that surrounds the listener.

[0011] Dolby Atmos can be adapted to different speaker configurations, making it particularly suitable for the complexities of a motor vehicle speaker system.

[0012] Furthermore, it is known that the effect of music can be enhanced using light effects, a technique used in nightclubs and, to some extent, in home entertainment systems. This can involve controlling one or more light sources, particularly multicolored light sources and / or lasers, etc., in sync with the music, generating corresponding effects depending on the audio signal. In addition to these light sources, content can be generated on screens via a computer program and displayed synchronously with the music playback. This completes the audiovisual presentation, alongside the music-synchronized control of light effects: audio playback, light effects using multiple, even multicolored, light sources, and the display of screen content are all linked via a system for synchronous audiovisual control.The IN 2021 21002766 A describes a system and method for generating enveloping 360-degree surround audio in a seat or chair of a viewer for trouble-free, hands-free, headphone-free immersion of sound, e.g., in virtual reality or a cinema experience, without obstructing the field of vision.

[0013] IN 202341038602 A describes a system and method for automated, real-time audiovisual control of the environment based on musical input. The system comprises an environment control system, including a capture subsystem configured to receive the musical input, a processing subsystem, and an audiovisual room control platform. The room control platform links the musical input with melody frequency cepstrum coefficients (MFCCs).

[0014] CN 1 18269 813 A describes a vehicle control method in which a target lamp group is determined from a multitude of lamps (or groups of lamps) arranged in the vehicle. This target lamp group is associated with a specific sound field position data. This lamp group contains at least one ambient light lamp. Different target lamp groups are provided for different sound field position data. The position of the target lamp group is used as a reference position, and during audio playback, the operating state of adjacent lamp groups is controlled based on changes in the audio data. This creates visual effects that visualize the position and effect of the sound field in the vehicle.

[0015] US 2021 / 0 314 724 A1 discloses an information processing device comprising: a loudspeaker array comprising a plurality of loudspeakers and performing wavefront synthesis using an output of the plurality of loudspeakers; and a presentation unit that presents visual information indicating a state of waves on a wavefront generated in the wavefront synthesis, or presents visual information based on positional information of a virtual sound image generated in a position distinct from the environment of the loudspeaker array in the wavefront synthesis. To date, no systems are known that enable the immersion of spatial audio content consumption through the use of lighting effects in motor vehicles.

[0016] The task therefore is to further develop multimedia systems of motor vehicles, methods for operating multimedia systems and motor vehicles of the type mentioned above in such a way that targeted support of spatial sound audio content in the motor vehicle becomes possible.

[0017] The problem is solved by a multimedia system of a motor vehicle according to claim 1, a method for operating a multimedia system according to dependent claim 6, and a motor vehicle according to dependent claim 10. Further embodiments and developments are the subject of the dependent claims.

[0018] A multimedia system for a motor vehicle is described, comprising an audio source with audio content, an acoustic playback arrangement configured for the spatial reproduction of audio content from the audio source in a motor vehicle interior, and an optical playback arrangement for generating optical effects in the motor vehicle interior in connection with the audio content of the audio source, wherein the optical playback arrangement has a plurality of luminaires that are spatially distributed in the motor vehicle interior, the respective position of the luminaires being defined relative to a motor vehicle coordinate system, and wherein a data processor is provided which is configured to extract spatial information from the audio content.The data processor controls the optical playback arrangement based on the audio content and spatial information to generate spatial lighting effects related to the audio content. Analogous to the assignment of audio content and spatially defined sound sources within the audio content to light elements, the audio content and spatially defined sound sources are assigned to graphic elements in a computer program ("visualizer engine") that outputs an algorithmically generated visual interpretation of the audio content. These graphic elements are then displayed on a screen.

[0019] The multimedia system can be partially integrated into a vehicle's human-machine interface (HMI) and thus form part of the vehicle's entertainment system. The multimedia system can be operated, for example, via a centrally located screen, with user input possible via a touchscreen or dedicated controls in the vehicle interior.

[0020] The audio source can be a local source, such as a CD or other data storage device. Alternatively, the audio source can be located outside the vehicle, for example, with an online music provider (such as a streaming service), where the audio data is stored at least locally in a buffer.

[0021] The acoustic system may include sound processors, amplifiers, and loudspeakers, as well as potentially integrated crossovers. Additionally, one or more microphones may be installed in the vehicle interior to adjust the sound and provide the best possible foundation for surround sound.

[0022] The loudspeakers can be positioned in various locations within the vehicle, for example, in front of and behind the passengers, as well as to the left, right, and center, and at different heights. Some loudspeakers may be positioned above typical passenger head height, while others may be below. Additionally, special speakers, such as subwoofers or tweeters, may be provided to optimize the sound in the passenger cabin.

[0023] Such speaker arrangements make it possible to provide sound to passengers from different directions and heights, thus creating a three-dimensional soundscape in which sound sources can be precisely positioned in space. This allows, for example, Dolby Atmos implementations to be realized in vehicle interiors.

[0024] The lighting elements can also be arranged in various locations within the vehicle, for example, ambient lighting, which typically runs below a window line, with lighting elements both above and below it. The lighting elements can be positioned differently from the respective loudspeakers. In particular, the lighting elements can be area-wide or linear and can be controlled uniformly or not uniformly.

[0025] Depending on the audio content, a variety of different lighting effects can be generated, varying in duration, color, and brightness. For example, a pulsating, warm light could represent the rhythm of a bass, or a brighter, colored light could depict a specific instrument like a guitar.

[0026] By using such lighting elements located near the position of the corresponding audio source in the room, the impression can be created as if a certain overall audiovisual impression is emerging from this direction, which significantly enhances the immersion effect.

[0027] In addition to controlling light sources via audio content, the "Visualizer Engine" computer program can generate a variety of graphic effects synchronized with both audio and lighting playback. The larger the display controlled by the "Visualizer Engine" and the more centrally located it is in the viewer's field of vision, the more intense and emotionally impactful the audiovisual experience will be. When combining lighting effects and display content, the coordination and similarity of light pulses and colors are particularly crucial for the emotional impact of the audiovisual system integration.

[0028] The corresponding loudspeakers, lighting elements, and the display to be controlled are located in a vehicle coordinate system, for example, vectorized with respect to an origin, so that a vector position relative to the origin is known for each sound and light source. Alternatively, it is possible to specify the positions using coordinates, for example, Cartesian coordinates, relative to the origin.

[0029] Furthermore, it is provided that the audio content includes audio data and metadata, wherein the metadata contains the spatial information about the audio data, and the data processor is configured to map the spatial information to the lighting elements, taking into account the respective position of the lighting elements.

[0030] Such metadata can, for example, contain vectors to different audio streams, indicating where in space the sound source should be positioned and, if applicable, in which direction it should move. For instance, a specific location can be selected for a drum kit, while a different location can be selected for a guitar. Combining this with spatial lighting effects can help create the impression for a passenger that a corresponding light source is located where the audio source is positioned.

[0031] In a further embodiment, it is provided that at least one of the lighting elements is arranged behind the passengers.

[0032] If the audio source is located behind the passenger, the passenger will still perceive it through their peripheral senses.

[0033] In a further embodiment, it is provided that the lighting elements have multicolored LEDs, wherein a spatial control algorithm is provided for controlling the multicolored LEDs, and / or at least one screen, wherein a visualization algorithm is provided for generating visualizations on the at least one screen in connection with the audio content.

[0034] The use of multi-colored LEDs allows not only the dynamic adjustment of brightness but also the dynamic adjustment of a color, thus enabling another dimension of display.

[0035] Furthermore, by incorporating a screen using the computer program "Visualizer Engine," a kaleidoscope effect or other effects can be achieved, generated by the visualization algorithm in sync with the music. These effects complement the control of the lighting elements with regard to, for example, pulse duration and color.

[0036] In a further, more advanced embodiment, the position of a media consumer is defined relative to the vehicle's coordinate system, with the data processor configured to map the spatial information onto the lighting elements, taking the media consumer's position into account. The media consumer's position can be determined, for example, by adjusting the relevant seat, and the head position can then be interpolated. In vehicles equipped with passenger monitoring systems, the data from the passenger monitoring system can also be used to directly capture the head position of the media consumer.

[0037] The media consumer can then be positioned at the center of the music's coordinate system, so that both the audio and visual content can be placed relative to the person's position, especially relative to the position of the person's head, within the vehicle.

[0038] In a further, more advanced embodiment, it is envisaged that the data processor uses mapping information with relative positions of the lighting elements to the position of the media-consuming person to control the lighting elements.

[0039] This mapping information can be generated through vector operations involving the relative position of the media consumer and the relative positions of audio sources and lighting elements. This makes it possible to mathematically select and control the appropriate speakers or lighting elements to achieve a specific effect with minimal computational effort.

[0040] A first independent subject matter relates to a method for operating a multimedia system of the type described above, wherein spatial information is extracted from audio content using the data processor and the optical playback arrangement is controlled on the basis of the music data and the spatial information to generate light effects in connection with the audio content.

[0041] In a first further development, it is envisaged that the metadata describes positions and properties of audio objects, whereby the data processor extracts the positions and / or properties and maps them to the lighting elements.

[0042] In a further, more advanced embodiment, it is provided that the data processor determines the relative positions of the lighting elements to at least one position of at least one media-consuming person and maps them to the lighting elements.

[0043] In a further, more advanced embodiment, it is provided that the data processor assigns at least one color to the properties of the audio objects.

[0044] Another independent subject matter relates to a computer program product, comprising a computer-readable storage medium on which instructions are embedded which, when executed by at least one computing unit, cause that at least one computing unit to be equipped to execute the procedure of the aforementioned type.

[0045] The process can be executed on one or more computing units, so that certain process steps are executed on one computing unit and other process steps on at least one other computing unit, whereby calculated data can be transmitted between the computing units if necessary.

[0046] Another independent item concerns a motor vehicle with a multimedia system of the type described above.

[0047] Further advantages, features, and details will become apparent from the following description, in which – possibly with reference to the drawing – at least one embodiment is described in detail. Identical, similar, and / or functionally equivalent parts are identified by the same reference numerals.

[0048] They show schematically:

[0049] Fig. 1 shows a top view of a motor vehicle with a multimedia system;

[0050] Fig. 2 shows the multimedia system from Fig. 1, as well as

[0051] Fig. 3 shows a flowchart of a procedure for operating the

[0052] Multimedia systems. Fig. 1 shows a motor vehicle 2 with a motor vehicle interior 2.1.

[0053] The motor vehicle 2 is equipped with a multimedia system 4, the components of which are shown outlined with dashed lines.

[0054] The multimedia system 4 comprises a loudspeaker arrangement 6 with loudspeakers 6.1 to 6.4 distributed throughout the vehicle. In other specific embodiments, more than four loudspeakers may be provided. In particular, the loudspeakers may be arranged at different heights. Additionally, the loudspeaker arrangement 6 may include subwoofers or other special loudspeakers to create a pleasing sound image and to achieve a 3D sound effect in the vehicle interior 2.1. For this purpose, it is further provided that some of the loudspeakers, here loudspeakers 6.1 and 6.2, are arranged in front of the passengers or in front of the seats in the vehicle interior 2.1, and certain loudspeakers are arranged behind the passengers or behind the corresponding seats. In the specific embodiment shown, loudspeakers 6.3 and 6.4 are arranged behind a vehicle seat 2.2, in this case, a driver's seat.

[0055] In vehicles with more than one row of seats, loudspeakers can also be positioned between the individual rows of seats. Furthermore, loudspeakers can be provided behind the last or rearmost row of seats.

[0056] The multimedia system 4 further comprises an optical display arrangement 8, which includes a plurality of light sources distributed throughout the vehicle interior 2.1. In the following exemplary embodiment, these include colored LEDs, i.e., RGB LEDs 8.1 to 8.4, and a screen 8.5. The RGB LEDs 8.1 to 8.4 are also distributed at the front and rear, as well as on the right and left sides of the vehicle interior 2.1. However, in other specific embodiments, they can also be located in other places, e.g., on a center console along door panels, along a dashboard, in a headliner, in a footwell, and in other effective locations. The RGB LEDs 8.1 to 8.4 can be point light sources or line / area light sources, e.g., light strips or light fields.

[0057] The screen 8.5 can be a screen of a human-machine interface (HMI) of the motor vehicle 2, with which, among other things, the multimedia system 4, but possibly also other functions of the motor vehicle can be operated.

[0058] In the vehicle interior 2.1, a Cartesian coordinate system is defined, the origin O of which is located in a front left corner of the vehicle interior 2.1. In other embodiments, other types of coordinate systems and other origins may be used. In a controller 10 of the multimedia system 4, the relative positions of the components of the loudspeaker arrangement 6 and the optical display arrangement 8 are stored. That is, for each of the loudspeakers 6.1 to 6.4, the RGB LEDs 8.1 to 8.4, and the screen 8.5, corresponding vectors VA1 to VA4, VL1 to VL5 are defined. The vectors can be represented as triplet values ​​in the XYZ coordinate system.

[0059] Furthermore, a coordinate vector VP relating to the position of a person consuming audio content is determined. This can be done in various ways, e.g., by determining the position of the vehicle seat 2.2 and using correlations or interpolations between this position and a typical head position, or, in the case of appropriately provided interior monitoring devices, also by using head positions specifically determined by the interior monitoring device.

[0060] By determining the coordinate vector VP in conjunction with the origin O of the vehicle interior 2.1, it is possible to use the position of the person in question as the origin or center point of a relevant coordinate system with regard to the spatial reproduction of acoustic and optical signals. This means that the corresponding positioning of audio objects in space, as well as the subsequently described use of optical signals, can be carried out relative to the person, which can be represented in the control system 10 by simple vector operations with minimal computational effort.

[0061] Fig. 2 shows the multimedia system 4 in more detail.

[0062] The controller 10 provides an audio source 20, for example a hard drive, which contains at least one audio file 22. The audio file 22 can, for example, be structured according to the Dolby Atmos standard, and the audio file 22 contains both music data 22.1 in multiple channels and metadata 22.2. Each of these channels of music data 22.1 corresponds to a certain object, e.g., an instrument or a sound effect, and the associated metadata 22.2 defines the position of the corresponding sound object relative to the listener, in this case the person sitting in the vehicle seat 2.2, and movements in space relative to the person.

[0063] On the audio side, the corresponding audio file 22 is read out and converted by means of an acoustic decoder 30 using spatial sound processing and forwarded to the loudspeakers 6.1 to 6.4 by means of a multi-channel amplifier 32 in such a way that a three-dimensional sound effect is created around the person.

[0064] On the optical side, a data processor 40 is provided, which also evaluates the audio file 22 according to the music data 22.1 and the metadata 22.2. The data processor 40 assigns different optical properties to the various audio objects, e.g., a distinct color to each channel. Furthermore, depending on the music data 22.1, various variables can be added to the acoustic object, e.g., variable brightness levels that reflect variable volumes and can thus also represent a rhythm, as well as color mutations that can represent pitch, so that high notes from an instrument emit a slightly different color than low notes.

[0065] Furthermore, depending on the position of the acoustic object in the room, one or more of the RGB LEDs 8.1 to 8.4 are activated to create the impression that a light phenomenon occurs where the acoustic object is placed in the room, or at least in the vicinity.

[0066] The corresponding evaluation from the data processor 40 is passed on to a visualization algorithm 42 (previously called "Visualization Engine"), which operates the screen 8.5 and generates, for example, corresponding kaleidoscope effects. On the other hand, the RGB LEDs 8.1 to 8.4 are addressed via a spatial control algorithm 44.

[0067] The acoustic decoder 30, data processor 40, visualization algorithm 42, and spatial control algorithm 44 can be implemented as software components in a computing unit of the controller 10. Fig. 3 shows a flowchart of the process.

[0068] First, the metadata 22.2 is extracted from the audio files 22, and then optical properties are assigned to the audio objects from the metadata 22.2. Next, the audio objects are located according to the metadata 22.2 and positioned relative to the person's position in the vehicle interior 2.1. In a further step, the distance of the audio objects to the lighting elements 8.1–8.5 is calculated, and then those lighting elements with the smallest distance to the audio objects are activated.

[0069] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as a further explanation in the description.

Claims

Mercedes-Benz Group AG Patent claims 1. Multimedia system (4) of a motor vehicle (2), comprising an audio source (20) with audio content (22), an acoustic playback arrangement (6) with loudspeakers (6.1 - 6.4) arranged at various positions in the motor vehicle (2) in front of and behind the passengers, configured for spatial playback of audio content (22) from the audio source (20) in a motor vehicle interior (2.1), and an optical playback arrangement (8) for generating optical effects in the motor vehicle interior (2.1) in connection with the audio content (22) of the audio source (20), wherein the optical playback arrangement (8) comprises a plurality of light elements (8.1 - 8.5) which are spatially distributed in the motor vehicle interior (2.1), wherein the respective position (VL1 - VL5) of the light elements (8.1 - 8.5) and the loudspeakers (6.1 - 6.5) are determined by the optical playback arrangement (8).4) is defined relative to a motor vehicle coordinate system (O, X, Y, Z), wherein a data processor (40) is provided which is configured to extract spatial information (22.2) from the audio content (22), wherein the data processor (40) controls the optical playback arrangement (8) on the basis of the audio content (22) and the spatial information (22.2) to generate spatial lighting effects in connection with the audio content (22), wherein the audio content (22) comprises audio data (22.1) and metadata (22.2), wherein the metadata (22.2) comprises the spatial information relating to the audio data (22), and wherein the data processor (40) is configured to map the spatial information onto the luminaires (8.1 - 8.5) taking into account the respective position (VL1 - VL5) of the luminaires (8.1 - 8.5).

2. Multimedia system (4) according to claim 1, characterized in that the lighting elements comprise multicolored LEDs (8.1 - 8.4), wherein a spatial control algorithm (44) is provided for controlling the multicolored LEDs (8.1 - 8.4), and / or at least one screen (8.5), wherein a Visualization algorithm (42) for generating visualizations on at least one screen (8.5) in connection with the audio content (22).

3. Multimedia system (4) according to one of the preceding claims, characterized in that a position of a media-consuming person (VP) is defined relative to the motor vehicle coordinate system (O, X, Y, Z), wherein the data processor (40) is configured to map the spatial information onto the lighting elements (8.1 - 8.5) taking into account the position of the media-consuming person (VP).

4. Multimedia system (4) according to claim 3, characterized in that the data processor (40) uses mapping information with relative positions of the light elements (8.1 - 8.5) to the position of the media-consuming person (VP) for controlling the light elements (8.1 - 8.5).

5. Method for operating a multimedia system (4) according to one of the preceding claims, characterized in that spatial information (22.2) is extracted from audio content (22) using the data processor (40) and the optical playback arrangement (8) is controlled on the basis of the music data (22.1) and the spatial information (22.2) to generate light effects in connection with the audio content (22).

6. Method according to claim 5, characterized in that the audio content (22) comprises the audio data (22.1) and the metadata (22.2), wherein the metadata (22.2) comprises the spatial information relating to the audio data (22), wherein the metadata (22.2) describes positions and properties of audio objects, wherein the data processor (40) extracts the positions and / or properties and maps them to the light elements (8.1 - 8.5).

7. Method according to claim 5 or 6 in conjunction with claim 3, characterized in that the data processor (40) determines the relative positions (VL1 - VL5) of the light elements (8.1 - 8.5) to the position of the at least one media-consuming person (VP) and maps them onto the light elements (8.1 - 8.5).

8. Method according to one of claims 6 or 7, characterized in that the data processor (40) assigns at least one color to the properties of the audio objects.

9. Motor vehicle with a multimedia system (4) according to one of claims 1 to 4.

Images