Immersive audio gradation

Abstract

Attached is an embodiment of immersive audio gradation. In some embodiments, a method includes: receiving object-based audio and metadata; rendering the object-based audio as a multi-channel audio representation for the first speaker layout based on metadata; determining a first mix based on the multi-channel audio representation and a second speaker layout associated with the vehicle; generating a first loudspeaker signal based on the first mixed sound for playback through loudspeakers in a second loudspeaker layout; receiving an input; determining a second mix different from the first mix based on the multi-channel audio representation and the input; and generating a second loudspeaker signal based on the second mixed sound for playback through the loudspeakers in the second loudspeaker layout.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 350,122, filed on June 8, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to audio processing. Background Art

[0004] Most vehicles or other listening environments contain speakers for stereo playback from tapes, CDs, or terrestrial and satellite broadcasts. For example, a car audio system typically includes a total of four speakers (a pair of front speakers and a pair of rear speakers for front and rear passengers). Figure 1 The typical two-row speaker position for the left side of the vehicle is shown. The corresponding speakers are on the right side of the vehicle. If tweeters are used in the front row, they can be on the dashboard or under the front side pillars. In recent years, DVD has introduced multi-channel surround sound. Figure 2 As shown, multi-channel surround sound introduces a center channel speaker and a subwoofer to support the multi-channel format. More recently, streaming services such as Spotify and Tidal have been integrated into automotive infotainment systems, either directly into the vehicle's hardware (often referred to as a "head unit") or through a smartphone using Bluetooth or Apple CarPlay or Android Auto.

[0005] As immersive audio becomes mainstream in theaters and homes, it's natural to assume that it will be integrated into car audio systems as well. For example, Dolby Atmos Music is currently available through various streaming services. Immersive audio is often distinguished from surround sound by including overhead audio content with specific height or vertical characteristics, making the audio content appear to be emanating from above the listener(s) position in the listening environment. To support overhead audio, height speakers are often part of the car audio system.

[0006] Therefore, many speakers, including height speakers, are needed in vehicles or other listening environments to fully support immersive audio formats. High-end vehicles often contain many speakers, beyond the traditional front and rear stereo pairs. Often, they will include height speakers. It is desirable to introduce a gradient into spatial audio playback systems that include many different speaker layouts, including speaker layouts with height channels. Summary of the invention

[0007] For immersive audio content, some techniques for gradually changing audio signals are often cumbersome, inefficient, or ineffective. For example, some existing techniques rely on manual linear panning between stereo speaker pairs. However, without considering the specific spatial cues (e.g., time, amplitude, and frequency processing) incorporated into the production and rendering of immersive audio content, as well as the unique physical speaker layout of the immersive audio system, these techniques cannot facilitate the playback of immersive audio content with accurate and consistent spatialization. Therefore, the existing techniques are ineffective in providing a component for optimizing the playback of immersive content while maintaining artistic intent during playback.

[0008] Thus, the present technology provides electronic devices with more efficient and effective techniques for gradient immersive audio. Such methods optionally supplement or replace other methods for processing or optimizing immersive audio content for playback. Such methods and related interfaces reduce the cognitive burden on users seeking to optimize immersive content playback for corresponding audio systems, and ultimately produce higher quality spatial audio output.

[0009] Attached is an example of an immersive audio transition.

[0010] In some embodiments, a method includes: receiving object-based audio and metadata with at least one processor of an audio system; rendering the object-based audio as a multi-channel audio representation for a first speaker layout based on the metadata with the at least one processor; determining, with the at least one processor, a first mix based on the multi-channel audio representation and a second speaker layout associated with a vehicle; generating, with the at least one processor, a first speaker signal based on the first mix for playback through speakers in the second speaker layout; receiving input with the at least one processor; determining, with the at least one processor, a second mix different from the first mix based on the multi-channel audio representation and the input; and generating, with the at least one processor, a second speaker signal based on the second mix for playback through speakers in the second speaker layout.

[0011] In some embodiments, the multi-channel audio representation includes at least one pair of stereo audio channels.

[0012] In some embodiments, the multi-channel audio representation includes at least one pair of stereo channels and at least one low frequency effects (LFE) channel.

[0013] In some embodiments, the input comprises a fader position of a fader control of the audio system.

[0014] In some embodiments, the input comprises a fader pattern indicating a preset modification to the multi-channel audio representation.

[0015] In some embodiments, the input includes occupancy data indicating the number of occupants in the vehicle and their seating positions within the interior of the vehicle.

[0016] In some embodiments, the vehicle interior is divided into two or more zones, and the second sound mix is ​​determined based at least in part on the two or more zones.

[0017] In some embodiments, the second mix applies a gain to at least one channel of the multi-channel audio representation.

[0018] In some embodiments, the gains are included in a set of gains that map channels in the multi-channel audio representation to speakers in the second speaker layout.

[0019] In some embodiments, the multi-channel audio representation includes more channels than speakers in the second speaker layout.

[0020] In some embodiments, the second speaker layout includes left / right front speakers and left / right rear speakers, and the multi-channel audio representation includes left / right / center speakers, left / right middle speakers, and left / right rear speakers.

[0021] In some embodiments, the second speaker layout further includes at least one of: left / right front height speakers and left / right rear height speakers.

[0022] In some embodiments, the number of channels in the multi-channel audio representation is equal to the number of speakers in the second speaker layout.

[0023] In some embodiments, the second speaker layout and multi-channel audio representation includes a left / right / center speaker, a left / right mid speaker, and a left / right rear speaker.

[0024] In some embodiments, the number of channels in the multi-channel audio representation is less than the number of speakers in the second speaker layout.

[0025] In some embodiments, the multi-channel audio representation includes a front center channel, and the method further includes: generating, with at least one processor, a phantom virtual center from the front center channel; and modifying a predefined spatial position or orientation of at least one speaker in the second speaker layout based on the input and the phantom virtual center.

[0026] In some embodiments, the multi-channel audio representation includes spatial positions or directions for the speakers in a horizontal plane and a height plane.

[0027] In some embodiments, the second mix applies a delay or filtering to the second speaker signal.

[0028] In some embodiments, determining the second mix based on the multi-channel audio representation and the input further comprises transitioning from the first spatialization mode to the second spatialization mode, the transition comprising reallocating portions of the first speaker signal to the second speaker signal.

[0029] In some embodiments, the reallocation is based in part on a distance from a listener or a listening position associated with the listener and at least one speaker associated with the second speaker layout.

[0030] In some embodiments, the redistribution moves a portion of the first speaker signal from at least one speaker in the second speaker layout at a first distance from the listener or listener position to at least one other speaker in the second speaker layout at a second, greater distance from the listener or listener position.

[0031] In some embodiments, the reallocation is performed based on speaker performance characteristics of at least one speaker in the second speaker layout.

[0032] In some embodiments, the reallocating includes reallocating portions of a first speaker signal from a center channel associated with the first speaker layout to two or more second speaker signals for non-center channel speaker channels associated with the second speaker layout.

[0033] In some embodiments, the redistributing includes attenuating one or more portions of the second speaker signal.

[0034] In some embodiments, the reallocating comprises: determining a signal coherence value between audio content in portions of the first speaker signal; and applying an increased attenuation to the second speaker signal in accordance with the coherence value exceeding one or more coherence thresholds.

[0035] In some embodiments, reallocating comprises high pass filtering at least one speaker signal of the second speaker signal corresponding to one or more height channels of the multimedia presentation.

[0036] In some embodiments, the multi-channel audio representation includes at least one pair of height audio channels.

[0037] In some embodiments, an audio playback system includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform any of the aforementioned methods.

[0038] In some embodiments, a non-transitory computer-readable storage medium including instructions thereon, when the instructions are executed by at least one processor, causes the at least one processor to perform any of the aforementioned methods.

[0039] Other embodiments disclosed herein are directed to systems, devices, and computer-readable media. Details of the disclosed embodiments are listed in the accompanying drawings and the following description. Other features, objects, and advantages can be apparent from the description, drawings, and claims.

[0040] Certain disclosed embodiments provide advantages over fading applied to traditional stereo / surround audio systems by applying fading to immersive audio systems including, for example, height speakers. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The diagram shows a typical two-row speaker location for the left side of a vehicle.

[0042] Figure 2 A typical multi-channel car audio system with a center channel speaker and a subwoofer is shown.

[0043] Figure 3A An example of a car with an immersive speaker layout is shown.

[0044] Figure 3B Shown with Figure 3A Same example, but with the speakers labeled with common immersive audio channel names.

[0045] Figure 4 is a conceptual diagram of how fading may be implemented in a 4-channel (front left / right and rear left / right) car audio system in a vehicle with two rows of seats, according to one or more embodiments.

[0046] FIG. 5A to FIG. 5C An example of a gain matrix that is multiplied with input audio to create output audio in a vehicle is shown in accordance with one or more embodiments.

[0047] Figure 6 is a conceptual diagram of how to implement gradual transition for a three-row vehicle according to one or more embodiments.

[0048] Figure 7 is a conceptual diagram of an immersive audio presentation in a vehicle with a stereo speaker pair, according to one or more embodiments.

[0049] Fig. 8A and Figure 8B Example matrix gains are shown for faders set to "center" and "front" in accordance with one or more embodiments.

[0050] Fig. 9 is a conceptual diagram of an immersive audio presentation in a vehicle including a front center speaker and height speakers with spatial playback capabilities in accordance with one or more embodiments.

[0051] Fig. 10AA fader matrix for a "center" setting is shown in accordance with one or more embodiments.

[0052] Fig. 10B A fader matrix is ​​shown for a fader position intended to be intermediate between "center" and "front" in accordance with one or more embodiments.

[0053] Fig. 10C A fader matrix for a "front" fader position is shown in accordance with one or more embodiments.

[0054] Fig. 10D A fader matrix is ​​shown for a fader position intended to be intermediate "center" and "back" in accordance with one or more embodiments.

[0055] Fig. 10E A fader matrix for a "rear" fader position is shown in accordance with one or more embodiments.

[0056] Fig.11 Illustrated is processing of a hub representation channel using a phantom virtual hub (PVC) technique in accordance with one or more embodiments.

[0057] Fig.12 is a flow chart of spatial audio fading in a car audio system according to one or more embodiments.

[0058] Fig.13 is suitable for implementing reference Figures 1 to 12 A block diagram of an example hardware architecture for the described systems and methods.

[0059] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements is shown in the drawings, such as schematic elements representing devices, units, instruction blocks, and data elements. However, it should be understood by those skilled in the art that the specific order or arrangement of the schematic elements in the drawings does not imply that a specific order or sequence of processing or separation processes is required. In addition, the inclusion of schematic elements in the drawings does not imply that such elements are required in all embodiments, or that the features represented by such elements may not be included in or combined with other elements in some embodiments.

[0060] In addition, in the accompanying drawings, where a connecting element, such as a solid line or a dotted line or an arrow, is used to illustrate the connection, relationship or association between two or more other schematic elements, the absence of any such connecting element does not mean that there is no connection, relationship or association. In other words, some connections, relationships or associations between elements in the accompanying drawings are not illustrated to avoid blurring the present disclosure. In addition, for ease of explanation, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, in the case where a connecting element represents the communication of a signal, data or instruction, it will be understood by those skilled in the art that the element represents one or more signal paths as may be required to affect the communication.

[0061] The use of the same reference symbols in the various drawings indicates similar elements. DETAILED DESCRIPTION

[0062] In the following detailed description, many specific details are set forth to provide a comprehensive understanding of the various embodiments described. It will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other instances, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily obscuring various aspects of the embodiments. Some features will be described below, each of which may be used independently of one another or in any combination with other features.

[0063] Although the disclosed embodiments are described below for use in an automotive audio system, the embodiments may also be used in any immersive listening environment where fading is required or desired, or where a predefined multi-channel representation is modified based on user input.

[0064] Glossary

[0065] As used herein, the term "including" and its variations should be understood as open-ended terms, meaning "including but not limited to". Unless the context clearly indicates otherwise, the term "or" should be understood as "and / or". The term "based on" should be understood as "based at least in part on". The terms "an example embodiment" and "example embodiment" should be understood as "at least one example embodiment". The term "another embodiment" should be understood as "at least one other embodiment". The term "determined", determiners and determining should be understood as obtaining, receiving, computing (computing and calculating), estimating, predicting or deriving. In addition, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0066] Example immersive speaker layout

[0067] Figure 3A An example of a passenger car with an immersive speaker layout is shown. Speakers for the left half and center portion of the car are shown. The corresponding speakers are on the right side of the vehicle. For example, as shown, there are bass speakers (e.g., mid-low or sub-bass speakers) or full-range speakers behind the rear seats (e.g., on the rear deck) or embedded in the front door or rear door of the vehicle, and height speakers are placed on or near the ceiling of the vehicle interior, above all other speakers (e.g., on the pillar separating the left and right windows of the vehicle interior); the tweeter and center speaker are embedded in the dashboard of the vehicle or low on the front windshield pillar. Figure 3B Shown with Figure 3A Same example, but with the speakers labeled with common immersive audio channel names. Other speaker layouts are possible.

[0068] Gradient

[0069] For many years, car audio systems have had the ability to "fade" or "pan" the audio to the front or rear of the vehicle as well as the left and right. From the user's perspective, some car audio systems offer variable front-to-back and left-to-right controls. Fading can be accomplished by reducing the level of a specific speaker channel signal. For example, the level of the "rear" speakers can be turned down relative to the level of the "front" speakers.

[0070] Figure 4 is a conceptual diagram of how faders may be implemented in a 4-channel (front left / right and rear left / right) car audio system 400 in a vehicle with two rows of seats, according to one or more embodiments. Stereo audio from any number of sources, including AM / FM radio, CD, MP3 file playback, streaming media, and satellite radio, enters a fader processor 401, which applies a gain matrix to the direct audio to any combination of speakers.

[0071] The multi-channel audio output of the fader processor 401 (multi-channel audio in a stereo pair) passes through a speaker processor 402, which typically applies at least one of equalization, crossover filtering (e.g., for a multi-way speaker with a woofer and a tweeter), speaker protection filtering, or level limiting. The speaker processor 402 outputs the multi-channel audio signal to a speaker amplifier.

[0072] Figures 5A-5C An example of a gain matrix that is multiplied with input audio to create output audio in a vehicle is shown in accordance with one or more embodiments. Figure 5A, when the fader is set to "Front", stereo audio is fed only to the front left and right speaker channels (where "1.0" means 100% linear gain and "0.0" means no linear gain). All other channel gains are set to zero. Figure 5B , when the fader is set to "Center", stereo audio is fed to both rows of speaker channels. Figure 5C , when the faders are set to "Rear", stereo audio is fed only to the rear left and rear right speaker channels. For vehicles with additional rows of loud seats and therefore for speaker left / right pairs, the fading of stereo content can be extended by expanding the matrix to have additional gain values ​​for the additional speaker pairs.

[0073] Figure 6 is a conceptual diagram of how fades may be implemented in a car audio system 600 for a vehicle including additional speakers (e.g., a three-row vehicle or a vehicle with a high trim level audio system) according to one or more embodiments. Stereo audio from any number of sources including AM / FM radio, CD, MP3 file playback, streaming, and satellite radio enters a fader processor 601, which applies a gain matrix to the direct audio to any combination of speakers. The output of the fader processor 601 is multi-channel audio for a stereo pair (front left and right, center left and right, rear left and right) plus an optional low frequency effects (LFE) channel (e.g., a subwoofer speaker channel).

[0074] The multi-channel audio output (multi-channel stereo pair) from the fader process 601 passes through a speaker processor 602, which typically applies at least one of equalization, crossover filtering (e.g., for a multi-way speaker with a woofer and a tweeter), speaker protection filtering, or level limiting. The speaker processor 602 typically uses a low pass filter on the LFE channel to remove midrange and high frequencies. The speaker processor 602 outputs a multi-channel audio signal to a speaker amplifier.

[0075] Gradient Immersive Audio

[0076] Both immersive audio content and vehicles with a large number of speaker channels (including height channels) face the challenge of implementing gradual transitions to create quieter zones in the front or rear while preserving the immersive audio experience for listeners sitting closest to the speakers that are still primarily active. The following disclosure describes methods for gradual transition of immersive audio in automotive audio systems for two categories of speaker layouts. Other categories of speaker layouts may also be used.

[0077] Figure 7is a conceptual flow diagram of an immersive automotive audio system 700 in a vehicle with a stereo speaker pair according to one or more embodiments. Input audio (e.g., spatial audio in the form of audio stems and metadata) is rendered by a spatial audio renderer 701 into a multi-channel audio representation format such as 5.1.2, 5.1.4, 7.1.4, or any other multi-channel audio representation format that currently exists or will exist in the future. The first number, here 7, refers to the number of channels in a horizontal plane around the listener. The second number, here 1, refers to the number of low-frequency effects (LFE) channels. The third number, here 4, refers to the number of height channels. The height channels are intended to provide acoustic frequencies from installed height speakers (see FIG. 1 ) in the listening environment. Figure 3A ) and the goal is to direct the acoustic output (directly or through reflections) from above the intended listening position to the intended listening position. Note that the number of speakers used for playback can be greater than the number of channels (e.g., the left channel consists of a tweeter, a midrange speaker, and a woofer located at different positions) or less than the number of channels (e.g., if the center speaker is missing, the center channel will be panned to the left and right speakers). Representations typically define spatial positions and directions for the speakers in horizontal and height planes relative to the listening position. For example, Figure 3B The 4 height speakers are left front height, right front height, left rear height and right rear height.

[0078] The multi-channel representation is input into a fade processor 702 which generates and outputs the multi-channel audio in a stereo pair plus an LFE channel. The multi-channel audio output by the fade processor 702 is optionally input into a speaker processor 703 which outputs speaker signals to additional processing and / or audio amplifiers for the speakers.

[0079] Fig. 8A and Figure 8B Example matrix gains for faders set to "Center" and "Front" are shown in accordance with one or more embodiments. In these figures, empty cells correspond to a linear gain of 0.0. In both examples, the front center input channel is extended to both the front left and right speakers. Figure 8BIn the example above, the surround and height channels are slightly attenuated when mixed to the left (L) and right (R) front speakers so as not to overwhelm the immersive audio content from the front left, center, and right input channels. Note also that the LFE channel is dropped because the subwoofer (SUB) is typically in the rear of the vehicle (in this example, we are trying to keep it quiet) and the LFE signal may be too low frequency for the capabilities of the front speakers. Alternatively, the LFE can be mixed to a subwoofer that can reproduce the desired low frequencies.

[0080] A higher number of input representation channels, including height channels, means that the fader process has many more input-to-output combinations, and thus a larger matrix. In addition to applying input-to-output gain, it may be beneficial to apply delays and filtering in the fader processors 401, 601, and 702. This will be described in more detail in some of the examples that follow.

[0081] Reference Figure 3A and 3B The vehicle shown in FIG. The speaker layout shown has a (front) center channel speaker, a left / right pair of front, center, and rear height speakers, a left / right pair of front, center, and rear height speakers, and a rear subwoofer speaker.

[0082] Fig. 9 90 is a conceptual flow chart of a vehicle audio system with spatial playback capability including a front center speaker and a height speaker according to one or more embodiments. Spatial audio information including audio and metadata is input into a spatial audio renderer 901, which generates and outputs a multi-channel audio representation (e.g., 7.1.4). The multi-channel representation is input into a fading processor 902, which generates and outputs multi-channel audio in the form of a stereo pair plus a height speaker and an LFE channel. The multi-channel audio is input into a speaker processor 903, which outputs a multi-channel audio signal to a speaker amplifier.

[0083] Fig. 10A A fader matrix for a "center" setting is shown in accordance with one or more embodiments. This configuration may also be referred to herein as "full surround" because there is almost a 1:1 mapping between the input representation audio channels and the output speaker channels.

[0084] Fig. 10BA fader matrix for a fader position or pattern intended to be intermediate between the "center" and "front" positions is shown according to one or more embodiments. This configuration is referred to below as "front surround" because it attempts to provide an immersive audio surround experience for the front row of seats while leaving the second row quiet. The representation channels for front left, front center, front right, and front height are mapped to corresponding output speakers in a 1:1 ratio. The left center representation channel and the right center representation channel are mixed into the front left speaker and the front right speaker and slightly attenuated to reduce interference with the front left channel and the front right channel. The left rear height representation channel and the right rear height representation channel are mapped to the center height output speaker position, and the left rear representation channel and the right rear representation channel are mapped to the left center height speaker position and the right center height speaker position. In addition, since speakers are generally omnidirectional at lower and low frequencies, and mid-range frequencies may be more annoying in the second row of seats (where they are intended to be quiet), in some embodiments, a high-pass filter is included from the left rear to the left center height (and the corresponding right channel). The high-pass filter provides some high-frequency spatial sound (from the rear presentation channels) for the front seats without the annoying mid- and low-frequency sounds.

[0085] Fig. 10C A fader matrix for a "front" fader position is shown in accordance with one or more embodiments. All input representation channels are mapped to either front speaker channels or front height speaker channels. This setup loses the front / rear spatial aspect of immersive sound, but retains the height aspect of immersive sound and provides maximum quietness in the rear of the vehicle.

[0086] Fig. 10D A fader matrix for fader positions intended to be between the "center" and "rear" positions is shown in accordance with one or more embodiments. This configuration may also be referred to herein as "rear surround" because it attempts to provide an immersive audio surround experience for rear seating positions while silencing the front rows.

[0087] Fig. 10E A fader matrix for a "rear" fader position is shown in accordance with one or more embodiments. All input representation channels are mapped to either rear speaker channels or rear height speaker channels. This setup loses the front / rear spatial aspect of immersive sound, but retains the height aspect of immersive sound and provides maximum quietness at the front of the vehicle.

[0088] Other fader positions

[0089] The examples mentioned above show changes in the front / rear fader positions. Additional fader matrices may include, but are not limited to, left / right fades, fades to corner positions (e.g., corresponding to the driver's seat position), and even up or down fades. Furthermore, while the examples show five specific positions, finer control over the fader positions, such as those shown, may be achieved with additional matrices or by interpolating gain values ​​between specific matrices. Fader matrices may also be designed to make specific seating positions quieter. For example, if the driver is on the phone, a "Phone While Driving" matrix may attempt to reduce the entertainment audio level at the driver's seating position.

[0090] Phantom Virtual Center Processing

[0091] Fig.11 A system 1100 for processing center representation channels using a phantom virtual center (PVC) technique according to one or more embodiments is illustrated. Consider again a vehicle with only a stereo pair of speaker channels, such as Figure 1 1101. As shown in FIG. 1102, instead of mixing the center presentation channel into the left and right speakers, in some embodiments, it is beneficial to process the center presentation channel with a PVC processor 1101. The stereo output of the PVC processing is input into a mixing matrix 1102 along with the other presentation channels. The output of the mixing matrix 1102 is a multi-channel audio speaker channel (e.g., door, front center, height channel, and LFE / subwoofer). In some embodiments, other input channel pairs (e.g., left channel pair and right channel pair) that may contain similar content may be processed by additional PVC processor instances and the processor outputs provided to the mixing matrix.

[0092] Occupancy sensing

[0093] In some embodiments, the input includes occupancy data provided by vehicle systems (e.g., seat pressure sensors, interior cameras). The occupancy data may include, but is not limited to, the number and seat positions of the vehicle's occupants. Based on this occupancy data, the multimedia audio representation is modified or replaced with a mix that optimizes the multi-channel audio experience for the listener based on their seat position. For example, if the occupant is sitting on the left side of the vehicle, the mix may be modified based on their listening position to improve the perception of the audio.

[0094] Zone-based immersive audio

[0095] In some embodiments, the vehicle interior is divided into two or more zones, and the mix is ​​determined based at least in part on the two or more zones. The zones may be the front, back, and sides of the vehicle, or divided vertically into multiple planes (e.g., a bottom plane, a horizontal plane, and a height plane). In some embodiments, the vehicle may include a "quiet" zone that receives less audio levels than other parts of the vehicle interior. This can be achieved, for example, by removing / attenuating the LFE speaker or other (multiple) speakers in the quiet zone.

[0096] Example Process

[0097] Fig.12 FIG. 12 is a flowchart of VLBR Ambisonics processing according to one or more embodiments. Process 1200 may be implemented using the electronic device architecture described with reference to FIG. 3 .

[0098] Process 1200 includes: receiving object-based audio and metadata (1201); rendering the object-based audio as a multi-channel audio representation for a first speaker layout based on the metadata (1202); determining a first mix based on the multi-channel audio representation and a second speaker layout associated with a vehicle (1203); generating a first speaker signal based on the first mix for playback through speakers in the second speaker layout (1204); receiving input (1205); determining a second mix different from the first mix based on the multi-channel audio representation and the input (1206); and generating a second speaker signal based on the second mix for playback through speakers in the second speaker layout (1207).

[0099] In some embodiments, the second speaker layout may not correspond to the speaker layout of the vehicle on a speaker-for-speaker basis (e.g., the number of channels / signals associated with the second speaker layout is different than the number of physical speakers in the vehicle audio system). For example, a front left channel / signal of a speaker signal (e.g., a backmix) may be sent to a crossover device that directs low-frequency audio signals (e.g., LFE content) to a woofer, such as in a vehicle door panel, and directs higher-frequency audio signals to a tweeter, such as in an instrument panel, based on a cutoff frequency. In such embodiments, the speaker signals represent a generic set of channels or signals, rather than a specific set of channels or signals, each of which is associated with a corresponding physical speaker.

[0100] In some embodiments, the speaker is processed 703 / 903 (see Figure 7 and Fig. 9 ) before being routed to the loudspeaker, the first loudspeaker signal and / or the second loudspeaker signal may be further processed before and / or after amplification.

[0101] In some embodiments, determining a second mix based on the first mix and the input further includes transitioning from the first spatialization mode to the second spatialization mode, the transition including reallocating a portion of the first speaker signal (e.g., full signal level, attenuated signal level, full bandwidth signal, non-full bandwidth signal) to the second speaker signal.

[0102] In some embodiments, the reallocation is based in part on a distance from the listener or a listening position associated with the listener and at least one speaker associated with the second speaker layout.

[0103] In some embodiments, the reallocation moves content from a speaker at a first distance from a particular listener or listener position to a speaker at a second distance greater from the particular listener or listener position.

[0104] In some embodiments, the reallocation moves a portion of the first speaker signal from at least one speaker in a first speaker layout at a first distance from a listener or listener position to at least one other speaker in a second speaker layout at a second, greater distance from the listener or listener position.

[0105] In some embodiments, the reallocation is performed according to speaker performance characteristics of at least one speaker associated with the second speaker layout.

[0106] In some embodiments, audio content in a multimedia audio presentation is reallocated from a first channel to a second channel when a speaker in a second speaker layout associated with the second channel has a frequency response required to re-produce the reallocated audio content from the first channel (e.g., assigning low-frequency content to a speaker with a sufficiently lower frequency response).

[0107] In some embodiments, the reallocating includes reallocating portions of a first speaker signal from a center channel associated with the first speaker layout to two or more second speaker signals for non-center channel speaker channels associated with the second speaker layout.

[0108] In some embodiments, the redistributing includes attenuating one or more portions of the second speaker signal.

[0109] In some embodiments, the reallocating comprises: determining a signal coherence value between audio content in portions of the first speaker signal; and applying an increased attenuation to the second speaker signal in accordance with the coherence value exceeding one or more coherence thresholds.

[0110] In some embodiments, reallocating comprises high pass filtering at least one speaker signal of the second speaker signal corresponding to one or more height channels of the multimedia presentation.

[0111] In some embodiments, the multi-channel audio representation includes at least one pair of height audio channels.

[0112] Example system architecture

[0113] Fig.13 A block diagram of an example electronic device architecture 1300 suitable for implementing an exemplary embodiment of the present disclosure is shown. Figures 1 to 6 As described, the architecture 1300 includes, but is not limited to, servers and client devices. As shown, the architecture 1300 includes a central processing unit (CPU) 1301, which can perform various processes according to a program stored in, for example, a read-only memory (ROM) 1302 or a program loaded from, for example, a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, data required when the CPU 1301 performs various processes is also stored as needed. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 804. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0114] The following components are connected to the I / O interface 1305: an input unit 1306, which may include a keyboard, a mouse, etc.; an output unit 1307, which may include a display, such as a liquid crystal display (LCD) and one or more speakers; a storage unit 1308, including a hard disk or another suitable storage device; and a communication unit 1309, including a network interface card such as a (e.g., wired or wireless) network card.

[0115] In some embodiments, input unit 1306 includes one or more microphones in different locations (depending on the host device) that are capable of capturing audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0116] In some embodiments, the output unit 1307 includes a system with various numbers of speakers. The output unit 1307 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., single channel, stereo, immersive, binaural, and other suitable formats). In some embodiments, the communication unit 1309 is configured to communicate with other devices (e.g., via a network). The drive 1310 is also connected to the I / O interface 1305 as needed. Removable media 1311, such as a disk, an optical disk, a magneto-optical disk, a flash drive, or other suitable removable media is installed on the drive 1310 so that the computer program read therefrom can be installed into the storage unit 1308 as needed. Those skilled in the art will understand that although the system 1300 is described as including the components described above, in actual applications, it is possible to add, remove, and / or replace some of the components, and all such modifications or changes fall within the scope of the present disclosure.

[0117] According to exemplary embodiments of the present disclosure, the methods described above may be implemented as a computer software program or on a computer-readable storage medium. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing the method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication unit 1309, and / or installed from the removable medium 1311, such as Fig.13 As shown.

[0118] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or dedicated circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the units discussed above may be implemented by a control circuit (e.g., CPU 1301 and Fig.13 Other components of the present disclosure may be combined to perform, so the control circuit can perform the actions described in the present disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor or other computing device (e.g., control circuit). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described in block diagrams, flow charts, or using some other graphical representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0119] In addition, the various blocks shown in the flowcharts may be viewed as method steps, and / or operations generated by operation of computer program code, and / or as multiple coupled logic circuit elements configured to perform the associated (multiple) functions. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the method described above.

[0120] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or connected to an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the above machine-readable media. More specific examples of machine-readable storage media will include an electrical connection with one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above machine-readable storage media.

[0121] The computer program code for executing the method of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device with a control circuit, so that the program code is executed by the processor of the computer or other programmable data processing device, so that the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the computer, partially on the computer as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server or distributed on one or more remote computers and / or servers.

[0122] Although this document contains many specific implementation details, these should not be interpreted as limitations on the scope that can be claimed, but should be interpreted as descriptions of features that can be specific to a specific embodiment. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the above features may be described as acting in certain combinations, even initially required, in some cases, one or more features in the required combination may be deleted from the required combination, and the required combination may be directed to a sub-combination or a variation of the sub-combination. The logic flow depicted in the accompanying drawings does not require the order of the specific order or sequence shown to achieve the desired result. In addition, other steps may be provided or eliminated from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

[0123] Exemplary Embodiments

[0124] EE1. A method comprising: at an electronic device including a user input device and a second subset of speakers of a second speaker type different from a first speaker type (e.g., a height planar speaker, a height speaker, a near-field speaker, etc.), a group of speakers arranged in a device speaker layout within an enclosed volume (e.g., a listening environment with speakers affixed around the enclosed volume, a vehicle cabin, a room, a theater, and a gaming venue, etc.): receiving audio program content as a first set of audio signals, the format of which corresponds to a presentation layout (e.g., a set of audio channel signals corresponding to a predetermined channel layout, such as 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4, and 5.1.2); based on the first set of audio signals, generating a second set of audio signals corresponding to the group of speakers (e.g., a set of speaker audio signals corresponding to the device speaker layout), the second set of audio signals having a gain corresponding to the gain using the first set of gain signals. providing a second group of audio signals or signals derived from the second group of audio signals to a group of speakers so that the group of speakers generates an acoustic output that spatializes audio program content represented by the first group of audio signals; receiving one or more user input sequences corresponding to a request to spatially modify the acoustic output on a user input device; and in response to receiving the one or more user input sequences, transitioning the operation of the electronic device from a first spatialization mode to a second spatialization mode operation by: generating a third group of audio signals corresponding to a group of speakers (e.g., speaker audio signals including modified spatial quality) from the first group of audio signals using a second group of gains that are different from the first group of gains; and providing the third group of audio signals or signals derived from the third group of audio signals to a group of speakers so that the group of speakers generates an acoustic output that spatializes audio program content represented by the first group of audio signals having modified spatial characteristics.

[0125] In some embodiments, the user input device is a touch screen or touch-sensitive surface, physical / mechanical controls (such as knobs, dials, sliders, buttons, rotatable and pressable input devices, voice input devices, etc.), and a group of speakers including a first subset of speakers of a first speaker type (for example, horizontal plane speakers, non-height speakers, far-field speakers, etc.).

[0126] In some embodiments, the electronic device is a vehicle, a media playback device, an infotainment system, a head unit, a multifunction device (eg, a phone, a tablet) coupled to a media playback system, or the like.

[0127] In some embodiments, the presentation layout is different than the device speaker layout.

[0128] In some embodiments, the predetermined speaker layout is the same as the device speaker layout.

[0129] In some embodiments, the first set of gains is a gain matrix corresponding to a default spatialization operating mode or spatialization setting.

[0130] In some embodiments, a set of signals is derived by further processing (eg, applying downmixing or equalization and / or amplifying the resulting signals) to drive an amplifier of a speaker.

[0131] EE2. Any method as disclosed herein, wherein the first speaker type is a non-height speaker type or a horizontal plane speaker type (e.g., a speaker mounted and intended to direct acoustic output from below a height plane speaker or from a speaker below or coplanar to the intended listening position (directly or via reflection) to the intended listening position).

[0132] EE3. Any method as disclosed herein, wherein the second speaker type is a height speaker type or a height plane speaker type (e.g., a speaker mounted and intended to direct acoustic output (directly or via reflection) to an intended listening position from above a horizontal plane speaker or from above the intended listening position).

[0133] EE4. Any method as disclosed herein, wherein the first speaker type is a far-field speaker type.

[0134] EE5. Any method as disclosed herein, wherein the second speaker type is a near-field speaker type.

[0135] EE6. Any method as disclosed herein, wherein a set of speakers includes one or more dedicated low-frequency speakers. In some embodiments, the low-frequency speakers include subwoofers, bass oscillators, force sensors, tactile sensors, or other low-frequency optimized sensors.

[0136] EE7. Any method as disclosed herein, wherein the first set of audio signals comprises one or more pairs of height audio channels.

[0137] EE8. Any method as disclosed herein, wherein the first set of audio signals corresponds to a speaker format selected from the following group: 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4 and 5.1.2.

[0138] EE9. Any method as disclosed herein further includes: before receiving audio program content as a first group of audio signals: receiving object-based audio representing the audio program content, the object-based audio including a set of one or more sound source essence signals having corresponding information (e.g., metadata) indicating spatial characteristics of corresponding sound sources; and rendering the object-based audio into a first group of audio signals in a format corresponding to a representation layout using an object renderer.

[0139] EE10. Any method as disclosed herein, wherein the spatial characteristics include at least one position selected from a set of positions of the corresponding source in three-dimensional space relative to the listener position, a size or level of dispersion of the corresponding source, and a distance of the corresponding source from the listener position. In some embodiments, the object-based audio is a media program, a music or video program, sound data related to the operation of a non-entertainment system (e.g., a security notification), etc. In some embodiments, the object-based audio is stored locally as a complete file or received sequentially via a streaming protocol.

[0140] EE12. Any method as disclosed herein, wherein the second set of gains is generated by the first set of gains and a third set of gains, the third set of gains being different from the first set of gains and different from the second set of gains.

[0141] EE13. Any method as disclosed herein, wherein transitioning the operation of an electronic device from a first spatialization mode to a second spatialization mode includes reallocating portions (e.g., full signal levels, attenuated signal levels, full bandwidth signals, non-full bandwidth signals) of a first group of audio signals from channels or signals associated with a representation layout to non-corresponding channels or a group of signals associated with a device speaker layout.

[0142] EE14. Any method as disclosed herein, wherein the reallocation is based in part on a distance from a listener or a seating position associated with the listener and at least one speaker in a set of speakers in a device speaker layout arranged within the enclosed volume. In some embodiments, the reallocation moves content from a speaker at a first distance from a particular listener or listener position to a speaker at a second, greater distance from the particular listener or listener position.

[0143] EE15. Any method as disclosed herein, wherein the reallocation is performed based on a speaker performance characteristic of at least one speaker in a set of speakers in a device speaker layout arranged within the closed volume. In some embodiments, content is reallocated from a first channel to a second channel when the speaker associated with the second channel has a frequency response necessary to reproduce the reallocated content from the first channel (e.g., low frequency content is only allocated to speakers with sufficiently low frequency response).

[0144] EE16. Any method as disclosed herein, wherein the reallocating comprises reallocating portions of the first group of audio signals from a center channel representing the layout to two or more associated non-center channel speaker channels in the device speaker layout.

[0145] EE17. Any method as disclosed herein, wherein the redistributing comprises attenuating one or more portions of the first group of audio signals.

[0146] EE18. Any method as disclosed herein, wherein the reallocating comprises determining signal coherence values ​​between contents of portions of the first set of audio signals; and applying increased attenuation based on coherence values ​​exceeding one or more coherence thresholds.

[0147] EE19. Any method as disclosed herein, wherein reallocating comprises high pass filtering an audio signal of the first group of audio signals corresponding to one or more height channels representing the layout.

[0148] EE20. Any method as disclosed herein, wherein a third set of audio signals or signals derived from the third set of audio signals are provided to a set of speakers so that the local sound field has a reduced acoustic output (such as sound pressure, perceptually weighted or absolute) at a first listening position in an enclosed volume relative to the acoustic output produced by providing the second set of audio signals or signals derived from the second set of audio signals, while maintaining spatialization (for example, perceived height effects vary with metadata in the source signals) within the local sound field at at least a second listening position in the enclosed volume.

[0149] EE21. Any method as disclosed herein, wherein the second group of audio signals or the third group of audio signals comprises one or more pairs of height audio channels.

[0150] EE22. Any method as disclosed herein, wherein the second set of audio signals or the third set of audio signals corresponds to a speaker format selected from the following group: 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4 and 5.1.2.

[0151] EE23. Any method as disclosed herein, wherein the second set of audio signals or the third set of audio signals corresponds to a speaker format that does not include height channels.

[0152] EE24. Any method as disclosed herein, wherein the enclosed volume includes seats arranged in a first row of listening positions or seating positions.

[0153] EE25. Any method as disclosed herein, wherein the second subset of speakers includes a first height pair of speakers mounted and intended to direct acoustic output (i) from a position in front of the first row and (ii) from a position above the first subset of speakers to a listening position located at a position corresponding to a first row listening position or seating position.

[0154] EE26. Any method as disclosed herein, wherein the enclosed volume includes a second row of listening positions or seating positions located behind the first row of listening positions or seating positions.

[0155] EE27. As in any method disclosed herein, the second subset of speakers includes a second height speaker pair mounted and intended to direct acoustic output (i) from a position behind a first row of listening positions or seating positions and / or (ii) from a position in front of a second row of listening positions or seating positions, (iii) from a position above each speaker of the first subset of speakers; and (iv) toward a listening position corresponding to a position in the second row of listening positions or seating positions.

[0156] EE28. Any method as disclosed herein, wherein the enclosed volume includes a third row of listening positions or seating positions located behind the second row of listening positions or seating positions.

[0157] EE29. As in any method disclosed herein, the second subset of speakers includes a third height speaker pair installed and intended to direct acoustic output (i) from above the first speaker subset and (ii) from behind the second row. In some embodiments, the third height speaker pair is installed at a location on the rear interior deck, C-pillar or D-pillar of the vehicle cabin. In some embodiments, the height speakers are designed to reflect sound downward from the surface to the listener in the seat position. In some embodiments, the height speakers are installed above the first subset of speakers.

[0158] EE30. Any method as disclosed herein, further comprising: after generating the second set of audio signals or after generating the third set of audio signals, and before providing the corresponding audio signals to a set of speakers, processing the corresponding audio signals to compensate for one or more of the following: speaker position, speaker response, absorption and reflection properties of nearby materials within the closed volume, hearing sensitivity of occupants or listeners located within the closed volume. In some embodiments, the processing includes time alignment (e.g., based on distance from one or more listener positions or seat positions), active filtering or passive filtering, etc.

Claims

1. A method comprising: receiving, with at least one processor of the audio system, object-based audio and metadata; rendering, with the at least one processor, the object-based audio into a multi-channel audio representation for a first speaker layout based on the metadata; determining, with the at least one processor, a first audio mix based on the multi-channel audio representation and a second speaker layout associated with a vehicle; generating, with the at least one processor, a first speaker signal based on the first mix for playback through speakers in the second speaker layout; receiving input with the at least one processor; determining, with the at least one processor, a second audio mix different from the first audio mix based on the multi-channel audio representation and the input; as well as A second speaker signal is generated with the at least one processor based on the second mix for playback through the speakers in the second speaker layout.

2. The method of claim 1, wherein: The multi-channel audio representation comprises at least one pair of stereo audio channels.

3. The method according to claim 1 or 2, wherein: The multi-channel audio representation includes at least one pair of stereo channels and at least one low frequency effects (LFE) channel.

4. The method according to any one of claims 1 to 3, wherein: The input comprises a fader position of a fader control of the audio system.

5. The method according to any one of claims 1 to 4, wherein: The input comprises a fader pattern indicative of a preset modification to the multi-channel audio representation.

6. The method according to any one of claims 1 to 5, wherein: The input includes occupancy data indicative of the number of occupants in the vehicle and their seating positions within the interior of the vehicle.

7. The method according to any one of claims 1 to 6, wherein: The vehicle interior is divided into two or more zones, and the second sound mix is ​​determined based at least in part on the two or more zones.

8. The method according to any one of claims 1 to 7, wherein: The second mix applies a gain to at least one channel of the multi-channel audio representation.

9. The method of claim 8, wherein: The gains are included in a set of gains that map channels in the multi-channel audio representation to the speakers in the second speaker layout.

10. The method according to any one of claims 1 to 9, wherein: The multi-channel audio representation includes more channels than speakers in the second speaker layout.

11. The method according to any one of claims 1 to 10, wherein: The second speaker layout includes left / right front speakers and left / right rear speakers, and the multi-channel audio representation includes left / right / center speakers, left / right middle speakers, and left / right rear speakers.

12. The method according to any one of claims 1 to 11, wherein: The second speaker layout also includes at least one of: left / right front height speakers and left / right rear height speakers.

13. The method according to any one of claims 1 to 12, wherein: The number of channels in the multi-channel audio representation is equal to the number of speakers in the second speaker layout.

14. The method according to any one of claims 1 to 13, wherein: The second speaker layout and the multi-channel audio representation include left / right / center speakers, left / right mid speakers, and left / right rear speakers.

15. The method according to any one of claims 1 to 14, wherein: The number of channels in the multi-channel audio representation is less than the number of speakers in the second speaker layout.

16. The method according to any one of claims 1 to 15, wherein: The multi-channel audio representation includes a front center channel, and the method further includes: generating, with the at least one processor, a phantom virtual center from the front center channel; and modifying a predefined spatial position or orientation of at least one speaker in the second speaker layout based on the input and the phantom virtual center.

17. The method according to any one of claims 1 to 16, wherein: The multi-channel audio representation includes spatial positions or directions for the speakers in a horizontal plane and a height plane.

18. The method according to any one of claims 1 to 17, wherein: The second mix applies a delay or a filter to the second loudspeaker signal.

19. The method according to any one of claims 1 to 18, wherein: Determining a second mix based on the multi-channel audio representation and the input further includes transitioning from a first spatialization mode to a second spatialization mode, the transition including reallocating portions of the first loudspeaker signal to the second loudspeaker signal.

20. The method of any one of claims 1 to 19, wherein: The reallocation is based in part on a distance from a listener or a listening position associated with the listener and at least one speaker associated with the second speaker layout.

21. The method of any one of claims 19-20, wherein: The reallocation moves a portion of the first loudspeaker signal from at least one loudspeaker in the second loudspeaker layout at a first distance from the listener or the listener position to at least one other loudspeaker in the second loudspeaker layout at a second, greater distance from the listener or the listener position.

22. The method of any one of claims 19 to 21, wherein: The reallocation is performed based on a speaker performance characteristic of at least one speaker in the second speaker layout.

23. The method of any one of claims 19 to 22, wherein: The reallocating includes reallocating portions of the first speaker signal from a center channel associated with the first speaker layout to two or more second speaker signals for non-center channel speaker channels associated with the second speaker layout.

24. The method of any one of claims 19 to 23, wherein: The reallocating includes attenuating one or more portions of the second loudspeaker signal.

25. The method of any one of claims 19 to 24, wherein: The reallocating includes determining a signal coherence value between audio content in portions of the first speaker signal; and applying an increased attenuation to the second speaker signal in accordance with the coherence value exceeding one or more coherence thresholds.

26. The method of any one of claims 19 to 25, wherein: The reallocating comprises high pass filtering at least one loudspeaker signal of the second loudspeaker signal corresponding to one or more height channels of the multimedia representation.

27. The method of any one of claims 1 to 26, wherein: The multi-channel audio representation includes at least one pair of height audio channels.

28. An audio playback system, comprising: at least one processor; A memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform the method of any one of the preceding claims 1-27.

29. A non-transitory computer-readable storage medium comprising instructions thereon, which when executed by at least one processor, cause the at least one processor to perform the method of any of the preceding claims 1-27.

Images