Real-time audio fading

Abstract

In some embodiments of immersive audio fading, the method includes receiving object-based audio and metadata, rendering the object-based audio into a multi-channel audio presentation for a first loudspeaker layout based on the metadata and a second loudspeaker layout associated with a vehicle, generating a first loudspeaker signal based on a first mix for playback through speakers in the second loudspeaker layout, receiving an input, determining a second mix different from the first mix based on the multi-channel audio presentation and the input, and generating a second loudspeaker signal based on the second mix for playback through speakers in the second loudspeaker layout.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 350,122, filed on June 8, 2022, which is hereby incorporated by reference in its entirety.

[0002] Technical Field of the Invention The present disclosure generally relates to audio processing.

Background Art

[0003] Most vehicles and other listening environments include speakers for stereo playback from tapes, CDs, terrestrial and satellite radio. An automotive audio system generally includes, for example, a total of four loudspeakers (a pair in the front and a pair in the rear, for front and rear passengers). FIG. 1 shows the positions of a typical two - row speaker on the left side of a vehicle. Corresponding loudspeakers are on the right side of the vehicle. When using tweeters in the front row, they may be installed on the dashboard or at the bottom of the front - side pillar. In recent years, DVDs have introduced multi - channel surround sound. As shown in FIG. 2, multi - channel surround sound has introduced a center - channel loudspeaker and a sub - woofer to support multi - channel formats. Recently, streaming services such as Spotify® and Tidal® have been incorporated directly into automotive hardware (usually known as the “head unit”) or via smartphones using Bluetooth®, Apple CarPlay®, or Android Auto®.

[0004] As immersive audio becomes mainstream in cinema and at home, it is natural to assume that immersive audio will also be integrated into automotive audio systems. For example, Dolby Atmos (registered trademark) music is currently available through various streaming services. Immersive audio is often distinguished from surround sound by including overhead audio content that has a specific height or vertical characteristic, such that the audio content appears to emanate from above the listener's position in the listening environment. To support overhead audio, height loudspeakers are often part of automotive audio systems.

[0005] Accordingly, there is a need to fully support immersive audio formats in vehicles or other listening environments having a plurality of loudspeakers including height loudspeakers. High-end vehicles are equipped with many loudspeakers in addition to the conventional front stereo and rear stereo pairs. Often, height loudspeakers are included. It is desirable to introduce fading in a spatial audio reproduction system including many different speaker layouts including a speaker layout having height channels. Summary of the Invention Problems to be Solved by the Invention

[0006] Some techniques for fading an audio signal are generally cumbersome, inefficient, or ineffective for immersive audio content. For example, some existing techniques rely on manual linear panning between a pair of stereo speakers. However, without considering the specific spatial cues (e.g., time, amplitude, and frequency processing) incorporated into the production and rendering of immersive audio content, as well as the unique physical speaker layout of an immersive audio system, such techniques cannot smooth the playback of immersive audio content with accurate and consistent spatialization. Thus, existing techniques are ineffective in providing a means to optimize the playback of immersive content while maintaining artistic intent during playback.

Means for Solving the Problem

[0007] Accordingly, the present technology provides an electronic device with a more efficient and effective technique for fading immersive audio. Such a method optionally complements or replaces other methods for processing or optimizing immersive audio content for playback. Such methods and related interfaces optimize the playback of immersive content for respective audio systems by reducing the user's cognitive burden and ultimately generate higher quality spatial audio output.

[0008] Embodiments of immersive audio fading are enclosed.

[0009] In some embodiments, the method comprises receiving object-based audio and metadata using at least one processor of an audio system; rendering the object-based audio into a multi-channel audio presentation for a first loudspeaker layout based on the metadata using at least one processor; determining, using at least one processor, a first mix based on the multi-channel audio presentation, a second loudspeaker layout associated with a vehicle; generating a first loudspeaker signal based on the first mix for playback through loudspeakers in the second loudspeaker layout using at least one processor; receiving an input using at least one processor; determining, using at least one processor, a second mix different from the first mix based on the multi-channel audio presentation and the input; and generating a second loudspeaker signal based on the second mix for playback through second loudspeakers in the second loudspeaker layout using at least one processor.

[0010] In some embodiments, the multi-channel audio presentation comprises at least one pair of stereo audio channels.

[0011] In some embodiments, the multi-channel audio presentation comprises at least one pair of stereo channels and at least one low frequency effects (LFE) channel.

[0012] In some embodiments, the input comprises a fader position of a fader control of the audio system.

[0013] In some embodiments, the input comprises a fader mode indicating a preset modification to the multi-channel audio presentation.

[0014] In some embodiments, the input includes occupancy data indicating the number of occupants in the vehicle and their seat positions within the vehicle.

[0015] In some embodiments, the interior of the vehicle is divided into two or more zones, and the second mix is determined based at least in part on the two or more zones.

[0016] In some embodiments, the second mix applies a gain to at least one channel of a multi-channel audio presentation.

[0017] In some embodiments, the gain is included in a set of gains that map channels in the multi-channel audio presentation to loudspeakers in the second loudspeaker layout.

[0018] In some embodiments, the multi-channel audio presentation includes more channels than there are loudspeakers in the second loudspeaker layout.

[0019] In some embodiments, the second loudspeaker layout includes left / right front loudspeakers and left / right rear loudspeakers, and the multi-channel audio presentation includes left / right / center loudspeakers, left / right midrange loudspeakers, and left / right rear loudspeakers.

[0020] In some embodiments, the second loudspeaker layout further includes at least one of left / right front height loudspeakers and left / right rear height loudspeakers.

[0021] In some embodiments, the number of channels in the multi-channel audio presentation is equal to the number of loudspeakers in the second loudspeaker layout.

[0022] In some embodiments, the second loudspeaker layout and multi-channel audio presentation include left / right / center loudspeakers, left / right mid loudspeakers, and left / right rear loudspeakers.

[0023] In some embodiments, the number of channels in the multi-channel audio presentation is less than the number of loudspeakers in the second loudspeaker layout.

[0024] In some embodiments, the multi-channel audio presentation includes a front center channel, and the method further includes generating a phantom virtual center from the front center channel using at least one processor, and modifying a predetermined spatial position or direction of at least one loudspeaker within the second loudspeaker layout based on the input and the phantom virtual center.

[0025] In some embodiments, the multi-channel audio presentation includes the spatial position or direction of the loudspeakers in the horizontal plane and the height plane.

[0026] In some embodiments, the second mix applies a delay or filtering to the second loudspeaker signal.

[0027] In some embodiments, the step of determining the second mix based on the multi-channel audio presentation and the input further includes transitioning from a first spatialization mode to a second spatialization mode, the transition including reassigning a portion of the first loudspeaker signal to the second loudspeaker signal.

[0028] In some embodiments, the reassigning is based in part on the distance from the listener or listener position associated with the listener, and at least one loudspeaker associated with the second speaker layout.

[0029] In some embodiments, the reassignment moves a portion of the first loudspeaker signal from at least one loudspeaker in the second speaker layout, at a first distance from the listener or the listener's position, to at least one other loudspeaker in the second speaker layout, separated by a second distance or more from the listener or the listener's position.

[0030] In some embodiments, the reassignment is performed according to the speaker performance characteristics of at least one loudspeaker in the second speaker layout.

[0031] In some embodiments, the reassignment includes reassigning a portion of the first loudspeaker signal from a center channel associated with the first speaker layout to two or more second loudspeaker signals for non-center channel loudspeaker channels associated with the second speaker layout.

[0032] In some embodiments, the reassignment includes attenuating one or more portions of the second loudspeaker signal.

[0033] In some embodiments, the reassignment includes determining a signal coherence value between audio contents in a portion of the first loudspeaker signal and applying increased attenuation to the second loudspeaker signal according to a coherence value exceeding one or more coherence thresholds.

[0034] In some embodiments, the reassignment includes applying a high-pass filter to at least one loudspeaker signal of the second loudspeaker signal corresponding to one or more height channels of the multimedia presentation.

[0035] In some embodiments, the multi-channel audio presentation includes at least one pair of height audio channels.

[0036] In some embodiments, the audio playback system comprises at least one processor and memory-stored instructions that, when executed by the at least one processor, cause the at least one processor to perform any of the above methods.

[0037] In some embodiments, a non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform any of the foregoing methods.

[0038] Other embodiments disclosed herein are directed to systems, apparatus, and computer-readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0039] The particular embodiments disclosed provide advantages over the panning applied to conventional stereo / surround audio systems, for example, by applying panning to an immersive audio system including height speakers.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 11

Figure 12

Figure 13

[0041] In the drawings, for ease of explanation, a particular arrangement or order of schematic elements, such as those representing devices, units, instruction blocks, and data elements, is shown. However, it should be understood by those skilled in the art that a particular ordering or arrangement of schematic elements in the drawings does not imply a particular order or sequence of processing, or that separation of processing is required. Further, including schematic elements in the drawings does not mean that such elements are required in all embodiments, or that features represented by such elements are not included in or combined with other elements in some embodiments.

[0042] Furthermore, in the drawings, when connection elements such as solid lines, dashed lines, or arrows are used to illustrate a connection, relationship, or association between two or more other schematic elements or between them, the absence of such a connection element does not mean that the connection, relationship, or association cannot exist. In other words, some of the connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the disclosure. Further, for ease of explanation, a single connection element is used to represent multiple connections, relationships, or associations between elements. For example, when a connection element represents the communication of signals, data, or instructions, it should be understood by those skilled in the art that such an element represents one or more signal paths necessary to affect the communication.

[0043] The same reference numerals used in the various drawings indicate like elements.

Best Mode for Carrying Out the Invention

[0044] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. It will be apparent to those skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features, each of which can be used independently or in any combination with other features, are described below.

[0045] The disclosed embodiments described below are for an automotive audio system, but can also be used in any immersive listening environment where fading is required or desired, or any immersive listening environment where a predetermined multichannel presentation is modified based on user input.

[0046] As used herein, the term "comprising" and variations thereof are to be read as an open-ended term meaning "including, but not limited to." The term "or" is to be read as "and / or" unless the context clearly dictates otherwise. The term "based on" is to be read as "at least partially based on," "one implementation" and "an implementation" are to be read as "at least one implementation," "another implementation" is to be read as "at least one other implementation," and "determined," "determines," or "determining" are to be read as obtaining, receiving, computing, calculating, estimating, predicting, or deriving. Further, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0047] [Example of Immersive Speaker Layout] Figure 3A shows an example of a vehicle having an immersive speaker layout. The left half and the center speaker of the vehicle are shown. The corresponding speakers are on the right side of the vehicle. For example, a woofer (e.g., a mid-bass or sub-bass woofer) or a full-range loudspeaker is behind the rear seat (e.g., on the rear deck) or embedded in the front or rear doors of the vehicle, and the height loudspeaker is arranged above all other loudspeakers at or near the ceiling inside the vehicle (e.g., arranged on the left and right pillars separating the windows inside the vehicle), and the tweeter loudspeaker and the center loudspeaker are either embedded in the dashboard of the vehicle as shown or arranged below the front windshield pillar. Figure 3B shows the same example as Figure 3A, but the loudspeakers are labeled with common immersive audio channel names. Other loudspeaker layouts are possible.

[0048] [Fading] Over the years, automotive audio systems have been able to "fade" or "pan" audio to the front or rear, left or right of the vehicle. From the user's perspective, some automotive audio systems provide variable front-to-back and left-to-right control. Fading can be achieved by reducing the level of a particular loudspeaker channel signal. For example, the level of the "rear" speakers may be reduced compared to the level of the "front" speakers.

[0049] Figure 4 is a conceptual diagram of how fading is implemented in a four-channel (front left / right and rear left / right) automotive audio system 400 within a vehicle having two rows of seats, according to one or more embodiments. Stereo audio from any number of sources, including AM / FM radio, CD, MP3 file playback, streaming, and satellite radio, enters a fader processor 401, which applies a gain matrix to direct the audio to any combination of loudspeakers.

[0050] The multi-channel audio output (multimedia audio in a stereo pair) by the fader processor 401 typically passes through a loudspeaker processor 402 that applies at least one of equalization, crossover filtering (e.g., for a multi-way speaker with a woofer and a tweeter), speaker protection filtering, or level limiting. The loudspeaker processor 402 outputs the multi-channel audio signal to a loudspeaker amplifier.

[0051] Figures 5A - 5C show examples of matrices of gains applied to input audio to produce output audio within a vehicle, according to one or more embodiments. Referring to FIG. 5A, when the fader is set to "front", stereo audio is supplied only to the left front and right front loudspeaker channels ("1.0" represents 100% of linear gain, and "0.0" represents no linear gain). All other channel gains are set to zero. Referring to FIG. 5B, when the fader is set to "center", stereo audio is supplied to the loudspeaker channels in both rows. Referring to FIG. 5C, when the fader is set to "rear", stereo audio is supplied only to the left rear and right rear loudspeaker channels. Fading of stereo content can be extended to vehicles with additional rows of loud seats, and thus left / right pairs of loudspeakers, by expanding the matrix to have additional gain values for additional loudspeaker pairs.

[0052] FIG. 6 is a conceptual diagram showing how fading is implemented in an automotive audio system 600 of a vehicle (e.g., a three - row vehicle or a vehicle with a high - trim - level audio system) that includes additional speakers, according to one or more embodiments. Stereo audio from any number of sources, including AM / FM radio, CD, MP3 file playback, streaming, and satellite radio, enters a fader processor 601, which applies a matrix of gains to direct the audio to any combination of loudspeakers. The output of the fader processor 601 is multi - channel audio for stereo pairs (front left - right, center left - right, rear left - right), with an optional low - frequency effects channel (e.g., a subwoofer channel) added.

[0053] The multi-channel audio output (multi-channel stereo pair) from the fader processor 601 passes through the loudspeaker processor 602, which typically applies at least one of equalization, crossover filtering (e.g., for a multi-way loudspeaker with woofer and tweeter), speaker protection filtering, or level limiting. The loudspeaker processor 602 typically uses a low-pass filter on the LFE channel to remove mid and high frequencies. The loudspeaker processor 602 outputs the multi-channel audio signal to the loudspeaker amplifier.

[0054] [Fading of Immersive Audio] Both immersive audio content and vehicles with a large number of loudspeaker channels (including height channels) pose challenges when implementing fading to create a quiet zone in front or behind while maintaining an immersive audio experience for a listener sitting mainly in the position closest to the active speakers. The following disclosure describes a method for fading immersive audio in an automotive audio system for two classes of speaker layouts. Other classes of speaker layouts can also be used.

[0055] FIG. 7 is a conceptual flow diagram of an immersive automotive audio system 700 in a vehicle having a stereo pair of loudspeakers according to one or more embodiments. Input audio is rendered by a spatial audio renderer 701 into a multi-channel audio presentation format such as 5.1.2, 5.1.4, 7.1.4, or other current or future multi-channel audio presentation formats. The first digit, here 7, indicates the number of channels in the horizontal plane around the listener. The second digit, here 1, is the number of LFE (Low Frequency Effect) channels. The third digit, here 4, indicates the number of height channels. The height channels are signals intended for reproduction in the listening environment from height speakers (see FIG. 3A), and the height speakers are mounted and aimed to direct acoustic output (either directly or via reflection) from above the intended listening position to the intended listening position. Note that the number of speakers for reproduction may be more than the number of channels (e.g., when the left channel consists of a tweeter, midrange, and woofer at different positions), or less than the number of channels (e.g., when there is no center speaker and the center channel is panned to the left and right speakers). The presentation typically defines the spatial position and orientation for the loudspeakers in the horizontal and height planes relative to the listening position. For example, the four height speakers in FIG. 3B are left front height, right front height, left rear height, and right rear height.

[0056] The multi-channel presentation is input to a fading processor 702, which generates and outputs multi-channel audio with the LFE channel added to the stereo pair. The multi-channel audio output by the fading processor 702 is optionally input to a speaker processor 703 that outputs speaker signals for further processing of the speakers and / or to an audio amplifier.

[0057] Figures 8A and 8B show examples of matrix gains for faders set to "center" and "front" according to one or more embodiments. In these figures, empty cells correspond to a linear gain of 0.0. In both examples, the center front input channel spreads to both the left front and right front speakers. In Figure 8B, the surround and height channels are slightly attenuated when mixed into the left front and right front speakers so as not to overwhelm the immersive audio content from the left front, center front, and right front input channels. Also note that the subwoofer is typically at the rear of the vehicle (in this example, trying to be quiet), and the LFE channel is discarded because the LFE signal may be too low in frequency for the front speaker's function. Alternatively, the LFE can be mixed into a woofer that can reproduce the required low frequencies.

[0058] The greater the number of input presentation channels including the height channel, the more the fader process increases the amount input to the output combination and the larger the matrix becomes. In addition to applying input-to-output gain, it may be beneficial to apply delay and filtering in the fader processors 401, 601, 702. This will be explained in more detail in some of the examples below.

[0059] Refer to the vehicle shown in Figures 3A and 3B. The illustrated loudspeaker layout has a (front) center channel speaker, left / right pairs of front, middle, and rear speakers, left / right pairs of front, middle, and rear height speakers, and a rear subwoofer.

[0060] FIG. 9 is a conceptual flow diagram of an automotive audio system having spatial playback capabilities within a vehicle, including a front center speaker and height speakers, according to one or more embodiments. Spatial audio information that constitutes audio and metadata is input to a spatial audio renderer 901, which generates and outputs a multi-channel audio presentation (e.g., 7.1.4). The multi-channel presentation is input to a fading processor 902, which generates and outputs multi-channel audio with height speakers and an LFE channel in addition to a stereo pair. The multi-channel audio is input to a loudspeaker processor 903 that outputs the multi-channel audio signal to a loudspeaker amplifier.

[0061] FIG. 10A shows a fade matrix for a "center" setting according to one or more embodiments. This configuration can also be referred to here as "full surround" because there is a nearly 1:1 mapping between the input presentation audio channels and the output speaker channels.

[0062] Figure 10B shows a fader matrix for a fader position or mode intended to be intermediate between the "center" and "front" positions. This configuration is hereinafter referred to as "front surround", which attempts to provide an immersive audio surround experience at the front seat positions while quieting the second row. The left front, center front, right front, and height front presentation channels are mapped 1:1 to the corresponding output speakers. The left and right intermediate presentation channels are mixed and slightly attenuated to the front left and right speakers to reduce interference with the front left and right channels. The left and right rear height presentation channels are mapped to the intermediate height output speaker positions, and the left rear and right rear presentation channels are mapped to the left rear and right rear intermediate height speaker positions. Also, the loudspeakers typically have omnidirectionality at low frequencies, and in some embodiments, the mid frequencies can be more troublesome in the second seat row (which is intended to be quiet), so a high-pass filter is included for the left rear to left intermediate height (and corresponding right channel). The high-pass filter provides some of the high-frequency ambient sound (from the rear presentation channels) to the front seat positions without discomfort at mid and low frequencies.

[0063] Figure 10C shows a fader matrix for a "front" fader position according to one or more embodiments. All input presentation channels are mapped to the front or front height loudspeaker channels. This setting loses the front-back spatial aspect of the immersive sound but retains the height aspect of the immersive sound and provides maximum quietness at the rear of the vehicle.

[0064] Figure 10D shows a fader matrix for a fader position intended to be intermediate between the "center" and "rear" positions according to one or more embodiments. This configuration can also be referred to herein as "rear surround". This is because this configuration attempts to give an immersive audio surround experience to the rear seat positions while quieting the front row.

[0065] Figure 10E shows a fader matrix for a "rear" fader position according to one or more embodiments. All input presentation channels are mapped to rear or rear height loudspeaker channels. This setting loses the front-back spatial aspect of the immersive sound but retains the height aspect of the immersive sound and provides maximum quietness in front of the vehicle.

[0066] [Other fader positions] In the example above, changes in the front-back fader position are shown. Additional fader matrices can include, but are not limited to, left-right fading, fading to corner positions (e.g., corresponding to the driver's seat position), and up-down fading. Also, the example shows five specific positions, but finer control of the fader position can be achieved by using additional matrices or interpolating the gain values between the specific matrices as shown. Also, the fader matrix can be designed to quiet specific seat positions. For example, if the driver is on a phone call, the "driving call" matrix can attempt to reduce the level of entertainment audio at the driver's seat position.

[0067] [Phantom virtual center processing] FIG. 11 shows a system 1100 for processing a center presentation channel using phantom virtual center (PVC) technology, according to one or more embodiments. Instead of mixing the center presentation channel to the left and right loudspeakers, considering again a vehicle having only a stereo pair of loudspeaker channels as in FIG. 1, in some embodiments, it is beneficial to process the center presentation channel using a PVC processor 1101. The stereo output of the PVC processing is input to a mix matrix 1102, along with other presentation channels. The output of the mix matrix 1102 is a multi-channel audio loudspeaker channel (e.g., door, front center, height channel, and LFE / subwoofer). In some embodiments, other input channel pairs (e.g., left and right channel pairs) that can include similar content can be processed via additional PVC processor instances, and the processor outputs can be provided to the mix matrix.

[0068] [Occupancy Sensing] In some embodiments, the input includes occupancy data provided by vehicle systems (e.g., seat pressure sensors, interior cameras). The occupancy data can include, but is not limited to, the number of occupants in the vehicle and their seat positions. Based on this occupancy data, the multimedia audio presentation is modified or replaced with a mix that optimizes the multi-channel audio experience for the listeners based on their seat positions. For example, if an occupant is sitting on the left side of the vehicle, the mix can be modified to improve the perception of the audio based on the listening position.

[0069] [Zone-Based Immersive Audio] In some embodiments, the interior of the vehicle is divided into two or more zones, and the mix is determined based at least in part on the two or more zones. The zones can be the front, rear, and sides of the vehicle, or can be divided perpendicular to several planes (e.g., the bottom plane, horizontal plane, and height plane). In some embodiments, the vehicle can include a "quiet" zone that receives a lower audio level than other parts of the vehicle interior. This can be achieved, for example, by removing / attenuating the LFE loudspeaker or other loudspeakers within the quiet zone.

[0070] [Process Example] FIG. 12 is a flow diagram of VLBR Ambisonics processing according to one or more embodiments. Process 1200 can be implemented using the electronic device architecture described with reference to FIG. 13.

[0071] Process 1200 includes receiving object-based audio and metadata (1201), rendering the object-based audio into a multi-channel audio presentation for a first loudspeaker layout based on the metadata (1202), determining a first mix based on the multi-channel audio presentation related to the vehicle and a second loudspeaker layout (1203), generating a first loudspeaker signal based on the first mix for playback through the speakers in the second loudspeaker layout (1204), receiving an input (1205), determining a second mix different from the first mix based on the multi-channel audio presentation and the input (1206), and generating a second loudspeaker signal based on the second mix for playback through the speakers in the second loudspeaker layout (1207).

[0072] In some embodiments, the second loudspeaker layout may not correspond to each speaker in the vehicle's loudspeaker layout (e.g., the number of channels / signals associated with the second loudspeaker layout may be different from the number of physical speakers in the vehicle's audio system). For example, the left front channel / signal of the loudspeaker signal (e.g., after mixing) may be sent, based on a cutoff frequency, to a crossover device that sends a low-frequency audio signal (e.g., LFE content) to a woofer in the vehicle's door panel and a high-frequency audio signal to a tweeter in the dashboard, for example. In such embodiments, the loudspeaker signals represent a general set of channels or signals rather than a specific set of channels or signals associated with each corresponding physical speaker.

[0073] In some embodiments, the first and / or second loudspeaker signals may be further processed before and / or after amplification before being sent to the loudspeakers via the loudspeaker processor 703 / 903 (see FIGS. 7 and 9).

[0074] In some embodiments, determining the second mix based on the first mix and input further includes transitioning from the first spatialization mode to the second spatialization mode, and transitioning includes reassigning a portion of the first loudspeaker signal (e.g., the level of the complete signal, the level of the attenuated signal, the full-bandwidth signal, the non-full-bandwidth signal) to the second loudspeaker signal.

[0075] In some embodiments, the reassigning is based at least in part on the distance from the listener or listener position associated with the listener and at least one loudspeaker associated with the second speaker layout.

[0076] In some embodiments, the reassignment moves the content from a loudspeaker at a first distance from a particular listener or the listener's location to a loudspeaker at a second distance greater than the first distance from the particular listener or the listener's location.

[0077] In some embodiments, the reassignment moves a portion of the first loudspeaker signal from at least one loudspeaker of a first speaker layout at a first distance from a listener or the listener's location to at least one other loudspeaker within a second speaker layout at a second distance greater than the first distance from the listener or the listener's location.

[0078] In some embodiments, the reassignment is performed according to the speaker performance characteristics of at least one loudspeaker associated with the second speaker layout.

[0079] In some embodiments, if the loudspeaker within the second loudspeaker layout associated with the second channel has the frequency response necessary to reproduce the audio content reassigned from the first channel (e.g., if the low-frequency content is assigned to a loudspeaker having a sufficiently low frequency response), the audio content is reassigned from the first channel to the second channel in a multimedia audio presentation.

[0080] In some embodiments, the reassignment includes reassigning a portion of the first loudspeaker signal from a center channel associated with the first speaker layout to two or more second loudspeaker signals for non-center channel loudspeaker channels associated with the second speaker layout.

[0081] In some embodiments, the reassignment includes attenuating one or more portions of the second loudspeaker signal.

[0082] In some embodiments, the reallocation includes determining a signal coherence value between audio contents in a portion of the first loudspeaker signal and applying increased attenuation to the second loudspeaker signal according to a coherence value that exceeds one or more coherence thresholds.

[0083] In some embodiments, the reallocation includes high-pass filtering at least one loudspeaker signal of the second loudspeaker signal corresponding to one or more height channels of the multimedia presentation.

[0084] In some embodiments, the multi-channel audio presentation includes at least one pair of height audio channels.

[0085] [Example of System Architecture] FIG. 13 shows a block diagram of an exemplary electronic device architecture 1300 suitable for implementing an exemplary embodiment of the present disclosure. Architecture 1300 includes, but is not limited to, a server and a client device as described above with reference to FIGS. 1-6. As shown, architecture 1300 can execute various processes according to, for example, a program stored in read-only memory 1302 or a program loaded from storage unit 1308 to random access memory 1303, and includes a central processing unit 1301. In RAM 1303, data required when CPU 1301 executes various processes is also stored as needed. CPU 1301, ROM 1302, and RAM 1303 are connected to each other via bus 804. Input / output interface 1305 is also connected to bus 1304.

[0086] The following components are connected to the input / output interface 1305: an input unit 1306 that can include a keyboard, a mouse, etc.; an output unit 1307 that can include a display such as a liquid crystal display and one or more speakers; a storage unit 1308 that includes a hard disk or another suitable storage medium; and a communication unit 1309 that includes a network interface card such as a network card (e.g., wired or wireless).

[0087] In some implementations, the input unit 1306 includes one or more microphones at different positions (depending on the host device) that enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0088] In some embodiments, the output unit 1307 includes a system having a varying number of speakers. The output unit 1307 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats). In some embodiments, the communication unit 1309 is configured to communicate with other devices (e.g., via a network). Also, a drive 1310 is connected to the input / output interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive, or another suitable removable medium is attached to the drive 1310 such that a computer program read therefrom is installed in the storage unit 1308 as needed. Those skilled in the art will understand that although the system 1300 is described as including the above-described components, in actual applications, some of these components can be added, removed, and / or replaced, and all of these modifications or changes are within the scope of the present disclosure.

[0089] According to an exemplary embodiment of the present disclosure, the above-described process can be implemented as a computer software program or on a computer-readable storage medium. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a computer-readable medium, the computer program including program code for performing the method. In such an embodiment, the computer program may be downloaded, mounted, and / or installed from a removable medium 1311 via a communication unit 1309 from a network as shown in FIG. 13.

[0090] In general, various exemplary embodiments of the present disclosure can be implemented in hardware or special purpose circuitry (e.g., control circuitry), software, logic, or any combination thereof. For example, the above-described units can be executed by a control circuit (e.g., CPU 1301 combined with other components in FIG. 13), and thus the control circuit can execute the actions described in the present disclosure. Some aspects can be implemented in hardware, and other aspects can be implemented in firmware or software executable by a controller, a microprocessor, or other computing device (e.g., control circuitry). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described using block diagrams, flowcharts, or some other pictorial representation, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, special purpose circuitry, or logic, general purpose hardware, or a controller, or other computing device, or some combination thereof.

[0091] Furthermore, the various blocks shown in the flowchart can be regarded as steps of the method, and / or actions resulting from the operation of the computer program code, and / or a plurality of combinational logic circuit elements configured to perform the related functions. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to execute the above method.

[0092] In the context of the present disclosure, a machine-readable medium may be any tangible medium that can store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may be non-transitory and may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or suitable combinations thereof. More specific examples of the machine-readable storage medium include electrical connections having one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or suitable combinations thereof.

[0093] Computer program code for carrying out the method of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing apparatus having a control circuit, so that, when the program code is executed by the processor of the computer or other programmable data processing apparatus, it causes the functions / operations specified in the flowchart and / or block diagram to be executed. The program code can be executed entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server, or on a computer distributed on one or more remote computers and / or servers.

[0094] This document contains many details of specific implementations, but these do not limit the scope of the claims. Rather, they should be construed as descriptions and interpretations of features specific to particular embodiments. The specific features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately, or in any suitable sub - combination, in multiple embodiments. Further, features are described above as acting in a particular combination, and even if initially claimed as such, one or more features of the claimed combination may, in some cases, be excised from that combination, and the claimed combination may be directed to sub - combinations or variations of sub - combinations. The logical flows shown in the figures are shown in a particular order, or no order, to achieve the desired result. Additionally, other steps may be provided, or steps may be removed from the flows described, other components may be added to the systems described, or components may be removed from the systems described. Accordingly, other embodiments are within the scope of the claims.

[0095] [Example] [EE1] In an electronic device including a user input device and a second subset of a second speaker type different from the first speaker type (e.g., height planar speaker, height speaker, near-field speaker, etc.), the set of loudspeakers is arranged in a device speaker layout within a sealed space (e.g., a listening environment, a vehicle cabin, a room, a theater, a game venue, etc., where loudspeakers are attached around the sealed space), and is in a format corresponding to a presentation layout (e.g., 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4, 5.1.receiving audio program content as a first set of audio signals in a set of audio channel signals corresponding to a predetermined channel layout such as 2; generating a second set of audio signals corresponding to a set of loudspeakers based on the first set of audio signals (e.g., a set of speaker audio signals corresponding to the speaker layout of the device), wherein the second set of audio signals has the same number of audio signals as the same number of loudspeakers in the set of loudspeakers using a first set of gains; supplying the second set of audio signals, or a signal derived from the second set of audio signals, to the set of loudspeakers, wherein the set of loudspeakers generates an acoustic output that spatializes the audio program content represented by the first set of audio signals; receiving, at the user input device, a sequence of one or more user inputs corresponding to a request to spatially modify the acoustic output; in response to receiving the sequence of one or more user inputs, transitioning the operation of the electronic device from a first spatialization mode to an operation in a second spatialization mode, wherein, from the first set of audio signals, a third set of audio signals corresponding to the set of loudspeakers is generated using a second set of gains different from the first set of gains (e.g., speaker audio signals including a modified spatial quality); and supplying the third set of audio signals, or a signal derived from the third set of audio signals, to the set of loudspeakers, wherein the set of loudspeakers generates an acoustic output that spatializes the audio program content represented by the first set of audio signals, thereby effecting the transition.

[0096] In some embodiments, the user input device is a physical / mechanical control device such as a touch screen or touch-sensitive surface, knob, dial, slider, button, rotatable and depressible input device, audio input device, and a first subset of loudspeakers including a first speaker type (e.g., horizontal plane speaker, non-height speaker, far-field speaker, etc.).

[0097] In some embodiments, the electronic device is a vehicle, media playback device, infotainment system, head unit, multifunctional device (e.g., phone, tablet) coupled to a media playback system, etc.

[0098] In some embodiments, the presentation layout is different from the device speaker layout.

[0099] In some embodiments, the layout of the predetermined speakers is the same as the layout of the device speakers.

[0100] In some embodiments, the first set of gains is a matrix of gains corresponding to a default spatialization operation mode or spatialization setting.

[0101] In some embodiments, the set of signals is derived by further processing (e.g., applying downmixing or equalization, and / or amplifying the obtained signals) to drive the amplifiers of the loudspeakers.

[0102] [EE2] A method disclosed herein, wherein the first speaker type is a non-height speaker type or a horizontal plane speaker type (e.g., a speaker aimed at directing acoustic output (directly or via reflection) to a predetermined listening position from below a height plane speaker, or from below or on the same plane as a predetermined listening position).

[0103] [EE3] A method disclosed in this specification, wherein the second speaker type is a height speaker type or a height plane speaker type (e.g., a speaker that is mounted and is intended to direct acoustic output from above a horizontal plane speaker or from above an intended listening position to the intended listening position (directly or via reflection)).

[0104] [EE4] Any method disclosed in this specification, wherein the first speaker type is a far-field speaker type.

[0105] [EE5] Any method disclosed in this specification, wherein the second speaker type is a near-field speaker type.

[0106] [EE6] Any method disclosed in this specification, wherein the set of loudspeakers includes one or more dedicated low-frequency speakers. In some embodiments, the low-frequency loudspeaker includes a subwoofer, a bass shaker, a force transducer, a tactile transducer, or other low-frequency optimized transducer.

[0107] [EE7] Any method disclosed in this specification, wherein the first set of audio signals includes one or more height audio channels.

[0108] [EE8] Any method disclosed in this specification, wherein the first set of audio signals corresponds to a speaker format selected from the sets of 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4, and 5.1.2.

[0109] [EE9] Before receiving audio program content as a first set of audio signals, receiving audio based on an object representing the audio program content, the audio based on the object including a set of one or more sound source essence signals having corresponding information (e.g., metadata) indicating the spatial characteristics of respective sound sources, and rendering the audio based on the object, using an object renderer, in a format corresponding to the presentation layout, to the first set of audio signals, further comprising, any method disclosed herein.

[0110] [EE10] Spatial characteristics include at least one selected from the set of the position of each source in three-dimensional space relative to the listener position, the size or level of the spread of each source, and the distance of each source from the listener position, any method disclosed herein. In some embodiments, the audio based on the object is a media program, music or video program, audio data related to the operation of a non-entertainment system (e.g., a safety bulletin), etc. In some embodiments, the audio based on the object is stored locally as a complete file or received sequentially via a streaming protocol.

[0111] [EE12] The second set of gains is generated from the first set of gains, and a third set of gains is different from the first set of gains and different from the second set of gains, any method disclosed herein.

[0112] [EE13] Transitioning the operation of an electronic device from an operation in a first spatialization mode to an operation in a second spatialization mode involves reassigning a portion of the first set of audio signals (e.g., full signal level, attenuated signal level, full bandwidth signal, non-full bandwidth signal) from a channel or signal associated with the presentation layout to a set of non-corresponding channels or signals associated with the device speaker layout, according to any method disclosed herein.

[0113] [EE14] The reassigning is performed at least in part based on the distance from the listener or the seating position associated with the listener, and at least one of the set of loudspeakers arranged within the device speaker layout in the enclosed space. In some embodiments, reassigning is performed by moving content from a loudspeaker at a first distance from a particular listener or listener position to a loudspeaker at a second, greater, distance from the particular listener or listener position, according to any method disclosed herein.

[0114] [EE15] The reassigning is performed according to at least one speaker performance characteristic of the set of loudspeakers arranged within the device speaker layout in the enclosed space. In some embodiments, if the loudspeaker associated with the second channel has the frequency response necessary to reproduce the content reassigned from the first channel, the content is reassigned from the first channel to the second channel (e.g., low frequency content is assigned only to loudspeakers with sufficient low frequency response).

[0115] [EE16] The reassigning includes reassigning a portion of the audio signals of the first set from the center channel of the presentation layout to two or more non-center channel loudspeaker channels associated with the device speaker layout, according to any method disclosed herein.

[0116] [EE17] In any of the methods disclosed herein, reassignment of one or more portions of the first audio signal set includes attenuating one or more portions of the first audio signal set.

[0117] [EE18] Reassignment includes determining a signal coherence value between contents of a part of the first audio signal set and applying an increased attenuation according to the coherence value exceeding one or more coherence thresholds, in any of the methods disclosed herein.

[0118] [EE19] Reassignment includes high-pass filtering one audio signal of the first set of audio signals corresponding to one or more height channels of the presentation layout, in any of the methods disclosed herein.

[0119] [EE20] By supplying the third set of audio signals or a signal derived from the third set of audio signals to the set of loudspeakers, the local sound field at a first listening position within the enclosed space is such that, with respect to the acoustic output generated by supplying the second set of the second set of audio signals or a signal derived from the second set of audio signals to the loudspeakers, a reduced acoustic output (e.g., sound pressure, perceptual weighting or absolute value) is achieved while maintaining spatialization (e.g., a perceived height effect that varies according to the metadata of the source signal) within at least a second listening position within the enclosed space, in any of the methods disclosed herein.

[0120] [EE21] The second or third set of audio signals includes one or more height audio channel pairs, in any of the methods disclosed herein.

[0121] [EE22] The second or third set of audio signals corresponds to a speaker format selected from the sets of 9.1.6, 9.1.4, 9.1.2, 7.1.6, 7.1.4, 7.1.2, 5.1.6, 5.1.4, and 5.1.2, by any method disclosed herein.

[0122] [EE23] The second or third set of audio signals corresponds to a speaker format that does not include a height channel, by any method disclosed herein.

[0123] [EE24] The enclosed space includes seats arranged in the first row of the listening or seating position, by any method disclosed herein.

[0124] [EE25] The second subset of loudspeakers includes a first pair of height speakers mounted for the purpose of directing acoustic output (i) from a position in front of the first row, and (ii) from a position above the first subset of speakers, to a listening position corresponding to the first row of the listening or seating position.

[0125] [EE26] The enclosed space includes a second row of the listening or seating position, located behind the first row of the listening or seating position, by any method disclosed herein.

[0126] [EE27] The second subset of loudspeakers includes a second pair of height speakers mounted for the purpose of directing acoustic output (i) from a position behind the first row of the listening or seating position, and / or (ii) from a position in front of the second row of the listening or seating position, (iii) from a position above each of the first subset of speakers, and (iv) to a listening position corresponding to the second row of the listening or seating position.

[0127] [EE28] The enclosed space includes a third row of the listening or seating position located behind the second row of the listening or seating position, and is any method disclosed herein.

[0128] [EE29] The second subset of loudspeakers includes a third pair of speakers mounted for the purpose of directing acoustic output (i) from a position above the first subset of speakers and (ii) from behind the second row. In some embodiments, the third pair of height speakers is mounted at the rear interior deck, C-pillar or D-pillar of the automobile cabin. In some embodiments, the height speakers are designed to reflect sound from the surface in a downward direction towards the listener at the seating position. In some embodiments, the height speakers are mounted on top of the first subset of speakers.

[0129] [EE30] After generating the second set of audio signals or after generating the third set of audio signals, and before supplying each audio signal to the loudspeaker set, each audio signal is further processed to compensate for one or more of speaker position, speaker response, absorption and reflection characteristics of nearby materials within the enclosed space, and the auditory sensitivity of an occupant or listener within the enclosed space. In some embodiments, processing includes time alignment (e.g., based on the distance to one or more listeners or seating positions), active or passive filtering, and the like.

Claims

1. Receiving object-based audio and metadata using at least one processor of an audio system; Rendering the object-based audio based on the metadata into a multi-channel audio presentation for a first loudspeaker layout using the at least one processor; Determining a first mix based on the multi-channel audio presentation and a second loudspeaker layout associated with the vehicle using the at least one processor; Generating a first loudspeaker signal based on the first mix for reproduction through loudspeakers within the second loudspeaker layout using the at least one processor; Receiving an input using the at least one processor; Determining a second mix different from the first mix based on the multi-channel audio presentation and the input using the at least one processor; Generating a second loudspeaker signal based on the second mix for reproduction through the loudspeakers within the second loudspeaker layout using the at least one processor; A method comprising the above steps.

2. The method according to claim 1, wherein the multi-channel audio presentation includes at least one pair of stereo audio channels.

3. The method according to claim 1 or 2, wherein the multi-channel audio presentation includes at least one pair of stereo channels and at least one low-frequency effect (LFE) channel.

4. The method according to any one of claims 1 to 3, wherein the input includes a fader position of a fader control of the audio system.

5. The method according to any one of claims 1 to 4, wherein the input includes a fader mode indicating a preset modification to the multi-channel audio presentation.

6. The method according to any one of claims 1 to 5, wherein the input includes occupancy data indicating the number of occupants of the vehicle and their seat positions within the vehicle.

7. The interior of the vehicle is divided into two or more zones, and the second mix is determined at least in part based on the two or more zones, according to the method of any one of claims 1 to 6.

8. The second mix applies a gain to at least one channel of the multi-channel audio presentation, according to the method of any one of claims 1 to 7.

9. The gain is included in a set of gains that map channels within the multi-channel audio presentation to the loudspeakers within the second loudspeaker layout, according to the method of claim 8.

10. The multi-channel audio presentation includes more channels than there are loudspeakers within the second loudspeaker layout, according to the method of any one of claims 1 to 9.

11. The second loudspeaker layout includes left / right front speakers and left / right rear speakers, and the multi-channel audio presentation includes left / right / center loudspeakers, left / right midrange loudspeakers, and left / right rear loudspeakers, according to the method of any one of claims 1 to 10.

12. The second loudspeaker layout further includes at least one of left / right front height speakers and left / right rear height speakers, according to the method of any one of claims 1 to 11.

13. The number of channels in the multi-channel audio presentation is equal to the number of loudspeakers in the second loudspeaker layout, according to the method of any one of claims 1 to 12.

14. The second loudspeaker layout and the multi-channel audio presentation include left / right / center loudspeakers, left / right midrange loudspeakers, and left / right rear loudspeakers, according to the method of any one of claims 1 to 13.

15. The number of channels in the multi-channel audio presentation is less than the number of loudspeakers in the second loudspeaker layout, according to the method of any one of claims 1 to 14.

16. The multi-channel audio presentation includes a front center channel, and the method is Using the at least one processor, generating a phantom virtual center from the front center channel, and further comprising modifying a predetermined spatial position or orientation of at least one loudspeaker in the second loudspeaker layout based on the input and the phantom virtual center, the method according to any one of claims 1 to 15.

17. The method according to any one of claims 1 to 16, wherein the multi-channel audio presentation includes a spatial position or orientation of the loudspeaker in a horizontal plane and a height plane.

18. The method according to any one of claims 1 to 17, wherein the second mix applies a delay or filtering to the second loudspeaker signal.

19. Determining a second mix based on the multi-channel audio presentation and the input further includes transitioning from a first spatialization mode to a second spatialization mode, and the transitioning includes reallocating a portion of the first loudspeaker signal to the second loudspeaker signal, the method according to any one of claims 1 to 18.

20. The method according to any one of claims 1 to 19, wherein the reallocating is at least partially based on a distance from a listener or a listener position associated with the listener, and at least one loudspeaker associated with the second loudspeaker layout.

21. The method according to any one of claims 19 to 20, wherein the reallocating moves a portion of the first loudspeaker signal from at least one loudspeaker in the second speaker layout that is at a first distance from the listener or the listener position to at least one other loudspeaker in the second speaker layout that is at a second, greater than the first distance from the listener or the listener position.

22. The method according to any one of claims 19 to 21, wherein the reallocating is performed according to speaker performance characteristics of at least one loudspeaker in the second speaker layout.

23. Reallocating includes reallocating a portion of the first loudspeaker signal from a center channel associated with the first loudspeaker layout to two or more second loudspeaker signals for non-center channel loudspeaker channels associated with the second loudspeaker layout, the method according to any one of claims 19 to 22.

24. Reallocating includes attenuating one or more portions of the second loudspeaker signal, the method according to any one of claims 19 to 23.

25. Reallocating includes determining a signal coherence value between audio contents in a portion of the first loudspeaker signal; and applying an increased attenuation to the second loudspeaker signal in response to the coherence value exceeding one or more coherence thresholds, the method according to any one of claims 19 to 24.

26. Reallocating includes applying a high-pass filter to at least one loudspeaker signal of the second loudspeaker signals corresponding to one or more height channels of the multimedia presentation, the method according to any one of claims 19 to 25.

27. The multi-channel audio presentation includes at least one pair of height audio channels, the method according to any one of claims 1 to 26.

28. At least one processor; A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to execute any of the methods according to claims 1 to 27, an audio playback system.

29. A non-transitory, computer-readable storage medium including instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods according to claims 1 to 27.

Images