Methods, apparatus, and systems for conversion between audio scene representations

Abstract

Some disclosed examples involve converting input audio data in an input audio format to output audio data in an output audio format. The output audio format may have a higher resolution than the input audio format. The converting may involve applying a biased decoding matrix to the input audio data. The biased decoding matrix may be biased according to an estimated energy distribution of the input audio data and may include a combination of constant matrices and a variable matrix. The variable matrix may be a covariance matrix corresponding to the input audio data. The biased decoding matrix may vary over time as a function of the covariance matrix.

Description

Dolby Ref. D23107WO01 METHODS, APPARATUS, AND SYSTEMS FOR CONVERSION BETWEEN AUDIO SCENE REPRESENTATIONS CROSS REFERNCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application Ser. No.63 / 520,997 filed on 22 August 2023, US Provisional Application Ser. No.63 / 574,602 filed on 4 April 2024 and EP Patent Application No.24168434.9 filed on 4 April 2024, each of which is incorporated by reference herein. TECHNICAL FIELD

[0002] The present disclosure relates to processing of multi-channel audio signals for representing acoustic scenes, and in particular, to the conversion between different multi- channel audio formats that represent the same acoustic scene. BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0004] A set of two or more audio signals, intended to be processed or transmitted as a group, may be referred to herein as a “multi-channel audio signal.” A listening experience may be derived from a multi-channel audio signal, and this listening experience may be intended to provide one or more listeners with an experience that replicates an “acoustic scene”, and in particular, the term “target acoustic scene” refers to the desired listening experience (the listening experience that the multi-channel audio signal is intended to recreate).

[0005] A multi-channel audio signal will typically be associated with additional information that defines the multi-channel audio format. Examples of multi-channel audio formats include:

[0006] Common speaker-based formats such as Stereo, 5-channel surround, 7-channel surround, etcetera, that contain ^ channels (in a multi-channel audio signal), and wherein each channel is associated with a pre-defined direction of arrival (the reference loudspeaker arrangement). For example, the directions of arrival associated with the 5-channel surround format are defined by Recommendation ITU-R BS.775.Dolby Ref. D23107WO01

[0007] Generalizations of speaker-based (or virtual-speaker-based) formats, beyond the commonly known speaker layouts, that contain ^ channels (in a multi-channel audio signal) along with ^ associated directions of arrival.

[0008] Panner-defined formats, wherein an ^-channel multi-channel audio signal is associated with a directional panning function.

[0009] A common family of panner-defined formats is Ambisonics. The first-order Ambisonics format defines a target acoustic scene by providing a multi-channel audio file consisting of ^ = 4 channels, wherein each of the 4 channels defines the signal that is expected to be received by a respective ideal microphone positioned at a central point within the target acoustic wave-field, and wherein each of the said microphones is responsive to incident sounds according to a specific directivity pattern (the directional panning function).

[0010] It is common, according to the convention adopted in the field of Ambisonics production, to define the incident direction of arrival of sounds according to a 3-dimensional coordinate system where the ^-axis points forward, the ^-axis points to the left, and the ^- axis points up.

[0011] In a first-order Ambisonics format, the 4 microphone directional panning functions are chosen to be an omni-directional pattern plus 3 dipole patterns where the 3 dipole patterns are aligned with the ^, ^ and ^ axes respectively. By way of example, an ideal dipole microphone, aligned with the ^-axis (for example), when exposed to an incident sound-wave from a direction defined by the unit-vector ^^, ^, ^^, will capture the incident sound with a gain equal to ^. An ideal omnidirectional microphone pattern can be considered to have a receiving gain of 1, independent of the incident direction of the sound-wave.

[0012] Equation 1 defines a panning function (^^^^, ^, ^^) that maps a direction of arrival (in the form of the unit-vector^^, ^, ^^) to a column vector of 4 gain values. This defines the first-order Ambisonics format (also commonly referred to as AmbiX): 1 (1)

[0013] It will be appreciated by those skilled in the art that multiple conventions exist for defining the gain-scale and channel ordering of the signals in an Ambisonics format. It willDolby Ref. D23107WO01 also be appreciated that the methods described in this disclosure may be applied to Ambisonics signals that adhere to alternative scale and channel-order conventions, without loss of generality.

[0014] It is known in the art to define second- or third-order Ambisonics formats, consistingof 9 or 16 channels respectively, with the associated panning functions (^^^^, ^, ^^ and^^^^, ^, ^^) shown in Equation 2 and Equation 3, respectively.1^)Dolby Ref. D23107WO01

[0015] Panner-defined audio formats, such as Ambisonics, are useful because they allow complex acoustic scenes to be represented in a multi-channel audio signal that allows for easy manipulation and analysis using readily available audio processing tools.

[0016] Some panner-defined formats, such as Ambisonics, map each direction of arrival toan ^-vector of real gain values. This means we may define the associated panning function as^: "^ → ℝ%, where "^ refers to the set of all ^^, ^, ^^ unit-vectors (the surrace of the 2-spherein 3D space).

[0017] It is also known that more complex panner-based formats arise when physical (or virtual) arrays of microphones are arranged so as to capture an acoustic scene. In this case, we may define a panner function (e.g. ^^^, ^, ^, &^) that, for each frequency &, maps each directional of arrival to an ^-vector of complex gain values: ^: "^× ℝ → ℂ%.

[0018] Multi-channel audio signals may be used to generate an acoustic playback experience for one or more listeners, utilizing loudspeakers, headphones, or other means for playback. Some form of audio processing may be required to translate the multi-channel audio format to a particular playback device.

[0019] It will be appreciated that a multi-channel audio signal may be intended to create a particular playback experience, which may be referred to as a target acoustic scene. It will be further appreciated that multi-channel audio formats with a larger number of channels may be capable of conveying a more accurate representation of the target acoustic scene, compared to multi-channel audio formats with a lesser number of channels. Accordingly, multi-channel audio formats having a relatively larger number of channels may be referred to herein as having a “higher resolution” than multi-channel audio formats having a relatively smaller number of channels.

[0020] It is often desirable to convert an input multi-channel audio signal associated with a panner-based audio format (the input audio format) to an output multi-channel audio signal associated with an alternate panner-based audio format (the output audio format). This conversion may be desirable for one or more of the following reasons:

[0021] The output audio format may contain a larger number of channels, thus providing a more accurate, higher-resolution representation of a target acoustic scene.Dolby Ref. D23107WO01

[0022] The input audio format may be associated with a specific acoustic capture device, and later processing steps may not be capable of dealing with the format of the specific acoustic capture device. SUMMARY

[0023] According to some examples, an acoustic scene represented by input audio data in an input audio data format may be converted into to output audio data in an output audio format. The output audio format may have a higher resolution than the input audio format. In some examples, the input audio data may be in the form of a panner-based audio signal—such as an Ambisonics audio signal—having a input audio data resolution. In some such examples, the input audio data may be converted from into an output panner-based audio format having a higher resolution than the input audio data resolution.

[0024] At least some aspects of the present disclosure may be implemented via methods, such as audio processing methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some such methods involve receiving, by a control system that includes one or more processors, the input audio data in an input audio format. Some methods involve converting, by the control system, the input audio data to output audio data in an output audio format. In some examples, the output audio format may have a higher resolution than the input audio format. Some methods involve storing or transmitting the output audio data. Some methods may involve providing the output audio data to a loudspeaker-enabled device or a loudspeaker-enabled system for playback.

[0025] In some examples, the converting may involve applying, by the control system, a biased decoding matrix to the input audio data to produce the output audio data. According to some examples, the biased decoding matrix may be biased according to an estimated energy distribution of the input audio data. In some examples, the biased decoding matrix may include a combination of constant matrices and a variable matrix. According to some examples, the variable matrix may include a covariance matrix corresponding to the input audio data. In some examples, the biased decoding matrix may vary over time as a function of the covariance matrix.

[0026] According to some examples, the biased decoding matrix may be based, at least in part, on a combination of a version of an A matrix and a B matrix. In some such examples, the A matrix may be an input format covariance matrix and the B matrix may be an outputDolby Ref. D23107WO01 format to input format cross covariance matrix. In some examples, the A matrix may be a biased input format covariance matrix and the B matrix may be a biased output format to input format cross covariance matrix. In some such examples, the A matrix and the B matrix may be biased according to the estimated energy distribution of the input audio data. According to some examples, the estimated energy distribution of the input audio data may be based on an energy estimation matrix. In some such examples, the energy estimation matrix may be based in part on a passive virtual source decode matrix. In some examples, the passive virtual source decode matrix may be based, at least in part, on a pseudo-inverse of a matrix used for decoding the input audio data to a set of virtual loudspeaker signals. In some examples, the version of the A matrix may be an inverse of the A matrix.

[0027] In some examples, the constant matrices may include a matrix corresponding to panning rules for the input audio data. According to some examples, the constant matrices may include a matrix corresponding to panning rules for the output audio data. In some examples, the output audio format may have a higher number of elements than the input audio format, the number of elements corresponding to a number of channels or an Ambisonics order.

[0028] According to some examples, the resolution of the input audio format may correspond to a number of channels of the input audio format or an Ambisonic order of the input audio format. According to some such examples, the resolution of the output audio format may correspond to a number of channels of the output audio format or an Ambisonic order of the output audio format.

[0029] In some examples, the covariance matrix may be a Hermitian matrix. In some such examples, applying the biased decoding matrix may involve performing a summation of elements above or below a diagonal of the Hermitian matrix.

[0030] According to some examples, the input audio data may include one of a plurality of band-limited components of a wider-bandwidth input audio data. In some such examples, the output audio data may include one of a plurality of band-limited components of a wider- bandwidth output audio data.

[0031] Some methods may involve producing a set of virtual loudspeaker signals. According to some such examples, the set of virtual loudspeaker signals may have a higher resolution than a resolution of the input audio format or the output audio format. In some examples, the set of virtual loudspeaker signals may be produced prior to the converting, storing or transmitting operations.Dolby Ref. D23107WO01

[0032] According to some examples, the converting operations and the storing or transmitting operations may be runtime operations. In some examples, the set of virtual loudspeaker signals may be produced prior to the runtime operations.

[0033] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0034] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0035] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0037] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0038] Figure 2A shows an arrangement wherein an input audio signal ^ is processed to form an output audio signal Y.

[0039] Figure 2B shows a process of generating a higher-order-Ambisonics (HOA) signal according to some examples.Dolby Ref. D23107WO01

[0040] Figure 3 shows another example of producing an output audio signal Y from an input audio signal X.

[0041] Figure 4 shows graphic representations of input audio signals X, output audio signals Y and virtual sources according to one example.

[0042] Figure 5 shows examples of two useful matrices.

[0043] Figure 6A shows the relationship between the input audio signal X and the matrix )*.

[0044] Figure 6B shows the relationship between the output audio signals Y and the matrix)+.

[0045] Figure 7 illustrates one method of computing a passive virtual-source decode matrix.

[0046] Figure 8 illustrates one method of computing an improved virtual-source decode matrix.

[0047] Figure 9 shows blocks of a complete upmixer according to one example.

[0048] Figure 10 shows blocks that represent example processes according to a further embodiment.

[0049] Figure 11 shows an example of the decorrelation process of Figure 10.

[0050] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0051] Figure 13 is a flow diagram that outlines another example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0052] Figure 14A illustrates a schematic block diagram of an example device architecture that may be used to implement various aspects of the present disclosure.

[0053] Figure 14B illustrates a schematic block diagram of an example CPU implemented in the device architecture of Figure 14A that may be used to implement various aspects of the present disclosure. DETAILED DESCRIPTION

[0054] Described herein are techniques related to conversion of audio signals from one format to another. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.Dolby Ref. D23107WO01

[0055] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0056] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0057] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by a control system that includes one or more processors. The control system may be controlled by one or more computer programs. This document also describes various equations relevant to implementing aspects of the present disclosure. In general these equations may be implemented by a control system as processing functions, for example according to instructions stored on one or more non-transitory, computer-readable media.

[0058] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.Dolby Ref. D23107WO01

[0059] In this example, the apparatus 101 includes an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. The control system 110 may, in some implementations, be configured for receiving, via the interface system 105, input audio data in an input audio format. In some examples, the control system 110 may be configured for converting the input audio data to output audio data in an output audio format, the output audio format having a higher resolution than the input audio format. Various examples of this conversion process are disclosed herein. According to some examples, the control system 110 may be configured for storing the output audio data, for transmitting the output audio data via the interface system 105, or combinations thereof.

[0060] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1. However, the control system 110 may include a memory system in some instances.

[0061] The control system 110 may, for example, include a general purpose single- or multi- chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0062] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0063] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media.Dolby Ref. D23107WO01 Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1 and / or in the control system 110. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1.

[0064] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.

[0065] According to some implementations, the apparatus 101 may include the optional loudspeaker system 125 shown in Figure 1. The optional loudspeaker system 125 may include one or more loudspeakers. Loudspeakers may sometimes be referred to herein as “speakers.” In some examples, at least some loudspeakers of the optional loudspeaker system 125 may be arbitrarily located . For example, at least some speakers of the optional loudspeaker system 125 may be placed in locations that do not correspond to any standard prescribed speaker layout, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc. In some such examples, at least some loudspeakers of the optional loudspeaker system 125 may be placed in locations that are convenient to the space (e.g., in locations where there is space to accommodate the loudspeakers), but not in any standard prescribed loudspeaker layout.

[0066] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0067] In some implementations, the apparatus 101 may include the optional display system 135 shown in Figure 1. The optional display system 135 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the optional display system 135 may include one or more organic light-emitting diode (OLED) displays.Dolby Ref. D23107WO01 In some examples wherein the apparatus 101 includes the display system 135, the sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 135. According to some such implementations, the control system 110 may be configured for controlling the display system 135 to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0068] Generally, an acoustic scene may be represented in the form of a multi-channel audio signal, ^, consisting of ^*channels, with an associated panner-based audio format. Let us define: ^= ,^0^, ^^, ⋯ , ^%. / (4)

[0069] The multi-channel audiofrom a multi-microphone acoustic capture. For example, a 4-capsule tetrahedral microphone may be used to capture a 4-channel A-format signal, and this A-format signal may constitute the multi-channel audio signal, ^. Alternatively, an A-format signal may be transformed into a B-format signal, and this B- format signal may constitute the multi-channel audio signal, ^. The B-format signal is alternately known in the art as an Ambisonics signal. In alternative use scenarios, microphone arrays with multiple microphones capsules may be used to capture a multi-channel input signal, ^, that may be representative of a second, third, or higher order Ambisonics signal.

[0070] The multi-channel audio signal, ^, may be created from various audio processing systems or methods, including digital audio workstations or other audio editing systems, wherein the multi-channel signal is intended to represent acoustic elements, each of which are associated with a respective direction of arrival. Such audio input signals may include Ambisonics signals, higher-order-Ambisonics signals, 5-channel surround signals, 7-channel surround signals, or other known multi-channel formats intended to represent an acoustic scene.

[0071] The multi-channel audio signal, ^, may be received from an acoustic capture device, from an audio synthesis device, from an audio processing device, or from a decoder that operates to produce the multi-channel audio signal from an encoded stream that may have been stored and / or transmitted prior to decoding.Dolby Ref. D23107WO01

[0072] It will be appreciated that the definition of ^, in Equation 4, may be referring to individual time-domain samples (e.g. the variable named ^^may refer to ^^^1^ ∈ ℝ at someinstant in time, 1), or frequency-domain samples (e.g. the variable named ^^ may refer to^^^&, 1^ ∈ ℂ at frequency & around time 1).

[0073] The following disclosure may refer to time-domain or frequency-domain signals, and all references to audio signals (e.g. ^^) may be considered to be references to either time- domain signals (e.g. ^^^1^∈ ℝ) or frequency-domain signals (e.g. ^^^&, 1^∈ ℂ).

[0074] In some examples, the multi-channel audio signal, ^, consisting of ^*channels, is associated with a panner-based audio format. This implies that the multi-channel audio signal, ^, represents an acoustic scene that can be defined in terms of acoustic elements that are spatially distributed around a central listening position. Furthermore, each acoustic element contributes to ^ according to a panning function: 3*,^^^, ^, ^^^^

[0075] Hence, if an scene wherein object 6 ∈ 1.. ^5produces an acoustic signal, ^8^1^, incident at a central capture location, from a direction of arrival according to unit-vector 98= ^^8, ^:, ^8^, then according to the panning function defined in Equation 5, multi-channel input signal ^ will be: ^ = ∑%8=<^ ^8^1^^*^^8, ^:, ^8^ (6)

[0076] Some aspects of this disclosure involve deriving a new multi-channel audio signal, ^, consisting of ^+channels, with an associated panner-based audio format, where the multi- channel audio signal, ^, is intended to represent the same acoustic scene as the multi-channel audio signal, ^. Let us define: ^ =,^ , ^ , / 0^^⋯ , ^%>(7)

[0077] The panner-based audio format associated with the multi-channel audio signal, ^, is assumed to be defined in terms of the panning function:Dolby Ref. D23107WO01 3+,^^^, ^, ^^^^ = æ 3+,^^^, ^, ^^ö

[0078] Ideally, we wish tothat is a close match to the ideal target, ^?@ABC?^1^, which would have resulted from the same ^5acoustic objects, according to the panning function defined in Equation 8: ^?@ABC?= ∑%8=<^ ^8^1^^+^^8, ^:, ^8^ (9)

[0079] However, the9, if the original acoustic elements ^8^1^ and their associated directions of arrival, 98= ^^8, ^:, ^8^, are known. Some aspects of this disclosure involve determining a suitable multi-channel output audio signal, ^, in the absence of the full knowledge of the original acoustic elements.

[0080] Figure 3 shows another example of producing an output audio signal Y from an input audio signal X. In this example, the multi-channel output signal, ^, is derived using a linear mixture of the multi-channel input signal, ^, according to the D^+× ^*E matrix F, so that: ^ = F × ^ (10)

[0081] Hence, some aspects of this disclosure involve determining a suitable matrix, F. The matrix D may be determined in a number of relevant of art contexts. For example, the matrix D may be precomputed, stored in memory and retrieved from the memory for subsequent processing. In some examples, the matrix D may be received from a transmitted bit stream. Alternatively, the matrix D may be computed during processing, such as at setup and / or at run time. Other matrices discussed in this document may similarly be determined.

[0082] It will be appreciated that the matrix F may vary as a function of frequency and / or time. Without loss of generality, the following discussion will describe the method employed for the derivation of F at one particular time and for one frequency (or one frequency-band).

[0083] The multi-channel output signal, ^, may provide a higher-resolution representation of the same acoustic scene that was inherent in the multi-channel input signal, ^. The output ^ may be intended for playback over a speaker array to one or more listener, or for playbackDolby Ref. D23107WO01 over headphones. The output ^ may, in some examples, represent the acoustic scene in terms of a higher-order-Ambisonics format.

[0084] The output signal ^ may be intended to be further processed, or encoded, for subsequent storage and / or transmission. Subsequent processing, transmission, storage or playback of the signal ^ may be more efficient, or may result in an improved listening experience, compared to the input signal ^.

[0085] In some examples, the input signal ^ may be a 4-channel signal encoded as a first- order-Ambisonics signal, according to the panning function in Equation 1, and the output signal may be a 16-channel signal encoded as third-order-Ambisonics signal, according to the panning function in Equation 3.

[0086] In an alternative example, the input audio signal ^ may be a 9-channel signal encoded as a second-order-Ambisonics signal, according to the panning function in Equation 2, and the output audio signal Y may be a 36-channel signal encoded as fifth-order-Ambisonics signal.

[0087] It is known in the art to derive an estimate of the covariance of a multi-channel signal. For example, the covariance of the signal, ^, may be defined as: G** = HD^ × ^IE (11)

[0088] In Equation 11, the HD E operation defines the “expected value.” In practice, an estimate may be derived according to: G**=∑LLJ=L?JK∑?=?K^^&, 1^× ^^&, 1^I(12)

[0089] In Equation 12, the summation is performed over a region of time 1^..1^(in the neighbourhood of time 1) and frequency &^.. &^(in the neighbourhood of frequency &) to form an estimate of G**at time 1 and frequency &.

[0090] Other methods are known in the art, whereby the covariance, G**, may be derived at a given time and frequency.

[0091] Figure 2A shows an arrangement 200 wherein an input audio signal ^, shown in Figure 2A as 201, is processed by a mixing process 212 to form the output audio signal ^, shown in Figure 2A as 202. In this example, the mixing process 212 applies the mixingDolby Ref. D23107WO01 matrix F, 204, according to Equation 10. According to this example, the covariance generation process 210 forms the time-varying covariance G**, shown in Figure 2A as 203, from the input ^, 201. According to some disclosed examples, a time-varying matrix generation process 211 produces the time-varying matrix F from the covariance G**, as detailed below. In this example, the covariance generation process 210, the time-varying matrix generation process 211 and the mixing process 212 are implemented by an instance of the control system 110 of Figure 1. In some such examples, the covariance generation process 210, the time-varying matrix generation process 211 and the mixing process 212 may be implemented via instructions, such as software, stored on one or more computer-readable, non-transitory media.

[0092] According to some examples, we define a set of ^Mvirtual-source locations on theunit-sphere, ^NO ∈ "^: P = 1.. ^M^, where ^M > ^* and ^M > ^+. In a further example, ^M ≫^* and ^M ≫ ^+. In other words, according to some examples the resolution of the virtual-source locations is greater than the resolution of the input audio data X or the resolution of the output audio data Y. In some examples, the resolution of the virtual-source locations may be much greater than the resolution of the input audio data X or the resolution of the output audio data Y. For example, the resolution of the virtual-source locations may, in some examples, be at least an order of magnitude (10x) greater than the resolution of the input audio data X or the resolution of the output audio data Y. It can be advantageous for the virtual-source locations to be approximately evenly distributed over the surface of the unit- sphere. The virtual-source locations also may be referred to herein as virtual loudspeaker locations. Audio signals corresponding to virtual loudspeakers at the virtual loudspeaker locations may be referred to herein as “virtual loudspeaker signals.”

[0093] Figure 4 shows graphic representations of input audio signals X, output audio signals Y and virtual sources according to one example. In this example, the input audio signal ^ is a 9-channel signal encoded as a second-order-Ambisonics (HOA2S) signal, the output audio signal Y is a 36-channel signal encoded as fifth-order-Ambisonics (HOA5S) signal, and there are 642 virtual-source locations S. Accordingly, in this example the resolution of the virtual- source locations is more than an order of magnitude (10x) greater than the resolution of the input audio data X or the resolution of the output audio data Y. One will observe that while the sizes of the blocks X, Y, and S generally represent the numbers of elements of X, Y, and S, the blocks X, Y, and S are not drawn to scale. Other examples may include different numbersDolby Ref. D23107WO01 of elements (e.g., channels) of the input and output audio signals, different numbers of virtual-source locations S, or combinations thereof. In one such example, the output audio data format may be another audio format, such as a Dolby 7.1.4 audio format, a Dolby 9.1 audio format, etc.

[0094] Figure 5 shows examples of two useful matrices. One is the D^*× ^ME matrix, )*, which describes the transfer function from ^Macoustic sources to the ^*channels of a signal, as per the panner-based format of the input signal, according to: )* = ,^*^N^^ ^*^N^^ ⋯ ^*,N%S / / (13)

[0095] In the exampleand X rows, where “X” represents the number of elements (such as channels) of the input audio signal X and S represents the number of virtual loudspeaker locations. The matrix )*may be referred to herein as a matrix corresponding to panning rules for the input audio data. Likewise, we may define theD^+× ^MEmatrix, )+, according to: )+= ,^+^N^^^+^N^^⋯ ^+,N%S / / (14)

[0096] The matrix )+may as a for the output audio data. In the example shown in Figure 5, the matrix, )+is represented as having S columns and Y rows, where “Y” represents the number of elements (such as channels) of the output audio signal Y. Figure 6A shows the relationship between the input audio signal X and )*. Figure 6B shows the relationship between the output audio signals Y and )+. If one had complete knowledge of the virtual loudspeaker signals corresponding to S, one would be able to compute Y in a straightforward manner. However, one would generally not have advance knowledge of the virtual loudspeaker signals corresponding to S.

[0097] Now, we can define the D^M× ^*E passive virtual-source decode matrix, T*M′, according to: T*M′ = )*V(15)

[0098] In Equation 15, the ▫Voperator indicates the Moore-Penrose pseudo-inverse, as is known in the art. Figure 7 illustrates one method of computing a passive virtual-source decode matrix. According to this example, an input audio bitstream 705 corresponding to theDolby Ref. D23107WO01 input audio signal X is being processed. In this example, )*V= )*Yx ^)*x )*Y^[^, wherethe operator ▫I indicates the Hermitian transpose. In some disclosed examples, the matrixT*M′ may be defined by alternative means. According to some such examples, the intentionis to ensure that )*× T*M′ ≈ ]%., where ]%.is the D^*× ^*E identity matrix.

[0099] According to some examples, the input signal ^ may be “decoded” to a set of ^Mvirtual speaker signals, " = T*M′ × ^, and the virtual speaker signals may then be mapped to an output signal, ^′ = )+× ", so that the resulting signal, ^′, may be defined according to the fixed matrix, F′ = )+× T*M′, so that ^′ = F′ × ^. However, some such methods have proven to be sub-optimal, in part because such methods tend to spread signal energy across a larger number of virtual-source locations than would be desirable.

[0100] Figure 8 illustrates one method of computing an improved virtual-source decode matrix. In this example, an input audio bitstream 805 corresponding to the input audio signal X is being processed. According to this example, an improved matrix, T*Mmay be formed according to: T*M= H × )*I×^)*× H × )*I^[^(16)

[0101] In Figure 8 and Equation 16, the D^M× ^ME matrix, H, is a diagonal matrix, wherein each of the ^Melements along the diagonal of H define a target power assigned to each one of the respective ^Mvirtual sources.

[0102] We may form the energy estimation matrix, H, according to: H = ^T*M′ × G**× T*M′I^ ∘ ]%S (17)

[0103] In Equation 17, the ∘ operator implies the Hadamard (element by element) product, and ]%Srepresents the D^M× ^ME identity matrix. This implies that H is an D^M× ^ME diagonal matrix (with real, positive entries).

[0104] Figure 9 shows blocks of a complete upmixer according to one example. In Figure 9, the matrices B and A are formed as follows: _ = )+× H × )*I(18) andDolby Ref. D23107WO01 `= )* × H × )*I (19)

[0105] Matrix A may be referred to herein as an “input format covariance matrix.” Matrix B may be referred to herein as an “output format to input format cross covariance matrix.” In some examples, we can compute the matrix F as: F= _ × `[^ (20)

[0106] In some examples, referring again to Figure 2A, the time-varying matrix generation process 211 may be performed according to the methods of Equations 17–20. It will be appreciated that, whilst the matrices )*, )+and T*M′ are static, the matrix G**, 203, will vary as a function of the input signal ^, 201, and hence the matrix F, shown in Figure 2A as 204, will vary over time as a function of G**.

[0107] The output signal format may be a super-set of the input signal format. For example, when the input signal is a 4-channel first-order-Ambisonics signal, and the output signal is a 16-channel third-order-Ambisonics signal, the first 4 channels of the output are expected to be identical to the input signal, as is known in the art. Figure 2B shows a process of generating a higher-order-Ambisonics (HOA) signal according to some examples. According to one embodiment, as shown in Figure 2B, a 16-channel higher-order-Ambisonics signal, 220, may be formed by combining the 4-channel input signal ^, 201, with the 12-channel output ^, 202, of the output signal generation process 251. In this example, the output signal generation process 251 is implemented by an instance of the control system 110 of Figure 1. The output signal Y generation process 251 may be implemented according to various methods described herein.

[0108] According to some examples of the process shown in Figure 2B, the output audio data format may be defined according to a panning function that is equivalent to the panning function of the desired higher-order-Ambisonics format, with the first ^*elements removed. By way of example, when the input format is defined according to the first-order-Ambisonics panning function of Equation 1, and the output signal 220 is defined according to the third- order-Ambisonics panning function of Equation 3, the signal ^, 220, may be defined by the 12-channel panning function:Dolby Ref. D23107WO01 √3^^æ √3^^ ö

[0109] In Equation 21,panning function with the first-order components removed. Improved efficiency

[0110] The method described above derives matrices _ and ` that are D^+ × ^*E andD^*× ^*E in size, respectively. If there is a reasonable number of channels in the input signal^ and output signal ^, then we can expect these matrices (_ and `) to be reasonable in size.

[0111] However, the procedure for computing _ and ` involves (potentially large) matricesH, )* ans )+, with sizes that are dependent on the number of virtual-sources, ^M.

[0112] We may simplify the calculation of the matrices _ and ` by noting that: _ = )+× a^T′*M× G**× T*bIM^∘ ]%Sc × )*I(22)and hence, we can form the matrix B by summing over the elements of G**as follows: _ =∑d,e)+× a,T′*M× ,G**∘ fd,e / × T*bIM / ∘ ]%Sc × )*I(23)

[0113] In Equation (23), fd,e represents the D^* × ^*E single-entry matrix with 1 at element^g, h^ and zero elsewhere. We can re-write this as:Dolby Ref. D23107WO01 _ = ∑d,e{ G**}d,ekd,e(24)

[0114] In Equation (23),of G**and: kd,e= )+× a,T′*M× fd,e× T′*IM / ∘ ]%Sc × )*I(25)

[0115] Likewise,` =∑d,e{ G**}d,e"d,e(26) , where: "d,e= )*× a,T′*M× fd,e× T′*IM / ∘ ]%Sc × )*I(27)

[0116] A set of,kd,e∈ ℂ%>×%.: g = 1.. ^*, h = 1.. ^* / and D^*× ^*E matrices, ,"d,e∈ ℂ%.×%.: g = 1.. ^*, h = 1.. ^* / may be pre-_ to be computed more efficiently, with computation complexity that is not dependent in ^M.

[0117] In other words, according to some examples the sets of D^+× ^*E and D^*× ^*E matrices may be computed—for example, by an instance of the control system 110 of Figure 1—prior to “run-time” operations during which input multi-channel audio signal X is converted into output multi-channel audio signal Y. Such implementations may allow run- time operations to proceed more quickly, to require less memory, to require less computational resources, or combinations thereof. In some examples in which the input and output audio formats (the formats of X and Y) are known, the S and T matrices may also be computed prior to run-time operations. According to some such examples, during run-time, the S and T matrices may be combined with the Rxxmatrix—for example, by an instance of the control system 110 of Figure 1—to compute matrices A and B, for example according to Equations 24 and 26. In some such examples, during run-time operations, matrices A and B may then be used to compute the matrix D, for example as shown in Equation 20. The matrix D may then be used to convert X into Y, for example as shown in Equation 10.Dolby Ref. D23107WO01

[0118] In some alternative implementations, one or more Gxand Gymatrices may be computed prior to run-time operations, along with T′*M(the “passive decode” matrix associated with Gx). Such implementations may allow run-time operations to proceed more quickly, to require less memory, to require less computational resources, or combinations thereof. Some such implementations may support multiple input formats (e.g., various A- format and / or Ambisonics formats and / or other multi-channel formats). For each input format, an associated Gx matrix and T′*Mmatrix may be pre-computed and stored. Some such implementations may support multiple output formats (e.g., Ambisonics formats and / or other multi-channel formats). For each output format, an associated Gy matrix may be pre- computed and stored. When the particular input and output formats are known—for example, as indicated by user input—the S and T matrices may be computed, for example by using Equations 25 and 27.

[0119] According to some alternative implementations, the Gxand / or Gymatrices may be computed when they are needed. In some such examples, the Gx and / or Gy matrices may be defined in terms of functions, such as the Ambisonic panning functions of Equation 2 or Equation 3. According to some examples, the Gx and Gy matrices may be computed according to Equation 13 or Equation 14. In such cases, the virtual object locations, Vs, must be known. For example, the virtual object locations may be pre-stored or determined by other means.

[0120] In some implementations, even if none of the foregoing matrices have been computed prior to run-time operations, a control system may, prior to the commencement of run-time audio processing, compute the S and T matrices by (1) defining the Vsvirtual object locations,(2) computing the Px() and Py() panning functions to form Gx and Gy, and (3) computing theT′*M passive decode matrix (by computing the Moore-Penrose pseudo-inverse of Gx).

[0121] According to some examples, the matrices kd,eand / or "d,emay be real-valued, rather than complex-valued.

[0122] Further efficiencies may be achieved by exploiting the fact that the matrix G**is known to be Hermitian, so that summation over ^g, h^ may be performed on the upper (or lower) triangular parts of G**—in other words, the portions of G**that are above or below the diagonal—for example. Other efficiencies may be gained by exploiting the Hermitian nature of `, so that the upper (or lower) part only of ` may be computed.Dolby Ref. D23107WO01

[0123] The methods described above may provide an improved output signal, ^, compared to previously-disclosed methods. However, further improvements may be made by the addition of some decorrelated signals, derived from the input, ^, as described below. Decorrelation

[0124] Figure 10 shows blocks that represent example processes according to a further embodiment. According to this example, the input audio signal ^, shown in Figure 10 as 201, is processed by a mixing process 212 to form the up-mixed signals 240. In this example, up-mixed signals 240 are equivalent to Y in the prior descriptions, including but not limited to the Y indicated in Figure 2A, which is output signal 202 in that example. According to this example, the covariance generation process 210 forms the time-varying covariance G**, shown in Figure 10 as element 203, from the input ^, shown as element 201. Here, the time-varying matrix generation process 211 produces the time-varying matrix F, shown as element 204, from the covariance G**. In the example shown in Figure 10, the mixing process 212 applies the mixing matrix F, 204, according to Equation 10. The elements 201, 203, 204, 210, 211 and 212 of Figure 10 may, in some examples, be instances of the corresponding elements of Figure 2A.

[0125] According to this example, blocks 210, 211, 212, 230, 232 and 234, and mixer 236, are implemented by an instance of the control system 110 of Figure 1. According to this example, element 235 is an additional set of signals that are produced by block 234 and are in the same format as Y, containing decorrelated signals that are suitable for adding, via the mixer 236, some decorrelated elements to the resulting combined signal 1002. In this example, the mixer 236 is configured to mix the up-mixed signals 240 and decorrelated signals 235, to produce the combined signal 1002. According to the arrangement 300 of Figure 10, a number ^lof additional decorrelator input signals 231 are derived as inputs to a decorrelation process 232, from input audio signals ^, which are also represented by element 201 in Figure 10, by the mixing process 230. In some examples, the mixing process 230 may derive the decorrelator input signals 231 as follows: m = n × ^G × `[^^ × ^ (28) In Equation (28), the o^l× ^*p matrix, G is chosen so as to extract a suitable subset of the input signal, ^. In one embodiment, where the input signal format is an Ambisonics format, wherein the first 4 channels of the input signal constitute a first-order Ambisonics signal, weDolby Ref. D23107WO01 may set ^l= 4 and G is theo^l× ^*pmatrix with all zero entries except for 1’s in the diagonal elements.

[0126] In some examples, the scale-factor n, in Equation 28, may be chosen to vary proportionally with the magnitude of the covariance, G**. In addition, the scale-factor n may vary so as to be larger when the input signal ^ is determined to be representative of a diffuse acoustic scene. In one example, we may determine q as follows: n = q × ^1q^G**^ − ||G**||s^ (29) In Equation 29, the 1q^ ^function indicates the trace of a matrix, and the || ||soperator indicates the Frobenius norm of a matrix. The matrix, n ×^G × `[^^, which is an example of the element 239 of Figure 10, may be formed by the time-varying matrix generation process 211 and provided as input to the mixing process 230.

[0127] In one embodiment, a decorrelation process takes, as input, the ^l-channel audio signal m, and produces an ^s-channel decorrelator output signal, t, that contains decorrelated signals derived from m, according to: t = uvwxq^m^. Other examples may implement various other methods for implementing decorrelation processes that are known in the art.

[0128] Figure 11 shows an example of the decorrelation process of Figure 10. Figure 11 shows an arrangement 232 of decorrelator processes 281, each adapted to receive a signal 282 from the decorrelator input signals 231, to produce a set of decorrelator output signals 285, 286 and 287, to form the decorrelator output signals 233.

[0129] According to some examples, the decorrelator output signals 233, represented in Equation 30 as t, may then be added to the mixed signal F × ^, so that an alternative version of Equation 7 may be used to firm an augmented output signal, ^: ^ = F × ^ + ) × t (30)

[0130] The scale factor q, in Equation 29, may be chosen so as to scale the decorrelator input signals, m, so as to ensure that decorrelator contribution, ) × t of Equation 30, is scaled so as to provide a suitable amount of decorrelation contribution. Choice of a suitable scale factor, q, may be made via subjective listening evaluations.Dolby Ref. D23107WO01

[0131] Referring again to Figure 10, the decorrelator output signals 233 (represented in Equation 30 as F) may have a larger magnitude when the input signal X is determined to be representative of a diffuse acoustic scene. Furthermore, it may commonly occur that, when the input signal X is representative of a diffuse acoustic scene, the up-mixed signals 240 output by the mixing process 212 will not contain the correct energy in some of the higher resolution components. For example, when the input format is the 4-channel First Order Ambisonics format, and the output format is the 16-channel 3rd-order Ambisonics format, and when the input X is representative of a diffuse sound-field, the up-mixed signals 240 output by the mixing process 212 will lack significant energy in the 12 channels that constitute the 2nd and 3rd order components of the output format. In these higher-resolution channels of the output format, in some examples the decorrelator output signals 233, F, may be used to replace this missing energy. The mapping of the decorrelator output signals 233, F, to the output format may be achieved according to the [NYx NF] matrix G that is applied in block 234.

[0132] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1200, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1200 may be performed concurrently. Moreover, some implementations of method 1200 may include more or fewer blocks than shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) a control system such as the control system 110 that is shown in Figure 1 and described above.

[0133] In this example, Figure 12 shows the method 1200 whereby an output signal ^ is formed from an input signal ^. In this example, block 1205 involves determining, by a control system, the covariance of the input signal. Block 1205 may, for example, involve implementing Equation 11, implementing Equation 12, or implementing another covariance determining method. According to this example, block 1210 involves determining, by the control system, _ and ` matrices. Block 1210 may involve any of the disclosed techniques for determining the _ and ` matrices, such as those described with reference to Figure 9. In some examples, block 1210 may involve implementing Equation 18, 19, 22, 23, 24 or 26. In this example, block 1215 involves determining, by the control system, the mixing matrix F. According to this example, block 1210 involves implementing Equation 20. According to this example, block 1220 involves applying, by the control system, the mixing matrix F toDolby Ref. D23107WO01 the input signal ^ to form the output signal, which is, or includes, the output audio data Y in this instance. In this example, block 1225 involves storing or transmitting, by the control system, the output audio data Y.

[0134] Figure 13 is a flow diagram that outlines another example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1300, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1300 may be performed concurrently. Moreover, some implementations of method 1300 may include more or fewer blocks than shown and / or described. The blocks of method 1300 may be performed by one or more devices, which may be (or may include) a control system such as the control system 110 that is shown in Figure 1 and described above.

[0135] In this example, method 1300 is a method for processing input audio data. According to this example, block 1305 involves receiving, by a control system that includes one or more processors, the input audio data in an input audio format. The input audio data may be what is referred to herein as X. The input audio format may be any one of various types of audio formats. According to some examples, the input audio format may be a panner-based audio format such as Ambisonics.

[0136] According to this example, block 1310 involves converting, by the control system, the input audio data to output audio data in an output audio format. The processes of block 1310 may be broken into sub-steps and performed in parallel and / or series. In this example, the output audio format has a higher resolution than the input audio format. The output audio data may be what is referred to herein as Y. The output audio format may be any one of various types of audio formats. According to some examples, the output audio format may be a panner-based audio format such as Ambisonics.

[0137] In this example, the converting comprises applying, by the control system, a biased decoding matrix to the input audio data to produce the output audio data. The biased decoding matrix may be an instance of the matrix D that is disclosed herein, for example as described herein with reference to Figures 3 et seq.

[0138] In this example, the biased decoding matrix is biased according to an estimated energy distribution of the input audio data. The energy distribution of the input audio data may, for example, be estimated by implementing Equation 17. According to this example, the biased decoding matrix is, or includes, a combination of constant matrices and a variable matrix. In this example, the variable matrix is, or includes, a covariance matrixDolby Ref. D23107WO01 corresponding to the input audio data. According to this example, the biased decoding matrix varies over time as a function of the covariance matrix. In some examples, the constant matrices may include a matrix (such as )*) corresponding to panning rules for the input audio data and a matrix (such as )+) corresponding to panning rules for the output audio data. As noted elsewhere herein, the matrices )*, )+and T*M′ are static, whereas the covariance matrix G**will vary as a function of the input signal ^, and therefore the matrix F will vary over time as a function of G**.

[0139] In this example, block 1315 involves storing or transmitting the output audio data. In some examples, method 1300 may involve providing the output audio data to a loudspeaker- enabled device or a loudspeaker-enabled system for playback.

[0140] According to some examples, the biased decoding matrix may be based, at least in part, on a combination of a version of an A matrix and a B matrix. The A matrix may be an input format covariance matrix and the B matrix may be an output format to input format cross covariance matrix. The version of the A matrix may be an inverse of the A matrix, for example as indicated in Equation 20.

[0141] In some examples, the A matrix may be a biased input format covariance matrix and the B matrix may be a biased output format to input format cross covariance matrix. According to some examples, the A matrix and the B matrix may be biased according to the estimated energy distribution of the input audio data. In some such examples, the estimated energy distribution of the input audio data is based on an energy estimation matrix. The energy estimation matrix may be based in part on a passive virtual source decode matrix. As noted in Equation 17 and the corresponding discussion, the passive virtual source decode matrix may be based, at least in part, on a pseudo-inverse of a matrix used for decoding the input audio data to a set of virtual loudspeaker signals.

[0142] According to some examples, the resolution of the input audio format may correspond to a number of channels of the input audio format or an Ambisonic order of the input audio format and the resolution of the output audio format may correspond to a number of channels of the output audio format or an Ambisonic order of the output audio format. In some examples, the output audio format may have a higher number of elements than the input audio format. The number of elements may, for example, correspond to a number of channels, such as a number of channels corresponding with an Ambisonics order (such as HOA2, HOA5, etc.).Dolby Ref. D23107WO01

[0143] Some examples of method 1300 may involve producing a set of virtual loudspeaker signals. The set of virtual loudspeaker signals may correspond to the set of ^Mvirtual-source locations on a unit sphere as disclosed herein. In this example, ^M> ^*and ^M> ^+. In some examples, ^M≫ ^*and ^M≫ ^+. Accordingly, in this example, the set of virtual loudspeaker signals has a higher resolution than the a resolution of the input audio format or the output audio format. In some examples, the set of virtual loudspeaker signals may be produced prior to the operations of blocks 1305, 1310 and 1315. According to some examples, the operations of blocks 1305, 1310 and 1315 may be runtime operations and the set of virtual loudspeaker signals may be produced prior to the runtime operations.

[0144] In some such examples, the set of virtual loudspeaker signals may have a number of elements than is more than ten times greater than the number of elements of the input audio format or the number of elements of the output audio format.

[0145] In some examples, the covariance matrix may be a Hermitian matrix. Applying the biased decoding matrix may involve performing a summation of elements above or below a diagonal of the Hermitian matrix.

[0146] According to some examples, the input audio data may be, or may include, one of a plurality of band-limited components of a wider-bandwidth input audio data. The output audio data may be, or may include, one of a plurality of band-limited components of a wider- bandwidth output audio data. Implementation Details

[0147] Figure 14A illustrates a schematic block diagram of an example device architecture 1400 (e.g., an apparatus 1400) that may be used to implement various aspects of the present disclosure. Architecture 1400 includes but is not limited to servers and client devices, systems, and methods as described in reference to Figures 1–13. As shown, the architecture 1400 includes central processing unit (CPU) 1401 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 1402 or a program loaded from, for example, storage unit 1408 to random access memory (RAM) 1403. The CPU 1401 may be, for example, an electronic processor 1401. In some implementations, the CPU 1401 may be an instance of the control system 110 of Figure 1. In RAM 1403, the data required when CPU 1401 performs the various processes is also stored, as required. CPU 1401, ROM 1402, and RAM 1403 are connected to one another via bus 1404. Input / output interface 1405 is also connected to bus 1404.Dolby Ref. D23107WO01

[0148] The following components are connected to I / O interface 1405: input unit 1406, that may include a keyboard, a mouse, or the like; output unit 1407 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 1408 including a hard disk, or another suitable storage device; and communication unit 1409 including a network interface card such as a network card (e.g., wired or wireless).

[0149] In some implementations, input unit 1406 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0150] In some implementations, output unit 1407 include systems with various number of speakers. Output unit 1407 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0151] In some embodiments, communication unit 1409 is configured to communicate with other devices (e.g., via a network). Drive 1410 is also connected to I / O interface 1405, as required. Removable medium 1411, such as a magnetic disk, an optical disk, a magneto- optical disk, a flash drive or another suitable removable medium is mounted on drive 1410, so that a computer program read therefrom is installed into storage unit 1408, as required. A person skilled in the art would understand that although apparatus 1400 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0152] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer- readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 1409, and / or installed from the removable medium 1411, as shown in Figure 14A.

[0153] Figure 14B illustrates a schematic block diagram of an example CPU 1401 implemented in the device architecture 1400 of Figure 14A that may be used to implementDolby Ref. D23107WO01 various aspects of the present disclosure. The CPU 1401 includes an electronic processor 1420 and a memory 1421. The electronic processor 1420 is electrically and / or communicatively connected to the memory 1421 for bidirectional communication. The memory 1421 stores encoding software 1422 and decoding software 1423. The memory 1421 may be, for example, a ROM, a RAM, or another non-transitory computer readable medium. The electronic processor 1420 may implement the encoding software 1422 stored in the memory 1421 to perform, among other things, the method 1400 of Figure 14. Additionally, the electronic processor 1420 may implement the decoding software 1423 stored in the memory 1421 to perform, among other things, the methods and processes described in connection with Figures 1-13.

[0154] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., CPU 1401 in combination with other components of Figure 14A), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device (e.g., control circuitry). While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non- limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0155] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0156] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instructionDolby Ref. D23107WO01 execution system, apparatus, or device. The machine-readable medium may be a machine- readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0158] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

[0159] Various aspects of the present invention may be appreciated from the following enumerated example embodiments (EEEs): EEE1. A method for processing input audio data, the method comprising:Dolby Ref. D23107WO01 receiving, by a control system that includes one or more processors, the input audio data in an input audio format; converting, by the control system, the input audio data to output audio data in an output audio format, the output audio format having a higher resolution than the input audio format; and storing or transmitting the output audio data, wherein: the converting comprises applying, by the control system, a biased decoding matrix to the input audio data to produce the output audio data; the biased decoding matrix is biased according to an estimated energy distribution of the input audio data; the biased decoding matrix comprises a combination of constant matrices and a variable matrix; the variable matrix comprises a covariance matrix corresponding to the input audio data; and the biased decoding matrix varies over time as a function of the covariance matrix. EEE2. The audio processing method of EEE1, wherein the biased decoding matrix is based, at least in part, on a combination of a version of an A matrix and a B matrix, the A matrix being an input format covariance matrix and the B matrix being an output format to input format cross covariance matrix. EEE3. The audio processing method of EEE2, wherein the A matrix is a biased input format covariance matrix and the B matrix is a biased output format to input format cross covariance matrix and wherein the A matrix and the B matrix are biased according to the estimated energy distribution of the input audio data. EEE4. The audio processing method of EEE3, wherein the estimated energy distribution of the input audio data is based on an energy estimation matrix and wherein the energy estimation matrix is based in part on a passive virtual source decode matrix. EEE5. The audio processing method of EEE4, wherein the passive virtual source decode matrix is based, at least in part, on a pseudo-inverse of a matrix used for decoding the input audio data to a set of virtual loudspeaker signals.Dolby Ref. D23107WO01 EEE6. The audio processing method of any one of EEEs 2–5, wherein the version of the A matrix is an inverse of the A matrix. EEE7. The audio processing method of any one of EEEs 1–6, wherein the constant matrices include a matrix corresponding to panning rules for the input audio data. EEE8. The audio processing method of any one of EEEs 1–7, wherein the constant matrices include a matrix corresponding to panning rules for the output audio data. EEE9. The audio processing method of any one of EEEs 1–8, wherein the output audio format has a higher number of elements than the input audio format, the number of elements corresponding to a number of channels or an Ambisonics order. EEE10. The audio processing method of any one of EEEs 1–9, wherein the resolution of the input audio format corresponds to a number of channels of the input audio format or an Ambisonic order of the input audio format, and wherein the resolution of the output audio format corresponds to a number of channels of the output audio format or an Ambisonic order of the output audio format. EEE11. The audio processing method of any one of EEEs 1–10, wherein the covariance matrix is a Hermitian matrix and wherein applying the biased decoding matrix involves performing a summation of elements above or below a diagonal of the Hermitian matrix. EEE12. The audio processing method of any one of EEEs 1–11, further comprising providing the output audio data to a loudspeaker-enabled device or a loudspeaker-enabled system for playback. EEE13. The audio processing method of any one of EEEs 1–12, wherein the input audio format is an Ambisonics format. EEE14. The audio processing method of any one of EEEs 1–13, wherein the input audio data comprises one of a plurality of band-limited components of a wider-bandwidth input audio data, and wherein the output audio data comprises one of a plurality of band-limited components of a wider-bandwidth output audio data. EEE15. The audio processing method of any one of EEEs 1–14, further comprising producing a set of virtual loudspeaker signals, the set of virtual loudspeaker signals having a higher resolution than a resolution of the input audio format or the output audio format.Dolby Ref. D23107WO01 EEE16. The method of EEE 15, wherein the set of virtual loudspeaker signals is produced prior to the operations of claim 1. EEE17. The method of EEE 15 or EEE 16, wherein the operations of claim 1 are runtime operations and wherein the set of virtual loudspeaker signals is produced prior to the runtime operations. EEE18. An apparatus configured to perform the audio processing method of any one of EEEs 1–17. EEE19. A system configured to perform the audio processing method of any one of EEEs 1– 17. EEE20. One or more computer-readable non-transitory media having instructions stored thereon for controlling one or more devices to perform the audio processing method of any one of EEEs 1–17.

Claims

Dolby Ref. D23107WO01 CLAIMS What Is Claimed Is:

1. A method for processing input audio data, the method comprising: receiving, by a control system that includes one or more processors, the input audio data in an input audio format; converting, by the control system, the input audio data to output audio data in an output audio format, the output audio format having a higher resolution than the input audio format, wherein the resolution corresponds to a number of channels; and storing or transmitting the output audio data, wherein: the converting comprises applying, by the control system, a biased decoding matrix to the input audio data to produce the output audio data; the biased decoding matrix is biased according to an estimated energy distribution of the input audio data; the biased decoding matrix comprises a combination of constant matrices and a variable matrix; the variable matrix comprises a covariance matrix corresponding to the input audio data and the constant matrices comprise a matrix corresponding to panning rules for the input audio data and a matrix corresponding to panning rules for the output audio data; and the biased decoding matrix varies over time as a function of the covariance matrix.

2. The audio processing method of claim 1, wherein the biased decoding matrix is based, at least in part, on a combination of a version of an A matrix and a B matrix, the A matrix being an input format covariance matrix and the B matrix being an output format to input format cross covariance matrix.

3. The audio processing method of claim 2, wherein the A matrix is a biased input format covariance matrix and the B matrix is a biased output format to input format cross covariance matrix and wherein the A matrix and the B matrix are biased according to the estimated energy distribution of the input audio data.Dolby Ref. D23107WO01 4. The audio processing method of claim 3, wherein the estimated energy distribution of the input audio data is based on an energy estimation matrix and wherein the energy estimation matrix is based in part on a passive virtual source decode matrix.

5. The audio processing method of claim 4, wherein the passive virtual source decode matrix is based, at least in part, on a pseudo-inverse of a matrix used for decoding the input audio data to a set of virtual loudspeaker signals.

6. The audio processing method of any one of claims 2–5, wherein the version of the A matrix is an inverse of the A matrix.

7. The audio processing method of any one of claims 1–6, wherein the covariance matrix is a Hermitian matrix and wherein applying the biased decoding matrix involves performing a summation of elements above or below a diagonal of the Hermitian matrix.

8. The audio processing method of any one of claims 1–7, further comprising providing the output audio data to a loudspeaker-enabled device or a loudspeaker-enabled system for playback.

9. The audio processing method of any one of claims 1–8, wherein the input audio format is an Ambisonics format.

10. The audio processing method of any one of claims 1–9, wherein the input audio data comprises one of a plurality of band-limited components of a wider-bandwidth input audio data, and wherein the output audio data comprises one of a plurality of band-limited components of a wider-bandwidth output audio data.

11. The audio processing method of any one of claims 1–10, further comprising producing a set of virtual loudspeaker signals, the set of virtual loudspeaker signals having a higher resolution than a resolution of the input audio format or the output audio format.

12. The method of claim 11, wherein the set of virtual loudspeaker signals is produced prior to the operations of claim 1.

13. The method of claim 11 or claim 12, wherein the operations of claim 1 are runtime operations and wherein the set of virtual loudspeaker signals is produced prior to the runtime operations.Dolby Ref. D23107WO01 14. An apparatus configured to perform the audio processing method of any one of claims 1–13.

15. A system configured to perform the audio processing method of any one of claims 1– 13.

16. One or more computer-readable non-transitory media having instructions stored thereon for controlling one or more devices to perform the audio processing method of any one of claims 1–13.

Images