Universal spatial audio format transcoder

Abstract

The disclosure relates to methods of determining an optimized conversion matrix for converting between an input spatial audio format and an output spatial audio format. The input spatial audio format is specified by an encoding scheme for encoding a first set of directions to the input spatial audio format and the output spatial audio format is specified by a decoding scheme for decoding the output spatial audio format to an output layout The method includes optimizing the conversion matrix based on a cost function. The cost function is based on parameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and the conversion matrix. The disclosure further relates to corresponding apparatus, computer programs, and computer-readable storage media.

Description

UNIVERSAL SPATIAL AUDIO FORMAT TRANSCODER Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from Spanish Application No. P202330824, filed on 29 September 2023, US Provisional Application No. 63 / 606,751 filed on 6 December 2023, European Application No. 23214758.7 filed on 6 December 2023, and US Provisional Application No.63 / 557,948 filed on 26 February 2024, each of which is incorporated by reference herein in its entirety. Technical Field

[0002] The present disclosure relates to techniques for determining an optimized conversion matrix (e.g., transcoding matrix) for converting between an input spatial audio format and an output spatial audio format. In particular, the present disclosure relates to techniques for determining a universal conversion matrix that is independent of the specific audio signals and only depends on details of the input spatial format and the output spatial format. Background

[0003] Spatial audio signals can come in very diverse formats. Some of these formats can be layout-independent, like Ambisonics (e.g., Zotter & Frank, 2019) or Spherical Wavelet Format (SWF) (e.g., Scaini & Arteaga, 2020), whereas some formats can be tailored to a specific loudspeaker layout, like multichannel formats such as 5.1, 7.1, or 7.1.4. Additionally, there can be spatial audio formats directly associated to a recording setup, such as the direct recording of a microphone array.

[0004] Layout-independent spatial audio formats, such as Ambisonics or SWF, cannot be reproduced directly to a loudspeaker setup; they must be first decoded to the specific loudspeaker layout. Oftentimes, the process of decoding those formats to a loudspeaker layout is not straightforward and state of the art techniques often involve decoding to virtual intermediate loudspeaker layouts (e.g., Zotter & Frank 2012; Zotter & Frank, 2018), or optimizing the decoding based on psychoacoustic criteria (e.g., Scaini & Arteaga, 2014). Likewise, spatial audio formats associated to recording setups cannot be reproduced directly, but they must be transformed, either to a multichannel format suitable for direct reproduction, or to anintermediate spatial format, such as Ambisonics. This transformation can be based on the knowledge of the principles of the acoustic field (e.g., Blanco Galindo, Coleman, & Jackson, 2020), on beamforming techniques (e.g., Blanco Galindo, Coleman, & Jackson, 2020), or on established stereo mixing principles (e.g., Leal & Elmar, 1991). Finally, even if multichannel formats can be reproduced directly to the loudspeaker setup if the intended reproduction layout is readily available, it is often the case that the reproduction layout differs from the multichannel format configuration (such as reproducing a 7.1 mix on a 5.1 setup, for example), or that the reproduction layout does not conform to the intended layout specification (such as reproducing a 7.1 mix on a 7.1 layout with the loudspeakers placed on irregular locations, for example).

[0005] There is thus need for improved techniques that are able, given any input spatial audio format and output spatial audio format, to generate an optimal conversion matrix (e.g., transcoding matrix) from the input format to the output format. Summary

[0006] In view of this need, the present disclosure provides methods of determining an optimized conversion matrix for converting between an input spatial audio format and an output spatial audio format, as well as corresponding apparatus, computer programs, and computer- readable storage media, having the features of the respective independent claims.

[0007] One aspect of the present disclosure relates to a method (e.g., computer-implemented method) of determining an optimized conversion matrix for converting between an input spatial audio format and an output spatial audio format. Conversion between formats may relate to transcoding, for example. Accordingly, the conversion matrix may be a transcoding matrix in some examples. The input spatial audio format may be specified by an encoding scheme for encoding a first set of directions to the input spatial audio format. The output spatial audio format may be specified by a decoding scheme for decoding the output spatial audio format to an output layout. The method may include optimizing the conversion matrix based on a cost function. Therein, the cost function may be based on parameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and the conversion matrix. Specifically, the cost function may be based on parameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and a parameterization of the conversion matrix (e.g., coefficients / elements of the conversion matrix, etc.).

[0008] Thereby, given the input format and the output format, the proposed method can provide an optimal conversion (e.g., transcoding) matrix tailored to the specific setup. This can be done independently of actual audio that is encoded, transcoded, and / or decoded, to provide a universal conversion matrix for given, but otherwise arbitrary, input and output schemes. Adaptation of the cost function, for example by adjusting (directional) weights can moreover cater to users’ preferences and / or specific implementations.

[0009] In some embodiments, the input spatial audio format may be specified by the first set of directions and an encoding matrix for encoding input audio signals from respective ones of the first set of directions to channels of the spatial input format. The first set of directions may be a set of sampling directions (e.g., sound source directions for which the input spatial audio format is obtained). Further, the output spatial audio format may be specified by a second set of directions and a decoding-to-speakers matrix for decoding channels of the output spatial audio format to output audio signals for loudspeakers at respective ones of the second set of directions. The second set of directions may be a set of loudspeaker directions. Loudspeaker locations may be on a unit sphere around the origin, with the directions in the second set of directions pointing towards respective loudspeaker locations. For such input and output spatial formats, the cost function may be based on the encoding matrix, the decoding-to-speakers matrix, and the (parameterization of the) conversion matrix.

[0010] In some embodiments the cost function may be based on a system matrix for linking the input audio signals to the output audio signals. This system matrix may be based on the conversion matrix. The system matrix (or transfer matrix) may be an input-output matrix of an encoding-transcoding-decoding processing chain. The system matrix may be based on the parameterization of the conversion matrix.

[0011] In some embodiments, the system matrix may be further based on the encoding matrix and the decoding-to-speakers matrix.

[0012] In some embodiments, the cost function may be further based the first set of directions and the second set of directions.

[0013] In some embodiments, at least one term of the cost function may be based on a psychoacoustic quantity. The psychoacoustic quantity may relate to a psychoacoustic quantity determined for an origin of the output layout, for example in terms of or based on the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0014] Thereby, compliance of the generated conversion matrix with specific psychoacoustic requirements towards the reproduced sound field can be ensured.

[0015] In some embodiments, at least one term of the cost function may be based on one or more of: sound pressure, acoustic intensity, sound energy, and acoustic velocity. The aforementioned psychoacoustic quantities may relate to psychoacoustic quantities determined (e.g., reconstructed) for an origin of the output layout, for example in terms of or based on the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0016] In some embodiments, the cost function may include a term penalizing non-sparsity of the system matrix. Thereby, optimization towards a transcoding matrix that minimizes the number of active speakers upon decoding can be enforced.

[0017] In some embodiments, the cost function may include a term indicative of a mean quadratic deviation from zero phase of the sound pressure.

[0018] In some embodiments, the cost function may include a term indicative of a mean quadratic deviation from zero phase of a radial component of the acoustic velocity.

[0019] In some embodiments, the cost function may include a term indicative of a mean quadratic deviation from zero phase of a transverse component of the acoustic velocity.

[0020] In some embodiments, said term may be based on the system matrix. That is, said term of the cost function may be a function of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0021] In some embodiments, the cost function may include a term that is quadratic in terms of coefficients of the system matrix. Said term of the cost function may be a function of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0022] In some embodiments, the cost function may include a term that is quartic in terms of coefficients of the system matrix. Said term of the cost function may be a function of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0023] In some embodiments, the cost function may include a term penalizing non-sparsity of the system matrix. Thereby, optimization towards a transcoding matrix that minimizes the number of active speakers in decoding can be enforced.

[0024] In some embodiments, the cost function may include a term penalizing large values of elements (or coefficients, entries, etc.) of the conversion matrix. Said term may apply a penalty that is positively correlated with a measure of magnitude depending on values of one or more(e.g., all) elements of the conversion matrix. Thereby, excessive gains in the conversion from the input format to the output format can be avoided.

[0025] In some embodiments, the cost function may include a term penalizing elements in the conversion matrix associated with channels whose noise level exceeds a predefined threshold.

[0026] In some embodiments, the encoding scheme may be complex-valued. For example, the encoding matrix may be a complex-valued matrix. Alternatively, the encoding scheme may be real-valued. For example, the encoding matrix may be a real-valued matrix.

[0027] In some embodiments, the output layout may relate to a physical loudspeaker layout. Alternatively, the output layout may relate to a virtual loudspeaker layout.

[0028] In some embodiments, the output spatial audio format may be one of Ambisonics, a multichannel format, and a linear decoding format.

[0029] In some embodiments, the input spatial audio format may be one of Ambisonics, a multichannel format, and a linear encoding format.

[0030] In some embodiments, encoding the first set of directions to the input spatial audio format may relate to (or may involve) recording a set of input audio signals via a microphone array.

[0031] In some embodiments, encoding the first set of directions to the input spatial audio format may involve a one-to-one mapping between the input audio signals and the channels of the input spatial audio format. That is, the encoding matrix may be a permutation matrix or the identity matrix. This scenario may allow for decoding optimization.

[0032] In some embodiments, decoding the output spatial audio format to the output layout may involve a one-to-one mapping between the channels of the output spatial audio format and the output audio signals. That is, the decoding-to-speakers matrix may be a permutation matrix or the identity matrix. This scenario may allow for encoding optimization.

[0033] In some embodiments, at least one of the encoding scheme and the decoding scheme may be frequency dependent. Then, the conversion matrix may be separately optimized in each of a plurality of frequency bands.

[0034] In some embodiments, the cost function may be frequency dependent.

[0035] In some embodiments, the conversion matrix may be optimized simultaneously across each of a plurality of frequency bands. In this case, the cost function may include a contributionfor each of the plurality of frequency bands and one or more terms penalizing discontinuities of the conversion matrix between neighboring frequency bands.

[0036] In some embodiments, the method may further include determining an effective cost function for a given frequency band. The conversion matrix may be optimized based on the effective cost function for the given frequency band. Further, determining the effective cost function for the given frequency band may be based on a plurality of cost functions at respective frequencies between a lower limit frequency of the given frequency band and an upper limit frequency of the given frequency band. The effective cost function for the given frequency band may be used for separate optimization of the conversion matrix in the given frequency band, or when simultaneously optimizing across frequency bands. The effective cost function may be an aggregated cost function (e.g., average, weighted average, sum, weighted sum, etc.) of cost functions at the plurality of frequencies between the lower limit frequency and the upper limit frequency. The cost functions at the plurality of frequencies may be determined for example based on a frequency-dependent encoding matrix.

[0037] According to another aspect, an apparatus is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform the methods or method steps outlined throughout the present disclosure.

[0038] According to a further aspect, a computer program is described. The computer program may comprise executable instructions for performing the methods or method steps outlined throughout the present disclosure when executed by a computing device (e.g., processor).

[0039] According to another aspect, a computer-readable storage medium is described. The storage medium may store a computer program adapted for execution on a computing device (e.g., processor) and for performing the methods or method steps outlined throughout the present disclosure when carried out on the computing device.

[0040] It should be noted that the methods and apparatus including its preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and apparatus disclosed in this document. Furthermore, all aspects of the methods and apparatus outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

[0041] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) (and, e.g., their steps) are understood to likewise apply to the corresponding apparatus (and, e.g., their blocks, stages, units), and vice versa. Brief Description of the Drawings

[0042] The invention is explained below in an exemplary manner with reference to the accompanying drawings, wherein

[0043] Fig. 1 schematically illustrates on overview of an example system for optimization of a conversion matrix according to embodiments of the disclosure;

[0044] Fig.2 schematically illustrates an example of a system matrix for the system of Fig.1 that may be subject to optimization, according to embodiments of the disclosure;

[0045] Fig.3 is a flowchart illustrating an example of a method of determining an optimized conversion matrix according to embodiments of the disclosure; and

[0046] Fig. 4 is a block diagram schematically illustrating an example of an apparatus suitable for implementing methods according to embodiments of the disclosure. Detailed Description

[0047] In the following, example embodiments of the disclosure will be described with reference to the appended figures. Identical elements in the figures may be indicated by identical reference numbers, and repeated description thereof may be omitted. Introduction

[0048] Broadly speaking, the present disclosure may be said to relate to a universal audio format transcoder method that, given any input spatial audio format (such as the recording of a circular array of microphones, third order Ambisonics, or 5.1 multichannel, for example) and an output spatial audio format (such as second order Ambisonics or 7.1 multichannel, for example), is able to generate an optimal transcoding matrix from the input to the output format, based on the optimization of a cost function. The cost function may be psychoacoustically-motivated. To this end, the proposed methods do not require knowledge of the detailed specifications of neither theinput nor the output formats; the only information required for the input format is the encoding of a plane wave on the said input format on a set of sampled directions, and for the output format, the additional decoding-to-speakers matrix from the output format to the loudspeaker layout.

[0049] The transcoding methods to which techniques according to the present disclosure are applicable include, as particular cases, the encoding from a recording setup to an output spatial audio format, and the decoding from an input spatial audio format to a loudspeaker layout. In the latter decoding case, the additional decoding-to-speakers matrix may simply be either an identity matrix or a routing matrix (e.g., permutation matrix), which maps each channel to each loudspeaker.

[0050] In another particular case, the proposed methods allow finding the optimal panning matrix from any input position to any loudspeaker layout. System Overview

[0051] Fig. 1 schematically illustrates on overview of an example system for optimization of a conversion matrix (e.g., transcoding matrix) 110 for converting between an input spatial audio format and an output spatial audio format, according to embodiments of the disclosure. While reference will be frequently made to a transcoding matrix throughout the disclosure, this shall be without prejudice and without intended limitation, and it is to be understood that the present disclosure likewise applies to any conversion matrices for converting between an input spatial audio format and an output spatial audio format.

[0052] The input spatial audio format (input format) is, without intended limitation, an ^^- channel format and is specified by an encoding scheme for encoding a first set of directions (e.g., ^^ sampling directions) to the ( ^^ channels of the) input spatial audio format. Specifically, the input spatial audio format may be specified by the first set of directions and an encoding matrix (input matrix) 120 for encoding input audio signals from respective ones of the first set of directions to channels of the spatial input format. For ^^ spatial directions and ^^ channels of the input format, the encoding matrix 120,will be an ^^ × ^^ matrix.

[0053] The output spatial audio format (output format) is, without intended limitation, an ^^- channel format and is specified by a decoding scheme for decoding the ( ^^ channels of the) output spatial audio format to an output layout. This output spatial audio format may be specifiedby a second set of directions (e.g., ^^ speaker directions) and a decoding-to-speakers matrix 130 for decoding channels of the output spatial audio format to output audio signals for loudspeakers at respective ones of the second set of directions. For ^^ second directions and ^^ channels of the output spatial audio format, the decoding-to-speakers matrix 130, , will be a ^^ × ^^matrix.

[0054] As noted above, the first set of directions may be a set of ^^ sampling directions (e.g., microphone directions) 125. The second set of directions may be a set of ^^ loudspeaker directions, where loudspeaker locations may be on a unit sphere around the origin, with the directions in the second set of directions pointing towards respective loudspeaker locations.

[0055] In practice, the input matrix 120 may be generated given the set of ^^ sampling directions 125 (first set of directions). Encoding the ^^ sampling directions to the input spatial audio format may relate to recording a set of input audio signals via a microphone array. For example, the input matrix 120 may be the spatially sampled polar pattern of a set of microphones (e.g., gain and phase, giving a complex valued matrix). As another example, the input matrix 120 may be the spatially sampled encoding gains of a linear encoding format such as Ambisonics, SWF or VBAP (giving a real-valued matrix). In yet other examples, the input matrix 120 may be an identity matrix (or permutation matrix) that assigns each sampling direction to a separate channel.

[0056] Depending on details and implementations thereof, the encoding scheme (e.g., encoding matrix 120) may be complex-valued or real-valued. Further, the encoding scheme may be frequency dependent.

[0057] Moreover, as described above, the input spatial audio format (input format) may be, without intended limitation, one of Ambisonics, a multichannel format, and a linear encoding format, for example.

[0058] In any case, it is understood that the input matrix 120 is a known input to methods according to embodiments of the disclosure. Likewise, also the sampling directions (first set of directions) 125 are understood to be known.

[0059] Also the output spatial audio format may be, without intended limitation, one of Ambisonics, a multichannel format, and a linear decoding format, for example.

[0060] Further, the aforementioned output layout may relate to a physical loudspeaker layout of ^^ physical loudspeakers at respective locations / directions, or alternatively, depending on the desired optimization, a virtual loudspeaker layout.

[0061] It is understood that also the decoding-to-speakers matrix 130 is a known input to methods according to embodiments of the disclosure. Likewise, also the loudspeaker directions (second set of directions) are understood to be known.

[0062] With the encoding scheme and the decoding scheme being a known input to the procedure, only the conversion matrix (e.g., transcoding matrix) 110 is subject to optimization. For ^^ channels of the input spatial audio format and ^^ channels of the output spatial audio format, the transcoding matrix 110, ^ ^^^^^, will be an ^^ × ^^ matrix.

[0063] The optimization process for optimizing the transcoding matrix 110 optimizes (e.g., minimizes) a cost function (described in more detail below), by adjusting the values of the transcoding matrix 110, which in general may be complex-valued.

[0064] In the example of Fig. 1, the cost function for optimization depends on (e.g., is a function of) the product (matrix product) of the transcoding matrix 110 with the decoding-to-speakers matrix 130, which may be typically real-valued. Said product may relate to a decoding matrix 140 for decoding the ^^ channels of the input spatial audio format to ^^ loudspeaker signals.

[0065] The product of the transcoding matrix 110 with the decoding-to-speakers matrix 130 may be referred to the decoding matrix 140, as noted above. The product of the input matrix 120 with the decoding matrix 140, as shown in the example of Fig. 2, may be referred to as system matrix 150. The system matrix 150 describes the whole encoding, transcoding, and decoding process. In other words, the system matrix (or transfer matrix) 150 may be said to relate to an input-output matrix of an encoding-transcoding-decoding processing chain, and thus may be said to link the ^^ input audio signals to the ^^ output audio signals (loudspeaker signals). For ^^ input audio signals and ^^ output audio signals, the system matrix 150, ^, will be an ^^ × ^^ matrix.

[0066] The aforementioned cost function for optimization may be based on the system matrix 150 that links the ^^ input audio signals to the ^^ output audio signals. The system matrix 150 in turn depends on the conversion matrix 110 (e.g., a parameterization thereof, such as in terms of its elements / coefficients / entries), and further on the encoding matrix 120 and the decoding-to-speakers matrix 130. In general, the cost function may thus be said to be based onparameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and the conversion matrix (e.g., a parameterization thereof, such as in terms of its elements / coefficients / entries). For example, the cost function may be based on the encoding matrix, the decoding matrix, and the conversion matrix. Further, as described in more detail below, the cost function may be further based the first set of directions and the second set of directions.

[0067] The (functional dependence of) the cost function on the conversion matrix 110 (e.g., via dependence on the system matrix 150 or its elements / coefficients / entries, etc.) may be determined by the type of the required transcoding and may be fixed at initialization time. For example, this may apply to (optional) directional weights (e.g., directional weights ^^^defined below) and / or multiplicative coefficients (e.g., ^^ weights defined below) of the cost function.

[0068] Notably, Fig. 1 and Fig.2 do not specifically show where and which user input may be needed and where it may be directed. This is in any case not relevant to techniques according to the present disclosure. In general, the user can configure all entities indicated by solid boxes in these drawings. The only part of the system that is not directly configurable by the user is the transcoding matrix 110 which is generated by the optimization procedure. Still, optimization may be dependent on user input or preferences, via the cost function. Notation and Definitions

[0069] Without intended limitation, it will be assumed throughout the present disclosure that M is the number of channels of the input format, N is the number of channels of the output format, L is a set of directions (first set of directions) sampling the (unit) sphere,^{ ^^^} is a set of P output speaker positions (second set of directions) and { ^^^^} is the ^^ × ^^ input matrix (encoding matrix) that encodes a point source of unit amplitude coming from each one of the L sampled directions into the M channels of the input format. The ^^ × ^^ decoding matrix { ^^^^} generates the set of ^^ × ^^ output signals { ^^^^}, viz. the signal fed to the speaker p while reproducing a source of unit amplitude coming from direction l,or, more simply written in matrix notation,^^ = ^^ ^^்(2) In a more general case, the full decoding matrix { ^^^^} is composed of a conversion matrix (e.g., transcoding matrix) { ^^^^} and an additional decoding-to-speakers matrix {or, more simply written in matrix notation, ^^ = ^^^^^^^ (4)

[0070] The ^^ × ^^ transcoding matrix ^^ converts the input format channels (M channels) into output format channels (N channels). The additional ^^ × ^^ decoding-to-speaker matrix ^^spkis assumed to be known, so the goal is to optimize the transcoding matrix ^^ only. The decoding-to- speakers matrix ^^spkmaps from the N output format channels to the P speakers’ channels.

[0071] In some embodiments, the output layout of loudspeakers may correspond to a real (i.e., physical) listening layout. In some other cases, the output layout may not correspond to any existing listening layout, and thus may relate to a virtual listening layout. Techniques according to the present disclosure may be agnostic of this distinction and may be equally applicable to either case.

[0072] In one embodiment (e.g., for decoding optimization), the output format (output spatial audio format) may be the multichannel setup corresponding to the listening venue. The input matrix (encoding matrix) may be real-valued, and the number ^^ of output loudspeakers may coincide with the number ^^ of output channels, ^^ = ^^. Further in this case, decoding the output format to the output layout may involve a one-to-one mapping between the ^^ channels of the output format and the ^^ = ^^ output audio signals. For example, the additional decoding-to- speakers matrix ^^spkmay be the identity matrix, ^^spk= 1^ ^ ^, and the transcoding matrix ^^ may correspond to the decoding matrix, ^^ = ^^. In this case, as noted above, the output layout of loudspeakers corresponds to the real listening layout. One example of this can be Ambisonics decoding, where the input matrix is the spatially sampled Ambisonics encoding, with dimensionscorresponding to the number of sampling directions times the number of Ambisonics channels. The decoding matrix maps from Ambisonics channels to speakers (e.g., Scaini & Arteaga, 2014). In a variation of this embodiment, the decoding-to-speakers matrix, instead of an identity matrix, can also a permutation matrix that routes each output channel to a respective loudspeaker.

[0073] Since the transcoding matrix ^^ corresponds to the decoding matrix ^^ in this embodiment, optimization according to the present disclosure will optimize the decoding matrix ^^.

[0074] In another embodiment (e.g., microphone encoding optimization), the input format (input spatial audio format) may be the direct recording of an array of microphones, evaluated for each frequency band, and therefore the input matrix (encoding matrix) per frequency band may be complex-valued. The output format (output spatial audio format) can be either a layout- independent format such as Ambisonics, or a multichannel format such as 7.1. In this case, normally the reproduction layout may be a convenient virtual layout that explores all relevant spatial locations but that does not necessarily correspond to any real (i.e., physical) layout. An example of this can be the optimization of encoding of the recording of a tetrahedral Soundfield microphone into a first order Ambisonics signal. The virtual loudspeaker layout could be an octagonal mesh of loudspeakers, for example. Other examples are the encoding of the recording of the microphones in a mobile phone into a 5.1 layout or first order Ambisonics in 2D. In each of these latter examples, the reproduction layout could be a virtual hexagonal setup in 2D.

[0075] In this situation, optimization according to the present disclosure will optimize the conversion matrix for a specific encoding matrix { ^^^^}.

[0076] In yet another embodiment (e.g., virtual source encoding optimization), the input format may consist of a set of L channels, one per each point of the sampling directions. Encoding the set of ^^ channels to the input format may involve a one-to-one mapping between the ^^ input audio signals (channels) and the ^^ = ^^ channels of the input format. For example, the input matrix { ^^^^} may be simply the identity matrix (and hence, ^^ = ^^), or a permutation matrix. The reproduction layout can be a real reproduction layout or a convenient virtual layout that explores all relevant spatial locations. This embodiment allows optimizing the encoding of an input source in each one of the sampling directions. In one example, this may be used to find the optimal encoding of a set of sampled point sources in Ambisonics. In another example, this method can be used to find the optimal panning function for a set of point sources in a multichannel 5.1 format.

[0077] In yet another embodiment (e.g., transcoding optimization), the input format may be complex or real, multichannel, or layout-independent, and the output format may similarly be complex or real, layout-dependent or independent. The loudspeaker layout under consideration may be real (i.e., physical) or virtual. One example is finding the optimal transcoding matrix from 5.1 to Ambisonics, with the loudspeaker layout corresponding to a virtual loudspeaker layout of 8 loudspeakers placed in the vertices of an octagon.

[0078] Regardless of specifics of the encoding scheme and decoding scheme, the present disclosure aims at obtaining an optimized transcoding matrix T by optimizing (e.g., minimizing) a cost function. Preferably, this cost function is a psychoacoustically motivated cost function.

[0079] Fig.3 is a flowchart illustrating an example of a method (e.g., computer-implemented method) 300 of determining an optimized conversion matrix (e.g., transcoding matrix) according to embodiments of the disclosure. Method 300 includes steps S310 through S330, wherein steps S310 and S320 may be performed in arbitrary order.

[0080] At step S310, parameters descriptive of the encoding scheme are obtained. This may involve obtaining (e.g., receiving as input), the first set of directions and obtaining (e.g., receiving as input, or determining) the encoding matrix. For example, if not received as input, the encoding matrix may be determined based on the first set of directions and the input spatial audio format.

[0081] At step S320, parameters of the decoding scheme are obtained. This may involve obtaining (e.g., receiving as input), the second set of directions and obtaining (e.g., receiving as input, or determining) the decoding-to-speakers matrix. For example, if not received as input, the decoding-to-speakers matrix may be determined based on the second set of directions and the output spatial audio format.

[0082] At step S330, the conversion matrix is optimized based on a cost function. This cost function, as described above, is based on the parameters descriptive of the encoding scheme, the parameters descriptive of the decoding scheme, and the conversion matrix.

[0083] For a frequency dependent setup. optimization may be performed separately in each of a plurality of frequency bands, or simultaneously across frequency bands, with respective contributions to the cost function for each frequency band.

[0084] Details and possible terms (cost function terms) of the cost function will be described next.Cost Function

[0085] The cost function according to the present disclosure may be a psychoacoustically motivated cost function, or in other words, at least one term (cost function term) of the cost function may be based on a psychoacoustic quantity. Examples of such cost function terms will be described in more detail below.

[0086] The cost function may for example contain, among others, any, some, or all of the following terms:

[0087] where the ^^ are the weights of each sub-cost (cost function term) of ^^. The ^^ weights may be user selected. For example, the weights may be chosen in accordance with optimization preferences. Assuming a psychoacoustically motivated cost function, the ^^ weights may be chosen to emphasize certain psychoacoustic quantities, correct reproduction of which is of particular interest, while potentially suppressing (or altogether nulling) others. Accordingly, the cost function may contain any, some, or all of the cost function terms listed in Eq. (5).

[0088] In the cost function above, two types of terms or sub-cost functions may be included, among others: quadratic terms and quartic terms. The former assumes a coherent behavior of the sound field and may be used for encoding processes (encoding optimization), due to the fact that they are directly linked to the precise reconstruction of the sound field. The latter assume an incoherent behavior of the sound field, and may be preferable for decoding optimization, since often incoherent behavior is found in reproduction venues. The choice of which terms to prefer during the optimization stage, for example via appropriate choice of ^^ weights, can also be related to the frequency range where the transcoding is of interest. As an example, at low frequencies reconstruction of pressure and velocity may be preferred, while at high frequencies energy and intensity reconstruction may be preferred. This preference is motivated by psychoacoustic considerations (e.g., (Gerzon, 1992; Frank, 2013).

[0089] As noted above, at least one term of the cost function may be based on a psychoacoustic quantity. This psychoacoustic quantity may relate to a psychoacoustic quantity determined (e.g.,reconstructed) for an origin of the output layout, for example in terms of or based on the aforementioned system matrix, assuming sources (e.g., input audio signals) of unit amplitude. For example, at least one term of the cost function may be based on sound pressure. Additionally or alternatively, at least one term of the cost function may be based on acoustic intensity. Additionally or alternatively, at least one term of the cost function may be based on sound energy. Additionally or alternatively, at least one term of the cost function may be based on acoustic velocity. Examples of specific cost function terms based on these psychoacoustic quantities will be described in more detail below. Quadratic Terms of the Cost Function

[0090] As noted above, the cost function may include one or more terms that are quadratic in terms of coefficients of the system matrix. These terms of the cost function may again be functions of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0091] In the following, additional quantities for describing the aforementioned cost function terms will be introduced. The pressure ^^^for input signal ^^ reconstructed at the origin is given byandwith real and imaginary parts ^^^ோand ^^^ூ, respectively, ^^^= ^^^ோ+ ^^ ^^^ூ(8) Further, real and imaginary parts of the acoustic velocity ^^^for input signal ^^ reconstructed at the origin are given by^andwhere^ ^^^is the unit vector pointing towards the direction of speaker p. The real part of the velocity vector, ^^^ோ, can be projected to the radial and transverse part as follows,where ^^^^is the unit vector pointing to the sampling location / direction ^^. The imaginary part of the velocity vector, ^^^ூ, can be similarly projected to the radial and transverse part as follows,

[0092] Therein, real radial part ^^^ோ^ோrepresents the desired component of the real intensity vector, whereas all the imaginary parts as well as ^^^்^ோrepresent unwanted components. In an ideal system, ^^ோ= 1 , ^ூ ோ^ோ ோ^ூ ^ோ ^ூ^ ^^= 0, ^^^= 1, ^^^= 0, ^^^்= 0, and ^^^்= 0.

[0093] The following six different cost function terms can be defined based on the above quantities,wherewherewherewherewhereandwhere

[0094] In the above, the ^^^are (optional) directional weights, that may give preference to some directions over some other directions.

[0095] These above contributions can be interpreted as follows:is the mean quadratic deviation from the correct pressure level; ^^^ುೃೀ^is the mean quadratic deviation from zero phase; is the mean quadratic deviation from the optimal directivity, where ^^|ோ^|> 0 means that the apparent size of the sources is not optimal; ^^ோ^ುೃೀ^is the mean quadratic deviation from the zero phase; and, finally,and ^^்^ುೃೀ^are respectively the mean quadratic value of the wrong component of the direction and its deviation from zero phase, so they represent a localization error.

[0096] The cost function according to the present disclosure may include cost function terms relating to any, some, or all of the above.

[0097] Specifically, the cost function may include a term indicative of a mean quadratic deviation from zero phase of the sound pressure (e.g., a term based on or corresponding to Additionally or alternatively, the cost function may include a term indicative of a mean quadratic deviation from zero phase of a radial component of the acoustic velocity (e.g., a term based on or corresponding to ^^ோ^ುೃೀ^). Additionally or alternatively, the cost function may include a term indicative of a mean quadratic deviation from zero phase of a transverse component of the acoustic velocity (e.g., a term based on or corresponding to ^^்^ುೃೀ^). Any of these terms may be based on the system matrix. That is, respective terms of the cost function may be functions of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude. Quartic Terms of the Cost Function

[0098] As noted above, the cost function may also include one or more terms that are quartic in terms of coefficients of the system matrix. These terms of the cost function may again be functions of coefficients of the system matrix, assuming sources (e.g., input audio signals) of unit amplitude.

[0099] Additional quantities for describing the aforementioned cost function terms will be described next. The energy for input signal ^^ reconstructed (e.g., under the incoherence assumption hypothesis) at the origin is given byFurther, the acoustic intensity for input signal ^^ reconstructed (e.g., under the incoherence assumption hypothesis) at the origin is given by^The vector ^^^can be projected to the radial and transverse parts as follows,The radial part ^^^ோrepresents the desired component of the intensity vector and may be related to the apparent source size. The tangential (i.e., transverse) part ^^^்represents the unwanted component and may be related to the localization error. In an ideal system, ^^^= 1, ^^^ோ= 1, and ^^^்= 0. The following three different cost function terms can be defined based on the above quantities,(32) and

[0100] In the above, the ^^^are (optional) directional weights, that may give preference to some directions over some other directions.

[0101] These contributions can be interpreted as follows: CEis the mean quadratic deviation from the correct energy reconstruction; CRI is the mean quadratic deviation from the optimal directivity, where CRI > 0 means that the apparent size of the sources is not optimal; and finally, CTI is the mean quadratic value of the wrong component of the direction, so it represents a localization error.

[0102] The cost function according to the present disclosure may include cost function terms relating to any, some, or all of the above. Other Cost Function Terms

[0103] In-phase decoding may be possible with an extra cost function term, to penalize the out-of-phase contributions from different loudspeakers. Two versions are proposed, based on a quadratic termand a quartic term ^^ℎ^The following two different cost function terms can be defined based on the above quantities,^

[0104] The above terms of the cost function ^^^ு^௨^ௗand ^^^ு^௨^^௧, being proportional to the square or forth power of the system matrix components, respectively, may favor smaller transcoding matrices (e.g., transcoding matrices with smaller entries) during the minimization process. To avoid this unwanted side effect, in some other examples, the cost function terms based on the quadratic term ^^ℎ^௨^ௗand the qu^௨^^௧^ artic term ^^ℎ^may thus be alternatively defined as:

[0105] The cost function according to the present disclosure may include cost function terms relating to any or both of the cost function terms in Eq. (38) and Eq. (39), or cost function terms relating to any or both of the cost function terms in Eq. (40) and Eq. (41).

[0106] Additionally, a symmetry cost can be added to the cost function to enforce that left-right symmetric directions generate similar reconstructed psychoacoustic variables. Once left-right symmetric pairs ( ^^,− ^^) have been identified in the destination layout, it is possible to quantify the amount of asymmetry present in the decoding (again, both in quadratic and linear versions),^In some examples, the following two different cost function terms can be defined based on the above quantities:To avoid favoring smaller transcoding matrices during the minimization process, cost function terms based on the aforementioned amounts of asymmetrycan be alternatively defined as:

[0107] Therein, the weighting function wl is an optional biasing factor that allows to improve the decoding performance in some regions of the space (at the expense of other regions). A non-biased decoding is given by ^^^= 1.

[0108] The cost function according to the present disclosure may include cost function terms relating to any or both of the cost function terms in Eq. (44) and Eq. (45), or cost function terms relating to any or both of the cost function terms in Eq. (46) and Eq. (47).

[0109] Additionally, quadratic and / or quartic sparsity enhancing terms can be added to the cost function. These terms penalize non-sparse solutions of the transcoding matrix (or system matrix).

[0110] In mathematical or computing context, a sparse vector or matrix is a vector or matrix in which most of the elements are zero or close to zero. In spatial audio, oftentimes reproduction methods may be preferred that for a given virtual sound source activate a low number of loudspeakers (e.g., typically, from 1 to 4, but the smaller the number, the better, depending on implementations). In the present context, this preference can be translated into demanding that each row of the system matrix, which is the matrix that connects the input sources with the loudspeakers, {is as sparse as possible.

[0111] Without intended limitation, one possible way to quantify the non-sparsity of the solution is based on the L1 and L2 norms of the rows of the system matrix. For example, the non-sparsity may be quantified based on the difference of the L1 and L2 norms of the rows of the system matrix:

[0112] Alternatively, non-sparsity measures may be defined based on a ratio of the L1 and L2 norms of the rows of the system matrix, noting that substantially equal L1 and L2 norms indicate a high degree of sparsity. Further examples of non-sparsity measures that may be used in the context of the present disclosure are summarized in Niall Hurley and Scott Rickard, “Comparing Measures of Sparsity,” arXiv:0811.4706v2 [cs.IT], 27 April 2009, which is herebyincorporated by reference in its entirety.

[0113] In some examples, the following two different cost function terms can be defined based on the above quantities:

[0114] The proposed cost function terms measure the non-sparsity of the rows of the system matrix by comparing the L2 and L1 norms. When one or more of these terms are present in the cost function, the optimizer will try to minimize these non-sparsity cost function terms, thereby leading to a more sparse decoding, with less loudspeakers active.

[0115] To avoid favoring smaller transcoding matrix during the minimization process, the cost function terms based on the aforementioned measures on non-sparsity (or equivalent measures of non-sparsity) can be alternatively defined as:

[0116] The cost function according to the present disclosure may include cost function terms relating to any or both of the cost function terms in Eq. (50) and Eq. (51), or cost function terms relating to any or both of the cost function terms in Eq. (52) and Eq. (53). In general, the cost function according to the present disclosure may include one or more cost function terms (e.g., quadratic or quartic) that depend on a measure of sparsity (or non-sparsity) of the system matrix, or put differently, that penalize non-sparsity of the system matrix.

[0117] Other cost function terms can be added as well, depending on implementations and circumstances. For example, cost function terms may be added that penalize large values of the transcoding matrix elements, to avoid excessive gains. One of such possible terms can beTherein, the value of exponent q can be 1 or 2, for example. In yet another example, the term could be only active whenever a certain value t is surpassed,

[0118] where ^^(·) is the Heaviside step function, allowing in this latter case the value ^^ = 0 for the exponent.

[0119] In any case, in some embodiments the cost function may include a term penalizing large values of elements of the conversion matrix. In general, said term may apply a penalty that is positively correlated with a measure of magnitude depending on values of one or more (e.g., all) elements of the conversion matrix. Thereby, excessive gains in the conversion from the input format to the output format can be avoided. Examples of such term are given in Eq. (54) and Eq. (55) above.

[0120] Another additional term can be added to the cost function in the case of microphone encoding, to limit the gain in the transcoding matrix with respect to a measure of the self-noise of the microphones. Given a measure of the self-noise of each of the input microphones (e.g., broad band or per frequency or frequency band), one example of such cost function term could be the sum of the differences between the transcoding matrix gain times the self-noise value and a certain fixed threshold,

[0121] where ^^^is the self-noise for each microphone ^^ (in dB), ^^ is the threshold (in dB) and ^^(·) is the Heaviside step function. That said, the cost function may in general include a term penalizing elements in the conversion matrix associated with channels whose noise level exceeds a predefined threshold. Frequency Dependent Cost Function Terms

[0122] The procedure described so far can be generalized to the case where any of the system components is frequency dependent. For example, at least one of the encoding scheme and the decoding scheme may be frequency dependent. In other words, the input matrix or the decoding matrix, or both, can be frequency dependent. This will in general lead to a frequency dependent cost function (e.g., via the frequency dependent system matrix).

[0123] In one embodiment the input (encoding) and decoding matrices are obtained in the desired frequency bands ^^^, ^^ = 1, .. , ^^. The cost function (and thereby, the transcoding matrix) is then optimized separately (independently) in each frequency band,

[0124] resulting in ^^ independent (optimized) transcoding matrices ^^( ^^^). This includes the possibility that the weights of the cost terms ^^ and the directional weights ^^^are frequency dependent.

[0125] An independent optimization of the transcoding matrix in each frequency band might result in an arbitrarily large variation of the (transcoding) gains across frequency, which could potentially have a negative impact on the listening experience.

[0126] To mitigate this aspect, in one embodiment the optimization is performed simultaneously across (a plurality of) frequency bands, with the total cost function ^^ including a contribution ^^( ^^^) for each of the plurality of frequency bands and one or more terms penalizing discontinuities of the conversion matrix between neighboring frequency bands. For example, thecost function ^^ may be the sum of the costs ^^( ^^^) at each frequency plus terms that penalizes discontinuities across neighboring bands by adding the absolute value of the difference between T( ^^^) at neighboring bands, for example viawhere‖∙‖denotes the Frobenius norm.

[0127] In another embodiment, an averaging method across frequency may be utilized. Each frequency band may be optimized in turn with a cost function relating to the emergent response at frequencies around the design points (e.g., the center frequencies in a frequency- dependent approach) given a specific averaging strategy between the design points. For example, assuming simple averaging,

[0128] where N is the number of frequency points between ^^^^^and ^^^^௫, which are respectively the lower and upper frequency limits of the band ^^. Optionally, the sum in Eq. (59) may include frequency dependent pre-factors to implement, for example, a window function across frequency.

[0129] In line with the above, methods according to embodiments of the disclosure may include determining an effective cost function ^^^for a given frequency band ^^, and optimizing the conversion matrix (e.g., transcoding matrix) based on the effective cost function ^^^for the given frequency band ^^. Therein, determining the effective cost function ^^^for the given frequency band ^^ may be based on a plurality of cost functions ^^( ^^) at respective frequencies ^^ between a lower limit frequency ^^^^^of the given frequency band and an upper limit frequency ^^^^௫of the given frequency band. For example, the frequencies ^^ between ^^^^^and ^^^^௫may be a given number ^^ of equally distributed frequencies between ^^^^^and ^^^^௫, optionally including ^^^^^and ^^^^௫.

[0130] In the optimization, the effective cost function for the given frequency band may be used for separate optimization of the conversion matrix in the given frequency band, for example as per Eq. (57) or when simultaneously optimizing across frequency bands, for example as per Eq. (58). The effective cost function may be an aggregated cost function (e.g., average, weighted average, sum, weighted sum, etc.) of cost functions at the plurality of frequencies between the lower limit frequency and the upper limit frequency. The cost functions at the plurality of frequencies may be determined for example based on a frequency-dependent system matrix (e.g., based on the frequency dependent input matrix), as described above. Obtaining the Input Matrix Spatial Audio Encoding Method

[0131] Any spatial audio encoding method that can encode a (virtual) source of unit amplitude coming from a certain direction of space into a spatial audio format can be used to generate the appropriate input matrix needed for purposes of the present disclosure. Without intended limitation, this includes the most common panning methods such as stereo, Vector Base Amplitude Panning (VBAP), Distance Based Amplitude Panning (DBAP), or the triple-balance Atmos® panner, and most common sound field encoding methods such as Ambisonics or SWF, for example.

[0132] As a first example, the VBAP method encodes each virtual source direction as a positive coefficient into a maximum of 3 channels. As a second example, each virtual source can be encoded in Ambisonics as a real coefficient, different for each Ambisonics channel. The number of channels depends on the order of Ambisonics. Microphone System

[0133] In the most general case, the signal from direction ^^ will result in a different amplitude and phase in each of the ^^ input channels, at each frequency. Such is the case with non-coincident microphones (e.g., a microphone array), where at every frequency and direction the signal is picked up with a different phase and amplitude by each microphone.

[0134] To capture such behavior, the input matrix { ^^^^} will be frequency dependent (i.e., a different matrix is obtained for each relevant frequency band) and complex-valued (or, alternatively, each entry has to contain amplitude and phase terms).

[0135] The input matrix for such a system can be obtained for example empirically by measuring or simulating the microphone signals resulting from a source in the direction ^^ at frequency ^^, for example by measuring or simulating the impulse responseand then obtaining its corresponding complex-valued frequency-domain representation ^^ ^^^,^( ^^) by a time-frequency transform such as the Discrete Fourier Transform. In some embodiments, the full-resolution frequency-domain ^^ ^^^,^( ^^) may be converted into a smaller set of frequency- band values ^^ ^^^,^( ^^^), ^^ = 1, … , ^^, each band containing a complex value which is the weighted sum of ^^ ^^^,^( ^^) in the frequencies belonging to the respective band.

[0136] It is also to be noted that the different input matrix terms { ^^^^} could be weighted, by appropriate multiplication by a different weight ^^^^in the cost function, for example to reflect the fact that some of the microphones may have a better signal-to-noise ratio than some other microphones. Optimization of the Cost Function

[0137] The goal of the optimization according to embodiments of the disclosure is to obtain the transcoding matrix coefficients ^^^^that minimize the value of the cost function C. The optimization problem may correspond to a generic non-linear programming problem.

[0138] There are several existing methods in the literature, and corresponding software packages implementing them, that can be used to find the optimal values of the transcoding matrix coefficients. Some examples of these methods are the gradient descent method, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method, the Newton method, and the Naider-Mead simplex method. Some of these methods, such the gradient descent method or the BFGS method, may require knowledge of the gradient of the cost function (first order derivatives); some others, such as the Newton method, may require additional knowledge of the Hessian of the cost function (second order derivatives). In contrast, some other methods, such as the Naider-Mead simplex method, do not require knowledge about the derivatives at all.

[0139] Should the first or second order derivatives be needed, they can be either computed manually from the analytic expressions of the cost function terms, or, more conveniently, they can be computed automatically from the software expression of the cost function using an automatic differentiation software framework, such as Jax, Pytorch orTensorflow, for example. As a third alternative, the derivatives could also be estimated numerically from finite differences using a numerical differentiation approach. Apparatus, Programs, and Storage Media

[0140] While methods of optimizing conversion matrices (e.g., transcoding matrices) have been described above, it is understood that the present disclosure likewise relates to apparatus (e.g., computer apparatus or apparatus having processing capability in general) for implementing these methods (or techniques in general).

[0141] An example of such apparatus 400 is schematically illustrated in Fig.4. The apparatus 400 comprises a processor 410 and a memory 420 coupled to the processor 410. The memory 420 may store instructions for execution by the processor 410. The processor 410 may be adapted to perform methods (e.g., methods of determining optimized conversion matrices) described throughout the disclosure. The apparatus 400 may receive inputs (e.g., indications of input and / or output schemes, indications of cost function weights, directional weights, etc.) and generate outputs (e.g., optimized conversion matrices, etc.) as described throughout the disclosure.

[0142] In addition, the present disclosure further relates to programs comprising instructions that, when executed by a processor, cause the processor to carry out methods described throughout the disclosure, and to computer-readable storage media storing such programs. Interpretation

[0143] Aspects of the systems described herein may be implemented in an appropriate computer-based processing network environment (e.g., server or cloud environment). Portions of these systems may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0144] One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or asdata and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[0145] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored on non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or application specific integrated circuits (“ASICs”). As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components, may be utilized to implement the embodiments. For example, apparatus described herein can include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0146] While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0147] Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted,” “connected,” “supported,” and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, andcouplings. Enumerated Example Embodiments Various Aspects and implementations of the invention may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims. EEE 1. A method of determining an optimized conversion matrix for converting between an input spatial audio format and an output spatial audio format, wherein the input spatial audio format is specified by an encoding scheme for encoding a first set of directions to the input spatial audio format and the output spatial audio format is specified by a decoding scheme for decoding the output spatial audio format to an output layout, the method comprising: optimizing the conversion matrix based on a cost function, wherein the cost function is based on parameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and the conversion matrix. EEE 2. The method according to EEE 1, wherein the input spatial audio format is specified by the first set of directions and an encoding matrix for encoding input audio signals from respective ones of the first set of directions to channels of the spatial input format; wherein the output spatial audio format is specified by a second set of directions and a decoding-to-speakers matrix for decoding channels of the output spatial audio format to output audio signals for loudspeakers at respective ones of the second set of directions; and wherein the cost function is based on the encoding matrix, the decoding-to-speakers matrix, and the conversion matrix. EEE 3. The method according to EEE 1 or 2, wherein the cost function is based on a system matrix for linking the input audio signals to the output audio signals; and wherein the system matrix is based on the conversion matrix. EEE 4. The method according to EEE 3 when depending on EEE 2, wherein the system matrix is further based on the encoding matrix and the decoding-to-speakers matrix. EEE 5. The method according to EEE 2 or any EEE dependent on EEE 2, wherein the cost function is further based the first set of directions and the second set of directions. EEE 6. The method according to any one of the preceding EEEs, wherein at least one term of the cost function is based on a psychoacoustic quantity.EEE 7. The method according to any one of the preceding EEEs, wherein at least one term of the cost function is based on one or more of: sound pressure; acoustic intensity; sound energy; and acoustic velocity. EEE 8. The method according to EEE 3 or any EEE dependent on EEE 3, wherein the cost function comprises a term based on a measure of sparsity or non-sparsity of the system matrix. EEE 9. The method according to any one of the preceding EEEs, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of the sound pressure. EEE 10. The method according to any one of the preceding EEEs, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of a radial component of the acoustic velocity. EEE 11. The method according to any one of the preceding EEEs, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of a transverse component of the acoustic velocity. EEE 12. The method according to any one of EEEs 9 to 11 when depending on EEE 3, wherein said term is based on the system matrix. EEE 13. The method according to EEE 3 or any EEE dependent on EEE 3, wherein the cost function comprises a term that is quadratic in terms of coefficients of the system matrix. EEE 14. The method according to EEE 3 or any EEE dependent on EEE 3, wherein the cost function comprises a term that is quartic in terms of coefficients of the system matrix. EEE 15. The method according to EEE 3 or any EEE dependent on EEE 3, wherein the cost function comprises a term penalizing non-sparsity of the system matrix. EEE 16. The method according to any one of the preceding EEEs, wherein the cost function comprises a term penalizing large values of elements of the conversion matrix.EEE 17. The method according to any one of EEEs 1 to 16, wherein the cost function comprises a term penalizing element in the conversion matrix associated with channels whose noise level exceeds a predefined threshold. EEE 18. The method according to any one of EEEs 1 to 17, wherein the encoding scheme is complex-valued. EEE 19. The method according to any one of EEEs 1 to 17, wherein the encoding scheme is real-valued. EEE 20. The method according to any one of EEEs 1 to 19, wherein the output layout relates to a physical loudspeaker layout. EEE 21. The method according to any one of EEEs 1 to 19, wherein the output layout relates to a virtual loudspeaker layout. EEE 22. The method according to any one of the preceding EEEs, wherein the output spatial audio format is one of Ambisonics, a multichannel format, and a linear decoding format. EEE 23. The method according to any one of the preceding EEEs, wherein the input spatial audio format is one of Ambisonics, a multichannel format, and a linear encoding format. EEE 24. The method according to any one of EEEs 1 to 23, wherein encoding the first set of directions to the input spatial audio format relates to recording a set of input audio signals via a microphone array. EEE 25. The method according to any one of EEEs 1 to 23, wherein encoding the first set of directions to the input spatial audio format involves a one-to-one mapping between the input audio signals and the channels of the input spatial audio format. EEE 26. The method according to any one of EEEs 1 to 25, wherein decoding the output spatial audio format to the output layout involves a one-to-one mapping between the channels of the output spatial audio format and the output audio signals. EEE 27. The method according to any one of the preceding EEEs, wherein at least one of the encoding scheme and the decoding scheme is frequency dependent; and wherein the conversion matrix is separately optimized in each of a plurality of frequency bands.EEE 28. The method according to EEE 27, wherein the cost function is frequency dependent. EEE 29. The method according to EEE 27 or 28, wherein the conversion matrix is optimized simultaneously across each of a plurality of frequency bands; and wherein the cost function comprises a contribution for each of the plurality of frequency bands and one or more terms penalizing discontinuities of the conversion matrix between neighboring frequency bands. EEE 30. The method according to any one of EEEs 27 to 29, further comprising: determining an effective cost function for a given frequency band, wherein the conversion matrix is optimized based on the effective cost function for the given frequency band; and wherein determining the effective cost function for the given frequency band is based on a plurality of cost functions at respective frequencies between a lower limit frequency of the given frequency band and an upper limit frequency of the given frequency band. EEE 31. An apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of EEEs 1 to 30. EEE 32. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of EEEs 1 to 30. EEE 33. A computer-readable storage medium storing the program of EEE 32.Literature Blanco Galindo, M., Coleman, P., & Jackson, P. J. (2020). Microphone Array Geometries for Horizontal Spatial Audio Object Capture With Beamforming. J. Audio Eng. Soc., Vol. 68, Issue 5, pp.324-337, May 2020. Frank, M. (2013). Phantom Sources using Multiple Loudspeakers in the Horizontal Plane. Gerzon, M. A. (1992). General Metatheory of Auditory Localisation. Audio Engineering Society Convention 92. Leal, O., & Elmar, H. (1991). Visual Mixing and Stereo Panning Techniques. Audio Engineering Society Convention 91. Scaini, D., & Arteaga, D. (2014). Decoding of Higher Order Ambisonics to Irregular Periphonic Loudspeaker Arrays. Audio Engineering Society Conference: 55th International Conference: Spatial Audio. Seefeldt, A. J., Lando, J. B., & Arteaga, D. (2021). Patent Application Publication No. WO2021021682A1. Zotter, F., & Frank, M. (2012). All-round ambisonic panning and decoding. J. Audio Eng. Soc., Vol.6, Issue 10, pp.807-820, October 2012. Zotter, F., & Frank, M. (2018). Ambisonic Decoding with Panning-Invariant Loudness on Small Layouts (AllRAD2). Audio Engineering Society Convention 144. Zotter, F., & Frank, M. (2019). Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality. Springer. Hurley, N. & Rickard, S. (2009). Comparing Measures of Sparsity. arXiv:0811.4706v2 [cs.IT].

Claims

Claims 1. A method of transcoding input audio signals having an input spatial audio format to output audio signals having an output spatial audio format via a conversion matrix for converting between the input spatial audio format and the output spatial audio format, wherein the input spatial audio format is specified by an encoding scheme for encoding a first set of directions to the input spatial audio format and the output spatial audio format is specified by a decoding scheme for decoding the output spatial audio format to an output layout, the method comprising: receiving the input audio signals, optimizing the conversion matrix based on a cost function, wherein the cost function is based on parameters descriptive of the encoding scheme, parameters descriptive of the decoding scheme, and the conversion matrix; and outputting the output audio signals by processing the input audio signals with the optimized conversion matrix.

2. The method according to claim 1, wherein the input spatial audio format is specified by the first set of directions and an encoding matrix for encoding the input audio signals from respective ones of the first set of directions to channels of the spatial input format; wherein the output spatial audio format is specified by a second set of directions and a decoding-to-speakers matrix for decoding channels of the output spatial audio format to the output audio signals for loudspeakers at respective ones of the second set of directions; and wherein the cost function is based on the encoding matrix, the decoding-to-speakers matrix, and the conversion matrix.

3. The method according to claim 1 or 2, wherein the cost function is based on a system matrix for linking the input audio signals to the output audio signals; and wherein the system matrix is based on the conversion matrix.

4. The method according to claim 3 when depending on claim 2, wherein the system matrix is further based on the encoding matrix and the decoding-to-speakers matrix.

5. The method according to claim 2 or any claim dependent on claim 2, wherein the cost function is further based on the first set of directions and the second set of directions.

6. The method according to any one of the preceding claims, wherein at least one term of the cost function is based on a psychoacoustic quantity.

7. The method according to any one of the preceding claims, wherein at least one term of the cost function is based on one or more of: sound pressure; acoustic intensity; sound energy; and acoustic velocity.

8. The method according to claim 3 or any claim dependent on claim 3, wherein the cost function comprises a term based on a measure of sparsity or non-sparsity of the system matrix.

9. The method according to any one of the preceding claims, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of the sound pressure.

10. The method according to any one of the preceding claims, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of a radial component of the acoustic velocity.

11. The method according to any one of the preceding claims, wherein the cost function comprises a term indicative of a mean quadratic deviation from zero phase of a transverse component of the acoustic velocity.

12. The method according to any one of claims 9 to 11 when depending on claim 3, wherein said term is based on the system matrix.

13. The method according to claim 3 or any claim dependent on claim 3, wherein the cost function comprises a term that is quadratic in terms of coefficients of the system matrix.

14. The method according to claim 3 or any claim dependent on claim 3, wherein the cost function comprises a term that is quartic in terms of coefficients of the system matrix.

15. The method according to claim 3 or any claim dependent on claim 3, wherein the cost function comprises a term penalizing non-sparsity of the system matrix.

16. The method according to any one of the preceding claims, wherein the cost function comprises a term penalizing large values of elements of the conversion matrix.

17. The method according to any one of claims 1 to 16, wherein the cost function comprises a term penalizing elements in the conversion matrix associated with channels whose noise level exceeds a predefined threshold.

18. The method according to any one of claims 1 to 17, wherein the encoding scheme is complex-valued.

19. The method according to any one of claims 1 to 17, wherein the encoding scheme is real-valued.

20. The method according to any one of claims 1 to 19, wherein the output layout relates to a physical loudspeaker layout.

21. The method according to any one of claims 1 to 19, wherein the output layout relates to a virtual loudspeaker layout.

22. The method according to any one of the preceding claims, wherein the output spatial audio format is one of Ambisonics, a multichannel format, and a linear decoding format.

23. The method according to any one of the preceding claims, wherein the input spatial audio format is one of Ambisonics, a multichannel format, and a linear encoding format.

24. The method according to any one of claims 1 to 23, wherein encoding the first set of directions to the input spatial audio format relates to recording a set of input audio signals via a microphone array.

25. The method according to any one of claims 1 to 23, wherein encoding the first set of directions to the input spatial audio format involves a one-to-one mapping between the input audio signals and the channels of the input spatial audio format.

26. The method according to any one of claims 1 to 25, wherein decoding the output spatial audio format to the output layout involves a one-to-one mapping between the channels of the output spatial audio format and the output audio signals.

27. The method according to any one of the preceding claims, wherein at least one of the encoding scheme and the decoding scheme is frequency dependent; and wherein the conversion matrix is separately optimized in each of a plurality of frequency bands.

28. The method according to claim 27, wherein the cost function is frequency dependent.

29. The method according to claim 27 or 28, wherein the conversion matrix is optimized simultaneously across each of a plurality of frequency bands; and wherein the cost function comprises a contribution for each of the plurality of frequency bands and one or more terms penalizing discontinuities of the conversion matrix between neighboring frequency bands.

30. The method according to any one of claims 27 to 29, further comprising: determining an effective cost function for a given frequency band,wherein the conversion matrix is optimized based on the effective cost function for the given frequency band; and wherein determining the effective cost function for the given frequency band is based on a plurality of cost functions at respective frequencies between a lower limit frequency of the given frequency band and an upper limit frequency of the given frequency band.

31. An apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 30.

32. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 30.

33. A computer-readable storage medium storing the program of claim 32.

Images