Processing parametrically encoded audio

Abstract

A method includes receiving a first input bitstream for a first parametrically encoded input audio signal. The first input bitstream includes data representing the first input core audio signal and a first set including at least one spatial parameter related to the first parametrically encoded input audio signal. A first covariance matrix for the first parametrically encoded audio signal is determined based on the first set of spatial parameters. A modified set including the at least one spatial parameter is determined based on the determined output covariance matrix. The modified set is different from the first set. An output core audio signal is determined based on or constructed from the first input core audio signal. An output bitstream for the parametrically encoded output audio signal is generated. The output bitstream includes data representing the output core audio signal and the modified set.

Description

[Technical Field]

[0001] (Reference to related applications) This application claims priority under U.S. Provisional Patent Application No. 63 / 075,889 filed September 9, 2020, and European Patent Application No. 20195258.7 filed September 9, 2020, with all disclosures of each application incorporated herein.

[0002] Embodiments of the present invention relate to audio processing. Specifically, embodiments of the present invention relate to the processing of parametrically encoded audio. [Background technology]

[0003] Audio codecs have evolved from strictly spectral coefficient quantization and coding (e.g., in the Modified Discrete Cosine Transform (MDCT) domain) to hybrid coding methods, including parametric coding methods, in order to extend bandwidth and / or channel count from mono (or low channel count) core signals. Examples of such (spatial) parametric coding methods include MPEG Parametric Stereo (High-Efficiency Advanced Audio Coding (HE-AAC) v2), MPEG Surround, and tools for joint coding of channels and / or objects in Dolby AC-4 audio systems, such as Advanced Coupling (A-CPL), Advanced Joint Channel Coding (A-JCC), and Advanced Joint Object Coding (A-JOC). Several audio streams may be combined (mixed) to produce an output bitstream. It is desirable to improve efficiency in processing parametrically coded audio. [Overview of the project]

Problems to be Solved by the Invention

[0004] A method, a system, and a non - transient computer - readable medium for processing parametrically encoded audio are disclosed.

Means for Solving the Problems

[0005] A first aspect relates to a method. The method includes receiving a first input bitstream for a first parametrically encoded input audio signal. The first input bitstream includes data representing a first input core audio signal and a first set including at least one spatial parameter related to the first parametrically encoded input audio signal. Based on the first set of spatial parameters, a first covariance matrix of the first parametrically encoded audio signal is determined. Based on the determined output covariance matrix, a modified set including at least one spatial parameter is determined. The modified set is different from the first set. An output core audio signal is determined based on or constituted by the first input core audio signal. An output bitstream for the parametrically encoded output audio signal is generated. The output bitstream includes data representing the output core audio signal and the modified set.

[0006] A second aspect relates to a system. The system includes one or more processors (e.g., computer processors). The system includes a non - transient computer - readable medium storing instructions configured to cause the one or more processors to perform the method according to the first aspect when executed by the one or more processors.

[0007] A third aspect relates to a non-temporary computer-readable medium, which stores instructions configured to cause one or more processors (e.g., computer processors) to perform the method according to the first aspect when executed by one or more processors.

[0008] Embodiments of the present invention can improve efficiency in processing parametrically encoded audio (e.g., without requiring complete decoding of the entire audio stream), provide higher quality (without requiring re-encoding of the audio stream), and have relatively low latency. Embodiments of the present invention are suitable for handling immersive audio signals (such as audio signals for conferences). Embodiments of the present invention are suitable for mixing immersive audio signals. Further advantages and / or technical effects relating to embodiments of the present invention are described and made apparent by the following description (e.g., the following description relating to the accompanying drawings).

[0009] Embodiments of the present invention are applicable, for example, to audio codecs that re-instate spatial parameters between channels. Examples of such audio codecs include MPEG Surround, HE-AAC v2 Parametric Stereo, AC-4 (A-CPL, A-JCC), AC-4 Immersive Stereo, or Binaural Cue Coding (BCC). These spatial parametric coding methods are described in Breebaart, J., Faller, C. (2007), "Spatial Audio Processing: MPEG Surround and other applications", Wiley, ISBN: 978-0-470-03350-0 (all provisions of the said document are incorporated herein by reference for all purposes). Embodiments of the present invention are also applicable to audio codecs that enable the combination of channel-based audio content, object-based audio content, and scene-based audio content. Examples of such audio codecs include Dolby Digital Plus Joint Object Coding (DD+JOC) and Dolby AC-4 Advanced Joint Object Coding (AC-4 A-JOC).

[0010] In the context of the present application, in the context of determining a modified set containing at least one spatial parameter based on a determined first covariance matrix (where the modified set is different from the first set), when we say that the modified set containing at least one spatial parameter is different from another set containing at least one spatial parameter (e.g., the first set), it may mean that at least one element (or spatial parameter) of the modified set is different from an element (or spatial parameter) of the first set. [Brief explanation of the drawing]

[0011] Embodiments of the present invention will be described in more detail with reference to the accompanying drawings illustrating embodiments of the present invention.

[0012] [Figure 1] Figure 1 is a schematic diagram of a system according to an embodiment of the present invention. [Figure 2] Figure 2 is a schematic diagram of a system according to an embodiment of the present invention. [Figure 3] Figure 3 is a schematic diagram of a system according to an embodiment of the present invention. [Figure 4] Figure 4 is a schematic diagram of a system according to an embodiment of the present invention. [Modes for carrying out the invention]

[0013] Detailed description of the embodiment When it is necessary to combine (mix) several audio streams to generate an output bitstream, conventional techniques for parametric spatial coding schemes such as MPEG parametric stereo coding may require the following steps: 1. Use a core encoder to decode a mono (or low-channel) core signal. 2. Transform the time-domain signal into an oversampled (and potentially complex-valued) representation (e.g., using a Discrete Fourier Transform (DFT) or a Quadratic Mirror Filter (QMF)). 3. A step to reconstruct a higher number of channel representation by restoring (re-instateting) the spatial parameters. 4. The reconstructed higher channel count representation is inversely transformed to generate a time-domain audio signal. 5. A step to mix time-domain audio signals from multiple audio streams. 6. The step of converting the mixed time-domain audio signal to an oversampled (and potentially complex-valued) representation (e.g., using DFT or QMF). 7. A step to generate a low-channel (mono) downmix by downmixing. 8. A step to extract spatial parameters from the mixture. 9. A step to convert the downmixed signal back into the time domain. 10. A step to encode the downmixed signal using a core encoder.

[0014] Steps 4, 5, and 6 above may be combined depending on the circumstances. However, mixing involves decoding, parametric reconstruction, mixing, parameter extraction, and re-encoding of all audio streams. These steps may have the following disadvantages: For example, in long-distance communication applications, latency (delay) introduced by multiple subsequent conversions can become significant, or even problematic. • Decoding and re-encoding may result in a perceived loss of sound quality that is undesirable to the user, especially when parametric encoding tools are employed. This perceived loss of sound quality may be due to parameter quantization and the replacement of residual signals by the decorrelator output. • The conversion, decoding, and re-encoding steps can introduce considerable complexity. This can place a significant computational burden on the provider or device performing the mixing. This can increase costs or reduce battery life for the device performing the mixing.

[0015] According to one or more embodiments of the present invention, one or more input bitstreams (or input streams) can be received for each parametrically encoded input audio signal. Based on the spatial parameters of each or any of the input bitstreams, a covariance matrix of the (desired) output presentation can be determined (e.g., reconstructed or estimated). The covariance matrices for two or more input bitstreams may be combined to obtain an output covariance matrix or a combined covariance matrix. A core audio signal or stream (e.g., a low-channel (mono, etc.) core audio signal or stream) can be combined for the two or more input bitstreams. New spatial parameters can be determined (e.g., extracted) from the output covariance matrix. An output bitstream can be generated from the determined spatial parameters and the combined core signal.

[0016] Embodiments of the present invention, such as those described below with reference to the above embodiments and accompanying drawings, can, for example, improve efficiency in the processing of parametrically encoded audio.

[0017] Figure 1 is a schematic diagram of a system 100 according to one embodiment of the present invention. The system 100 may include one or more processors and a non-temporary computer-readable medium storing instructions configured to cause the one or more processors to perform a method according to one embodiment of the present invention when executed by the one or more processors.

[0018] A first input bitstream 10 is received for a first parametrically encoded input audio signal. The first input bitstream includes data representing a first input core audio signal and a first set of data including at least one spatial parameter relating to the first parametrically encoded input audio signal. System 100 may include a demultiplexer 20 (e.g., a first demultiplexer) which can be configured to separate (e.g., multiplex desegregate) the first input bitstream 10 into a first input core audio signal 21 and a first set 22 including at least one spatial parameter relating to the first parametrically encoded input audio signal. The demultiplexer 20 may also be referred to as a (first) bitstream processing unit, a (first) bitstream separation unit, and so on.

[0019] The first input bitstream 10 may include, or be composed of, a core audio stream, such as an audio signal encoded by a core encoder.

[0020] Based on the spatial parameters of a first set, a first covariance matrix 31 of a first parametrically encoded audio signal is determined. To do this, the system 100 may include a covariance matrix determination unit 30, which may be configured to determine the first covariance matrix 31 of a first parametrically encoded audio signal based on the spatial parameters of a first set 22. As illustrated in Figure 1, the first set 22 may be output from a demultiplexer 20 and then input to the covariance matrix determination unit 30.

[0021] Determining the first covariance matrix 31 may involve determining at least some or all of the diagonal elements and off-diagonal elements of the first covariance matrix 31.

[0022] A modified set 41 containing at least one spatial parameter is determined based on a determined first covariance matrix, where the modified set is different from the first set. To do this, the system 100 may include a spatial parameter determination unit 40, which may be configured to determine a modified set 41 containing at least one spatial parameter based on a determined first covariance matrix 31. As illustrated in Figure 1, the determined first covariance matrix 31 may be output from the covariance matrix determination unit 30 and then input to the spatial parameter determination unit 40.

[0023] The output core audio signal may be determined based on or composed of a first input core audio signal. According to an embodiment of the present invention illustrated in Figure 1, the output core audio signal is composed of the first input core audio signal 21.

[0024] An output bitstream 51 is generated for a parametrically encoded output audio signal. This output bitstream includes data representing the output core audio signal and the modified set 41. To do this, the system 100 may include an output bitstream generation unit 50, which can be configured to generate an output bitstream 51 for a parametrically encoded output audio signal, where the output bitstream 51 includes data representing the output core audio signal and the modified set 41. As illustrated in Figure 1, the output bitstream generation unit 50 may receive an output core audio signal (composed of a first input core audio signal 21 according to the embodiment of the invention illustrated in Figure 1) and the modified set 41 as inputs, and output an output bitstream 51. The output bitstream generation unit 50 may be configured to multiplex the output core audio signal and the modified set 41. The output core audio signal may be determined, for example, by the output bitstream generation unit 50.

[0025] The first parametrically encoded input audio signal may represent sounds captured from at least two different microphones, such as sounds captured from a stereo or first-order ambisonic microphone. This is just one example, and it should be understood that, in general, the first parametrically encoded input audio signal (or the first input bitstream 10) can, in principle, represent any captured sound or any captured audio content.

[0026] Compared to conventional techniques for processing parametrically encoded audio, the processing of parametrically encoded audio illustrated in Figure 1 may require less or no complete decoding and / or re-encoding of all audio streams. As a result, the processing of parametrically encoded audio, as illustrated in Figure 1, can have relatively high efficiency and / or quality.

[0027] A first parametrically encoded input audio signal and a parametrically encoded output audio signal may use the same spatial parameterization encoding type. Alternatively, a first parametrically encoded input audio signal and a parametrically encoded output audio signal may use different spatial parameterization encoding types. Different spatial parametric encoding types may include, for example, MPEG parametric stereo parameterization, binaural cue coding, spatial audio reconstruction (SPAR), object parameterization in Joint Object Coding (JOC) or Advanced JOC (A-JOC) (e.g., object parameterization in A-JOC for Dolby AC-4), or Dolby AC-4 Advanced Coupling (A-CPL) parameterization. Thus, the first parametrically encoded input audio signal and the parametrically encoded output audio signal may use different types of encoding, such as MPEG parametric stereo parameterization, binaural cue coding, SPAR (or similar encoding types), JOC, A-JOC, or A-CPL parameterization. Therefore, using systems and methods according to one or more embodiments of the present invention, it is possible to convert codes between one spatial parametric encoding method and another spatial parametric encoding method without requiring complete decoding and re-encoding of the output signal.SPAR is described, for example, in the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), "Immersive Audio Coding for Virtual Reality Using a Metadata-assisted Extension of the 3GPP EVS Codec", McGrath, Bruhn, Purnhagen, Eckert, Torres, Brown, and Darcy, 12-17 May 2019, and in the 3GPP TSG-SA4 #99 meeting, Tdoc S4-180806, 9-13 July 2018, Rome, Italy. All of the contents of both documents are incorporated herein by reference for any purpose. JOC and A-JOC are described, for example, in Villemoes, L., Hirvonen, T., Purnhagen, H. (2017), "Decorrelation for audio object coding", 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), and Purnhagen, H., Hirvonen, T., Villemoes, L., Samuelsson, J., Klejsa, J., "Immersive Audio Delivery Using Joint Object Coding", Dolby Sweden AB, Stockholm, Sweden, Audio Engineering Society (AES) Convention: 140 (May 2016) Paper Number: 9587 (all of the contents of said documents are incorporated herein by reference for any purpose).

[0028] Spatial parameterization tools and techniques can be used to determine (e.g., reconstruct or estimate) a normalized covariance matrix, such as a covariance matrix independent of the total signal level. In such cases, several solutions can be used to determine the covariance matrix. For example, one or more of the following methods can be used: The signal level can be measured from the core audio representation. Then, the normalized covariance estimate can be scaled to ensure that the signal autocorrelation is correct. • Bitstream elements can be added to represent the (total) signal level at each time / frequency tile. • Instead of normalized covariance, unnormalized covariance can be included in the bitstream. • A quantized representation of audio levels in time / frequency tiles may already exist in some bitstream format. This data can then be used to appropriately scale the normalized covariance matrix. For example, any combination of the above methods, by adding (delta) energy data in the bitstream that represents the difference between the estimated total power derived from the core audio representation and the actual total power.

[0029] According to one or more embodiments of the present invention, the covariance matrix can be determined (e.g., reconstructed or estimated) and parameterized in individual time / frequency tiles, subbands, or audio frames.

[0030] Although the elements of system 100 have been described above as separate components, it should be understood that system 100 may include one or more processors that can be configured to implement the above functions of the demultiplexer 20, covariance matrix determination unit 30, spatial parameter determination unit 40, and output bitstream generation unit 50. Each or any of these functions may be implemented by, for example, one or more processors. For example, one (e.g., a single) processor may implement the above functions of the demultiplexer 20, covariance matrix determination unit 30, spatial parameter determination unit 40, and output bitstream generation unit 50. Alternatively, each of the above functions of the demultiplexer 20, covariance matrix determination unit 30, spatial parameter determination unit 40, and output bitstream generation unit 50 may be implemented by separate processors.

[0031] According to one or more embodiments of the present invention, there may be input bitstreams having spatial parameters (e.g., the first input bitstream 10 illustrated in Figure 1) or input bitstreams that do not have spatial parameters and are mono-only. In addition to the processing of parametrically encoded audio illustrated in Figure 1 (or Figure 2), a second input bitstream for a mono audio signal may be received (the second input bitstream for a mono audio signal is not shown in Figure 1). The second input bitstream may contain data representing the mono audio signal. A second covariance matrix may be determined based on the mono audio signal and a matrix containing desired spatial parameters for the second input bitstream (this second input bitstream is mono-only). A composite core audio signal may be determined based on the first input core audio signal and the mono audio signal. A composite covariance matrix may be determined based on the determined first covariance matrix and the determined second covariance matrix (e.g., by summing the first and second covariance matrices). A modified set may be determined based on the determined composite covariance matrix. Here, the modified set is different from the first set. The output core audio signal can be determined based on the synthesized core audio signal. For example, the second covariance matrix is ​​the energy of the mono audio signal (if the mono audio signal is denoted by matrix Y, the energy is YY). * It is given by, * The desired spatial parameters for the second input bitstream may be determined based on a matrix containing the conjugate transpose and desired spatial parameters for the second input bitstream. The desired spatial parameters for the second input bitstream may include, for example, one or more of the amplitude panning parameters or head transfer function parameters (for mono objects associated with mono audio signals).

[0032] Figure 2 is a schematic diagram of a system 200 according to another embodiment of the present invention. The system 200 may comprise one or more processors and a non-temporary computer-readable medium storing instructions configured to cause the one or more processors to perform a method according to one embodiment of the present invention when executed by the one or more processors. The system 200 illustrated in Figure 2 is similar to the system 100 illustrated in Figure 1. The same reference numerals in Figures 1 and 2 indicate the same or similar elements having the same or similar functions. The following description of the embodiment of the present invention illustrated in Figure 2 will mainly describe the differences from the embodiment of the present invention illustrated in Figure 1. Therefore, features common to both embodiments may be omitted in the following description. Thus, features of the embodiment of the present invention illustrated in Figure 1 should be considered to be implemented, or at least implementable, in the embodiment of the present invention illustrated in Figure 2, unless otherwise specifically stated in the following description.

[0033] In contrast to the system 100 illustrated in Figure 1, in the system 200 illustrated in Figure 2, the determined first covariance matrix 31 is modified based on the output bitstream presentation transformation data of the first input bitstream 10 before the modified set 41 is determined. Here, the output bitstream presentation transformation data includes one set of signals intended for playback on a selected audio playback system. To do this, the system 200 may include a covariance matrix modification unit 130. The covariance matrix modification unit 130 may be configured to modify the determined first covariance matrix 31 based on the output bitstream presentation transformation data 132 of the first input bitstream 10. As illustrated in Figure 2, the covariance matrix modification unit 130 receives, as input, (1) output bitstream presentation transformation data 132 of the first input bitstream 10 and (2) the first covariance matrix 31 after it has been output from the covariance matrix determination unit 30, and may output a modified first covariance matrix 131 (compared to the first covariance matrix 31 before it was output from the covariance matrix determination unit 30 and modified in the covariance matrix modification unit 130). Based on the modified first covariance matrix 131, the covariance matrix modification unit 130 determines a modified set 41 containing at least one spatial parameter, where the modified set 41 is different from the first set 22. The spatial parameter determination unit 40 illustrated in Figure 2 may be configured to determine the modified set 41 based on the modified first covariance matrix 131.

[0034] Thus, according to the embodiment of the present invention illustrated in Figure 2, presentation transforms (such as mono, stereo, or binaural) can be integrated into the processing of parametrically encoded audio based on the manipulation or modification of the covariance matrix.

[0035] Examples of presentation transformations that can (effectively) modify the covariance matrix include, but are not limited to, the following: (1) A transformation that can be described as a matrix operation (time and / or frequency dependent, and optionally complex valued) from an input signal to an output signal. When representing a stereo input signal as matrix Y, an output signal as matrix X, and the transformation as matrix D, the presentation transformation can be expressed as X = DY. Thus, the covariance matrix R XX of the output signal X is related to the covariance matrix R YY of the input signal Y such that R XX = DR YY D * . Here, * represents the conjugate transpose. Thus, in these cases, the presentation transformation can be achieved by changing the covariance matrix given by R XX = DR YY D * . Examples of such presentation transformations include downmixing, remixing, rotation of a scene, or conversion from a (binaural) loudspeaker presentation to a headphone presentation. (2) A change based on auditory scene analysis that is derived from and changes the covariance matrix (such as changing the position of one or more speakers in a conference call or rotating the sound field) (see US9,979,829B2, the entire content of which is incorporated herein by reference for all purposes).

[0036] For example, referring to the above example (1) and further to Figure 2, the output bitstream presentation transformation data 132 may include, for example, downmixing transformation data for downmixing the first input bitstream 10, remixing transformation data for remixing the first input bitstream 10, or headphone transformation data for transforming the first input bitstream 10. The headphone transformation data may include a set of signals intended for playback on headphones.

[0037] The following explains how presentation transforms are used in the covariance domain. Assume that one subband of a multichannel signal is represented as X[c,k], where k is the sample index and c is the channel index. XX Given , the covariance matrix of X[c,k] is given as follows:

number

number

[0038] The resulting output signal R YY The covariance matrix is ​​given by the following:

number

[0039] In other words, transformation C is R XX This can be applied by pre and post matrices applied to it. One example where this transformation may be particularly useful is when there are several incoming input bitstreams (e.g., Figure 3 and its description) and one input bitstream represents a mono microphone feed that needs to be transformed into a binaural presentation in the output bitstream. In that case, the subband matrix C may consist of complex-valued gains representing the desired head transfer function in the subband domain.

[0040] Although the elements of system 200 have been described above as separate components, it should be understood that system 200 may comprise one or more processors that can be configured to implement the above functions of the demultiplexer 20, covariance matrix determination unit 30, covariance matrix modification unit 130, spatial parameter determination unit 40, and output bitstream generation unit 50. Each or any of these functions may be implemented by, for example, one or more processors. For example, one (e.g., a single) processor may implement the above functions of the demultiplexer 20, covariance matrix determination unit 30, covariance matrix modification unit 130, spatial parameter determination unit 40, and output bitstream generation unit 50, or each of the above functions of the demultiplexer 20, covariance matrix determination unit 30, covariance matrix modification unit 130, spatial parameter determination unit 40, and output bitstream generation unit 50 may be implemented by separate processors.

[0041] Figure 3 is a schematic diagram of a system 300 according to another embodiment of the present invention. The system 300 may comprise one or more processors and a non-temporary computer-readable medium storing instructions configured to cause the one or more processors to perform a method according to a certain embodiment of the present invention when executed by the one or more processors. The system 300 illustrated in Figure 3 is similar to the system 100 illustrated in Figure 1. The same reference numerals in Figures 1 and 3 indicate the same or similar elements having the same or similar functions. The following description of the embodiment of the present invention illustrated in Figure 3 will mainly describe the differences from the embodiment of the present invention illustrated in Figure 1. Therefore, features common to both embodiments may be omitted in the following description. Thus, features of the embodiment of the present invention illustrated in Figure 1 should be considered to be implemented, or at least implementable, in the embodiment of the present invention illustrated in Figure 3, unless otherwise specifically stated in the following description.

[0042] Compared to Figure 1, Figure 3 receives one more input bitstream.

[0043] As shown in Figure 3, a first input bitstream 10 is received for a first parametrically encoded input audio signal. The first input bitstream includes data representing a first input core audio signal and a first set of data including at least one spatial parameter relating to the first parametrically encoded input audio signal. The system 300 may include a demultiplexer 20 (e.g., a first demultiplexer) which can be configured to separate (e.g., multiplex desegregate) the first input bitstream 10 into a first input core audio signal 21 and a first set 22 including at least one spatial parameter relating to the first parametrically encoded input audio signal. The demultiplexer 20 may also be referred to as a (first) bitstream processing unit, a (first) bitstream separation unit, and so on.

[0044] A first covariance matrix 31 of a first parametrically encoded audio signal is determined based on a first set of spatial parameters. To do this, the system 300 may include a covariance matrix determination unit 30, which can be configured to determine the first covariance matrix 31 of the first parametrically encoded audio signal based on a first set of spatial parameters 22. This first set 22 may be output from a demultiplexer 20 and then input to the covariance matrix determination unit 30, as shown in Figure 3.

[0045] Determining the first covariance matrix 31 may involve determining at least some or all of the diagonal elements and off-diagonal elements of the first covariance matrix 31.

[0046] As further illustrated in Figure 3, a second input bitstream 60 is received for a second parametrically encoded input audio signal. The second input bitstream includes data representing a second input core audio signal and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal. The system 300 may include a demultiplexer (or second demultiplexer) 70 that can be configured to separate (e.g., multiplex desegregate) the second input bitstream 60 into a second input core audio signal 71 and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal 72. The (second) demultiplexer 70 may also be referred to as a (second) bitstream processing unit, a (second) bitstream separation unit, and so on.

[0047] Each or both of the first input bitstream 10 and the second input bitstream 60 may include, or be composed of, a core audio stream, such as an audio signal encoded by a core encoder.

[0048] Based on the spatial parameters of a second set, a second covariance matrix 81 of the second parametrically encoded audio signal is determined. To do this, the system 300 may include a covariance matrix determination unit 80 (e.g., a second covariance matrix determination unit) which can be configured to determine the second covariance matrix 81 of the second parametrically encoded audio signal based on the spatial parameters of a second set 72. This second set 72 may be output from the demultiplexer 70 and then input to the covariance matrix determination unit 80, as shown in Figure 3.

[0049] Determining the second covariance matrix 81 may involve determining at least some or all of the diagonal elements and off-diagonal elements of the second covariance matrix 81.

[0050] A synthesized core audio signal 91 is determined based on a first input core audio signal 21 and a second input core audio signal 71. An output covariance matrix 92 is determined based on the determined first covariance matrix 31 and a determined second covariance matrix 81. To do this, the system 300 may include a synthesizer unit 90. The synthesizer unit 90 may be configured to determine a synthesized core audio signal 91 based on a first input core audio signal 21 and a second input core audio signal 71. The synthesizer unit 90 may be configured to determine an output covariance matrix 92 based on the determined first covariance matrix 31 and a determined second covariance matrix 81. As shown in Figure 3, the first input core audio signal 21 and the second input core audio signal 71 are output from the demultiplexer 20 and demultiplexer 70, respectively, and then input to the combiner unit 90. The determined first covariance matrix 31 and the determined second covariance matrix 81 are output from the covariance matrix determination unit 30 and covariance matrix determination unit 80, respectively, and then input to the combiner unit 90.

[0051] Determining the output covariance matrix 92 may include, for example, the step of calculating the sum of the determined first covariance matrix 31 and the determined second covariance matrix 81. The sum of the first covariance matrix 31 and the second covariance matrix 81 may constitute the output covariance matrix 92.

[0052] Examples of methods for mixing or synthesizing parametrically encoded audio signals and covariance matrices are described below, using the notation of Villemoes, L., Hirvonen, T., Purnhagen, H. (2017), “Decorrelation for audio object coding”, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (all content of that document is incorporated herein by reference for all purposes).

[0053] Consider the original N-channel signal X. The original N-channel signal X is downmixed into an M-channel signal Y=DX in the encoder, where D is the M×N downmix matrix. In the decoder, the input signal is approximated... JPEG0007829561000004.jpg52 can be reconstructed from the downmix signal Y as follows:

number

number

[0054] The parameters C, P, and Q are calculated for each time / frequency tile, and a complete covariance reinstatement is performed. The calculation is performed so that JPEG0007829561000007.jpg516 is achieved. Here, R UV =Re(UV * ) is the sample covariance matrix. The calculation of C, P, and Q is performed using the original covariance matrix R as input. XX And it may only require the downmix matrix D. These parameters are such that the upmix is ​​"downmix compatible," i.e., It is possible to calculate the result to be JPEG0007829561000008.jpg511. The covariance of the decoded signal is given by the following:

number

[0055] The two spatial signals X1 and X2 can be combined into a mixed signal with N3 channels, which is a weighted sum.

number

[0056] If signals X1 and X2 are available in a parametrically encoded form, signals X1 and X2 can be decoded and added to obtain the following:

number

[0057] It may be desirable to mix the signals in the parametric / downmix domain because this involves the complete decoding, mixing, and mixing of the two signals. 3C This is because it can have various advantages, such as one or more of the following, compared to re-encoding it afterward. 1. Lower computational complexity. 2. Lower latency by avoiding the computation of filter banks required to process time / frequency tiles. 3. Improved quality by avoiding cascading decorrelation.

[0058] In the following, N, M, K, and D are: The same applies to JPEG0007829561000014.jpg818, where D is known beforehand, and the mixing weight matrix is ​​the identity matrix G1=G2=I with N1=N2=N3=N, so we assume the desired mixed signal is simply the sum of the two original signals. The input to the mixing process in the parametric / downmix domain is given by parameters C1, P1, Q1 and C2, P2, Q2 along with the downmix signals Y1 and Y2. What we need to do here is, first Y 3P and C 3P , P 3P Q 3P The calculation is as follows: Here, the subscript "P" indicates that mixing occurs in the parametric / downmix domain.

[0059] The downmix of the sum X3 can be determined without approximation as follows:

number

[0060] Covariance matrix R of the desired mixing product X3 X3X3 The calculation (or approximation) of the decoded signal is not very simple. JPEG0007829561000016.jpg818's sum X 3C The covariance matrix can be written as follows:

number

[0061] The first two contributions can be derived as follows:

number

number

[0062] Assuming that all decorrelators d1() and d2() are decorrelated with respect to each other, it should be correct to assume that all elements of this sum are zero except for the first element. This is true in R X3CX3C This means that the last two contributions to can be approximated using the following:

number

[0063] Taking this approximation into account, the sum X 3C The covariance matrix can now be written as follows:

number

[0064] This is R X3CX3C For an approximation of this to be calculated, when mixing signals in the parametric / downmix domain, R Y1Y1 , R Y2Y2 , and R Y1Y2 This means that R must be known. Y1Y1 , R Y2Y2 , and R Y1Y2 This can be derived by analyzing the actual downmix signals Y1 and Y2 (which may require some form of analysis filter bank or transformation to allow access to time / frequency tiles, and may imply some latency). Alternatively, R Y1Y1 and R Y2Y2 It can even be transmitted as a bitstream (per time / frequency tile), and furthermore, for example, a downmixed signal is uncorrelated, i.e., R Y1Y2 We can assume that R = 0. X3CX3COne of these approximations is R X3PX3P It is used with the known D, and C is used in the same way as in the original parametric encoder. 3P , P 3P , and Q 3P Calculate the above and determine Y 3P It can be used together with other items.

[0065] As described above, the covariance of the downmix signal (for example, R Y1Y1 and R Y2Y2 The covariance of the downmix signal (e.g., R) can be determined (e.g., calculated) from the received bitstream. Y1Y1 and R Y2Y2 Information about ) can be embedded in the received bitstream. Downmix is ​​uncorrelated (e.g., R Y1Y2 It can be assumed that = 0.

[0066] The following may apply to parametric stereo implemented in Dolby AC-4A-CPL.

number

number

[0067] Specifically, if the first input stream has A-CPL parameters (a1, b1) and the second input stream has A-CPL parameters (a2, b2), and these two input streams represent independent signals, then the sum of these two streams has A-CPL parameters (a, b) given by the following:

number

[0068] In addition to the above description illustrating a method and an example of a covariance matrix for mixing or combining parametrically encoded audio signals, the following examples illustrate a method for determining the covariance matrix of a parametrically encoded audio signal, using the same notation as described above. The step of determining the covariance matrix of a parametrically encoded audio signal (e.g., a first covariance matrix 31 or a second covariance matrix 81) based on spatial parameters relating to the parametrically encoded audio signal that may be contained within the bitstream of the parametrically encoded audio signal may include, for example, (1) determining a downmix signal of the parametrically encoded audio signal, (2) determining the covariance matrix of the downmix signal, and (3) determining the covariance matrix based on the covariance matrix of the downmix signal and the spatial parameters relating to the parametrically encoded audio signal. For example, as described above illustrating a method and an example of a covariance matrix for mixing or combining parametrically encoded audio signals, the original N-channel signal X can be downmixed in the encoder to an M-channel signal Y=DX. Here, D is the M×N downmix matrix. In the decoder, the input signal is approximated. JPEG0007829561000025.jpg52 is derived from the downmix signal Y,

number

number

[0069] Embodiments of the present invention are not limited to the step of determining the output covariance matrix 92 by calculating the sum of a determined first covariance matrix 31 and a determined second covariance matrix 81. For example, the step of determining the output covariance matrix 92 may include determining the output covariance matrix 92 as the one of the determined first covariance matrix 31 and the determined second covariance matrix 81 that has a larger sum of diagonal elements. The step of determining the output covariance matrix 92 in this way may involve determining the output covariance matrix 92 based on an energy criterion across inputs, for example, determining the output covariance matrix 92 as the one of the determined first covariance matrix 31 and the determined second covariance matrix 81 that has the maximum energy across all inputs.

[0070] Referring further to Figure 3, a modified set 111 containing at least one spatial parameter is determined based on the determined output covariance matrix. Here, the modified set 111 is different from the first set 22 and the second set 72. To do this, the system 300 may include a spatial parameter determination unit 110, which can be configured to determine the modified set 111 containing at least one spatial parameter based on the determined output covariance matrix 92. This determined output covariance matrix 92 may be output from the synthesizer unit 90 and then input to the spatial parameter determination unit 110, as shown in Figure 3.

[0071] The output core audio signal is determined based on the synthesized core audio signal 91. The output core audio signal may be composed of, for example, the synthesized core audio signal 91. More generally, the output core audio signal may be based on a first input core audio signal 21 and a second input core audio signal 71.

[0072] An output bitstream 121 is generated for a parametrically encoded output audio signal. This output bitstream includes data representing the output core audio signal and the modified set. To do this, the system 300 may include an output bitstream generation unit 120, which can be configured to generate an output bitstream 121 for a parametrically encoded output audio signal. Here, the output bitstream 121 includes data representing the output core audio signal and the modified set 111. As shown in Figure 3, the output bitstream generation unit 120 may receive the output core audio signal and the modified set 111 output from the combiner 90 as input and output the output bitstream 121. The output bitstream generation unit 120 may be configured to multiplex the output core audio signal and the modified set 111. The output core audio signal may be determined, for example, by the output bitstream generation unit 120.

[0073] The first parametrically encoded input audio signal and / or the second parametrically encoded input audio signal may represent sounds captured from at least two different microphones, such as sounds captured from a stereo or first-order ambisonic microphone. This is merely an example, and it should be understood that, in general, the first parametrically encoded input audio signal and / or the second parametrically encoded input audio signal (or the first input bitstream 10 and / or the second input bitstream 60) may, in principle, represent any captured sound or any captured audio content.

[0074] Compared to conventional techniques for processing parametrically encoded audio, the processing of parametrically encoded audio illustrated in Figure 3 may require less or no complete decoding and / or re-encoding of all audio streams. As a result, the processing of parametrically encoded audio, as illustrated in Figure 3, can have relatively high efficiency and / or quality.

[0075] Furthermore, if the input bitstreams (for example, the first input bitstream 10 and the second input bitstream 60, and optionally any further input bitstreams) have synchronized frames, no (further) latency is introduced by combining the input bitstreams using a system according to one or more embodiments of the present invention, such as the system 300 illustrated in Figure 3. Thus, compared to conventional techniques for processing parametrically encoded audio, the processing of parametrically encoded audio illustrated in Figure 3 can have relatively low latency for processing such as mixing.

[0076] The first parametrically encoded input audio signal, the second parametrically encoded input audio signal, and the parametrically encoded output audio signal can all use the same spatial parametric encoding type.

[0077] At least two of the first parametrically encoded input audio signal, the second parametrically encoded input audio signal, and the parametrically encoded output audio signal may use different spatial parametric encoding types. These different spatial parametric encoding types may include, for example, MPEG parametric stereo parameterization, binaural cue coding, spatial audio reconstruction (SPAR), object parameterization in JOC or A-JOC (e.g., object parameterization in A-JOC for Dolby AC-4), or Dolby AC-4 Advanced Coupling (A-CPL) parameterization. Thus, at least two of the first parametrically encoded input audio signal, the second parametrically encoded input audio signal, and the parametrically encoded output audio signal may use different methods, such as MPEG parametric stereo parameterization, binaural cue coding, SPAR (or similar encoding types), object parameterization in JOC or A-JOC, or A-CPL parameterization.

[0078] The first parametrically encoded input audio signal and the second parametrically encoded input audio signal may use different spatial parametric encoding types. The first parametrically encoded input audio signal and the second parametrically encoded input audio signal may use a spatial parametric encoding type that is different from the spatial parametric encoding type used by the parametrically encoded output audio signal. The spatial parametric encoding type may be selected from, for example, MPEG parametric stereo parameterization, binaural cue coding, object parameterization in SPAR, JOC or A-JOC, or Dolby AC-4 Advanced Coupling (A-CPL) parameterization.

[0079] Thus, using systems and methods according to one or more embodiments of the present invention, code conversion can be performed between one spatial parametric coding method and another spatial parametric coding method without requiring complete decoding and recoding of the output signal.

[0080] The step of synthesizing (e.g., mixing) the core audio signal or core audio stream may depend on the design and representation of the audio in the audio codec used. The step of synthesizing (e.g., mixing) the core audio signal or core audio stream is sufficiently independent of the step of synthesizing the covariance matrix, as described herein. Therefore, the processing of parametrically encoded audio based on the determination of the covariance matrix / matrix according to embodiments of the present invention can, in principle, be used with substantially any audio codec based, for example, on covariance estimation (encoder) and reconstruction (decoder).

[0081] A commonly used core codec and its signal synthesis is a transform-based codec. A transform-based codec may represent audio frames in a transformed domain using the modified discrete cosine transform (MDCT) before quantizing the MDCT coefficients. A well-known MDCT-based audio codec is MPEG-1 Layer 3, or MP3 for short (see "ISO / IEC 11172-3:1993 - Information technology -- Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbit / s -- Part 3: Audio"; all of the contents of that document are incorporated herein by reference for all purposes). The MDCT, as a linear process, transforms audio input frames into MDCT coefficients; therefore, the MDCT of the sum of audio signals is equal to the sum of MDCT transforms. For such a transform-based codec, the MDCT representations of input streams can be synthesized (e.g., summed): • Decode the core input bitstream and reconstruct the MDCT transform for each input. • Calculate the sum of MDCT transformations across the input streams (assuming the same transformation size and window shape were used for all input streams). • Re-encode the sum of MDCT transforms (e.g., quantize the magnitude of the MDCT based on the estimated masking curve).

[0082] In practice, it may be necessary to determine the masking curves for the sum of MDCT transforms. One method involves the step of calculating the sum of masking curves in the power domain of each input stream.

[0083] In the embodiment of the present invention illustrated in Figure 3, two input bitstreams (a first input bitstream 10 and a second input bitstream 60) are received and processed, but it should be understood that more than two input bitstreams may be received and processed (in principle, any number of input bitstreams). If more than two input bitstreams may be received and processed, the processing of each input bitstream other than the first input bitstream 10 and the second input bitstream 60 may be carried out in the same or similar manner as the processing of the first input bitstream 10 and the second input bitstream 60 described above with reference to Figure 3. Thus, for each input bitstream other than the first input bitstream 10 and the second input bitstream 60, the input core audio signal and covariance matrix may be determined in the same or similar manner as the first input core audio signal 21 and the second input core audio signal 71 and the first covariance matrix 31 and the second covariance matrix 81 for the first input bitstream 10 and the second input bitstream 60, respectively, and more than three covariance matrices may be obtained. As illustrated in Figure 3 for the first input bitstream 10 and the second input bitstream 60, each input bitstream may be processed individually. Each or any of the input bitstreams may include, or be composed of, a core audio stream, such as an audio signal encoded by a core encoder.

[0084] If two or more input bitstreams are received and processed, the step of determining the output covariance matrix 92 may involve truncating or abstracting one or more covariance matrices with relatively low energy, while the output covariance matrix 92 may be determined based on the remaining covariance matrices. Such truncation or abstraction may be useful, for example, if one (or more) of the input bitstreams has one or more silent frames, or substantially silent frames. For example, the sum of the diagonal elements for each covariance matrix may be determined, and the covariance matrix with the smallest sum of diagonal elements (which may result in the covariance matrix having the smallest energy across all inputs) may be abstracted, and the output covariance matrix 92 may be determined based on the remaining covariance matrices (for example, by calculating the sum of the remaining covariance matrices as described above).

[0085] According to one or more embodiments of the present invention, as with the above, an input bitstream that is mono-only and has no spatial parameters may be received as a possible addition to the processing of parametrically encoded audio illustrated in Figure 1. Thus, in addition to the processing of parametrically encoded audio illustrated in Figure 3 (or Figure 4), a further (such as a third) input bitstream for the mono audio signal may be received (such further or third input bitstream for the mono audio signal is not illustrated in Figure 3). The further input bitstream may contain data representing the mono audio signal. A third covariance matrix may be determined based on the mono audio signal and a matrix containing desired spatial parameters for the third input bitstream (therefore, the third input bitstream is mono-only). A composite core audio signal may be determined based on the first input core audio signal, the second input core audio signal, and the mono audio signal. Based on the determined first covariance matrix, the determined second covariance matrix, and the determined third covariance matrix, a composite covariance matrix can be determined (for example, by calculating the sum of the first, second, and third covariance matrices). Based on the determined composite covariance matrix, a modified set can be determined, where the modified set is different from the first and second sets. Based on the composite core audio signal, an output core audio signal can be determined. For example, the third covariance matrix is ​​the energy of the mono audio signal (if the mono audio signal is represented by matrix Y, then the energy is YY). * It is given by, * The desired spatial parameters for the third input bitstream may be determined based on a matrix containing the conjugate transpose and the desired spatial parameters for the third input bitstream. The desired spatial parameters for the third input bitstream may include, for example, one or more of the amplitude panning parameters or head transfer function parameters (for a mono object associated with a mono audio signal).

[0086] Although the elements of system 300 have been described above as separate components, it should be understood that system 300 may comprise one or more processors that can be configured to implement the above functions of the demultiplexers 20 and 70, covariance matrix determination units 30 and 80, combiner 90, spatial parameter determination unit 110, and output bitstream generation unit 120. Each or any of these functions may be implemented by, for example, one or more processors. For example, one (e.g., a single) processor may implement the above functions of the demultiplexers 20 and 70, covariance matrix determination units 30 and 80, combiner 90, spatial parameter determination unit 110, and output bitstream generation unit 120, or each of the above functions of the demultiplexers 20 and 70, covariance matrix determination units 30 and 80, combiner 90, spatial parameter determination unit 110, and output bitstream generation unit 120 may be implemented by separate processors.

[0087] Figure 4 is a schematic diagram of a system 400 according to another embodiment of the present invention. The system 400 may comprise one or more processors and a non-temporary computer-readable medium storing instructions configured to cause the one or more processors to perform a method according to one embodiment of the present invention when executed by the one or more processors. The system 400 illustrated in Figure 4 is similar to the system 300 illustrated in Figure 3. The same reference numerals in Figures 3 and 4 indicate the same or similar elements having the same or similar functions. The following description of the embodiment of the present invention illustrated in Figure 4 will mainly describe the differences from the embodiment of the present invention illustrated in Figure 3. Therefore, features common to both embodiments may be omitted in the following description. Thus, features of the embodiment of the present invention illustrated in Figure 3 should be considered to be implemented, or at least implementable, in the embodiment of the present invention illustrated in Figure 4, unless otherwise specifically stated in the following description.

[0088] In the embodiment of the present invention illustrated in Figure 4, presentation transformation is integrated into the processing of parametrically encoded audio, similar to the examples and descriptions with reference to Figure 2. In the embodiment of the present invention illustrated in Figure 4, presentation transformation is integrated into the processing of parametrically encoded audio for the first input bitstream 10 and the second input bitstream 60, respectively.

[0089] In contrast to the system 300 illustrated in Figure 3, in the system 400 illustrated in Figure 4, before the step of determining the output covariance matrix 92, the determined first covariance matrix 31 is modified based on output bitstream presentation transformation data (e.g., output bitstream presentation transformation data of the first input bitstream 10). This output bitstream presentation transformation data may include a set of signals intended for playback on a selected audio playback system. Furthermore, also before the step of determining the output covariance matrix 92, the determined second covariance matrix 81 is modified based on output bitstream presentation transformation data (e.g., output bitstream presentation transformation data of the second input bitstream 60). This output bitstream presentation transformation data may include a set of signals intended for playback on a selected audio playback system. It should be understood that one of the modifications to the determined second covariance matrices 31 and 81 may be omitted, and in some cases, only one of the determined second covariance matrices 31 and 81 may be modified based on the output bitstream presentation transformation data, while the other of the determined second covariance matrices 31 and 81 may not be based on the output bitstream presentation transformation data.

[0090] The system 400 may include a covariance matrix modification unit 140 which can be configured to modify a determined first covariance matrix 31 based on output bitstream presentation transformation data 142 of a first input bitstream 10, and / or a covariance matrix modification unit 150 which can be configured to modify a determined second covariance matrix 81 based on output bitstream presentation transformation data 152 of a first input bitstream 60. As illustrated in Figure 4, the covariance matrix modification unit 140 may receive as input (1) output bitstream presentation transformation data 142 of a first input bitstream 10, and (2) the first covariance matrix 31 after it has been output from the covariance matrix determination unit 30, and may output a modified first covariance matrix 141 (compared to the first covariance matrix 31 before it was output from the covariance matrix determination unit 30 and modified in the covariance matrix modification unit 140). As further illustrated in Figure 4, the covariance matrix modification unit 150, as illustrated in Figure 4, receives as input (1) output bitstream presentation transformation data 152 of the second input bitstream 60, and (2) the second covariance matrix 81 after it has been output from the covariance matrix determination unit 80, and can output a modified first covariance matrix 151 (compared to the first covariance matrix 81 before it was output from the covariance matrix determination unit 80 and modified in the covariance matrix modification unit 150).

[0091] In contrast to the system 300 illustrated in Figure 3, in the system 400 illustrated in Figure 4, the synthesizer unit 90 may be configured to determine the output covariance matrix 92 based on the determined first covariance matrix 31 and the determined second covariance matrix 81 (i.e., the modified first covariance matrix 141 and the modified first covariance matrix 151, respectively) which have been modified in the covariance matrix modification unit 140 and the covariance matrix modification unit 150, respectively.

[0092] The output bitstream presentation conversion data may include at least one of the following: downmixing conversion data for downmixing the first input bitstream 10, downmixing conversion data for downmixing the second input bitstream 60, remixing conversion data for remixing the first input bitstream 10, remixing conversion data for remixing the second input bitstream 60, headphone conversion data for converting the first input bitstream 10, or headphone conversion data for converting the second input bitstream 60. The headphone conversion data for converting the first input bitstream 10 and / or the second input bitstream 60 may include a set of signals intended for playback on headphones. For example, the output bitstream presentation conversion data 142 may include at least one of the following: downmixing conversion data for downmixing the first input bitstream 10, remixing conversion data for remixing the first input bitstream 10, or headphone conversion data for converting the first input bitstream 10, and the output bitstream presentation conversion data 152 may include at least one of the following: downmixing conversion data for downmixing the second input bitstream 60, remixing conversion data for remixing the second input bitstream 60, or headphone conversion data for converting the second input bitstream 60.

[0093] As described above with reference to Figure 3, the determination of the first covariance matrix 31 may include determining at least some or all of the diagonal elements and off-diagonal elements of the first covariance matrix 31, and the determination of the second covariance matrix 81 may include determining at least some or all of the diagonal elements and off-diagonal elements of the second covariance matrix 81.

[0094] For example, when integrating presentation transformation into the processing of parametrically encoded audio for a first input bitstream 10 and a second input bitstream 60, as illustrated in Figure 4, it may be useful to consider not only the diagonal elements but also the off-diagonal elements of the covariance matrix. Consider the case where the input bitstreams (e.g., the first input bitstream 10 and the second input bitstream 60) may represent one or more spatial objects that exist in two or more channels (e.g., as a result of amplitude panning, binaural rendering, etc.). This can result in a large number of off-diagonal elements in the covariance matrix (e.g., the first covariance matrix 31 and the second covariance matrix 81) that are important to consider in the processing of parametrically encoded audio for the input bitstream, either to facilitate or to ensure that the playback of the presentation after processing the parametrically encoded audio (e.g., mixing) has an accurate covariance structure. To illustrate the usefulness of considering not only the diagonal elements but also the off-diagonal elements of the covariance matrix, the above case can be compared, for example, to a case where individual objects (streams) that can each represent individual speakers by mono signals are mixed. In that case, it is reasonable to assume that the streams are uncorrelated with respect to each other, and as a result, there is no (off-diagonal) covariance structure that needs to be considered for the mixed streams.

[0095] Finally, a method is disclosed comprising the step of receiving a first input bitstream for a first parametrically encoded input audio signal. The first input bitstream includes data representing a first input core audio signal and a first set of spatial parameters relating to the first parametrically encoded input audio signal. A first covariance matrix of the first parametrically encoded audio signal is determined based on the spatial parameters of the first set. Based on the determined first covariance matrix, a modified set is determined, which includes at least one spatial parameter, and is different from the first set. An output core audio signal is determined, which is based on or composed of the first input core audio signal. An output bitstream is generated for a parametrically encoded output audio signal. The output bitstream includes the output core audio signal and data representing the modified set. Also disclosed is a system comprising one or more processors and a non-temporary computer-readable medium storing instructions configured to cause the one or more processors to perform the method when executed by the one or more processors. Furthermore, the present invention discloses a non-temporary computer-readable medium that stores instructions configured to cause one or more processors to perform the method when executed by one or more processors.

[0096] One or more of the modules, components, blocks, processes, or other functional components described herein may be implemented via a computer program that controls the execution of the processor-based computing device of the system. It should also be noted that various functions disclosed herein may be described with respect to their behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media in which such data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-temporary), non-volatile storage media, such as optical, magnetic, or semiconductor-based storage media.

[0097] While one or more implementation examples have been described and with respect to specific embodiments, it should be understood that the one or more implementation examples are not limited to the disclosed embodiments. Conversely, as will be obvious to those skilled in the art, it is intended to cover a variety of modifications and similar configurations. Therefore, the appended claims should be given the broadest possible interpretation to encompass all such modifications and similar configurations.

[0098] List of enumerated example embodiments (EEE)

[0099] EEE1. Steps include receiving a first input bitstream for a first parametrically encoded input audio signal, wherein the first input bitstream is data representing a first input core audio signal and a first set of spatial parameters relating to the first parametrically encoded input audio signal, The steps of determining a first covariance matrix of the first parametrically encoded audio signal based on the spatial parameters of the first set, A step of determining a modified set of spatial parameters, which includes at least one spatial parameter, based on the first covariance matrix determined, wherein the modified set is different from the first set. The steps include determining an output core audio signal based on or composed of the first input core audio signal, A step of generating an output bitstream for a parametrically encoded output audio signal, wherein the output bitstream includes the output core audio signal and data representing the modified set, A method that includes this.

[0100] Eez2. A method according to EEE1, further comprising the step of determining a determined first covariance matrix based on output bitstream presentation transform data of the first input bitstream, prior to the step of determining the modified set, wherein the output bitstream presentation transform data includes a set of signals intended for playback on a selected audio playback system.

[0101] EEE3. A method relating to EEE2, wherein the output bitstream presentation conversion data includes at least one of downmixing conversion data for downmixing the first input bitstream, remixing conversion data for remixing the first input bitstream, or headphone conversion data for converting the first input bitstream, the headphone conversion data includes a set of signals intended for playback on headphones.

[0102] EEE4. A method relating to one of EEE1 to 3, wherein the first parametrically encoded input audio signal and the parametrically encoded output audio signal use different spatial parameterization encoding types.

[0103] EEE5. The aforementioned different spatial parametric coding types include methods relating to EEE4, such as MPEG parametric stereo parameterization, binaural cue coding, spatial audio reconstruction (SPAR), object parameterization in Joint Object Coding (JOC) or Advanced JOC (A-JOC), or Dolby AC-4 Advanced Coupling (A-CPL) parameterization.

[0104] EEE6. A method relating to any one of EEE1 to 5, wherein the step of determining the first covariance matrix includes determining at least some of the diagonal elements and off-diagonal elements of the first covariance matrix.

[0105] EEE7. A method relating to one of EEE1 to 6, wherein the first parametrically encoded input audio signal represents sound captured from at least two different microphones.

[0106] Eee8. The step of determining the first covariance matrix of the first parametrically encoded audio signal based on the first set of spatial parameters is: The steps include determining a downmix signal of the first parametrically encoded audio signal, The steps include determining the covariance matrix of the downmix signal, The steps include determining the first covariance matrix based on the covariance matrix of the downmix signal and the spatial parameters of the first set, including, A method relating to one of EEE1 through EEE7.

[0107] EEE9. Steps include receiving a second input bitstream for a second parametrically encoded input audio signal, wherein the second input bitstream includes data representing a second input core audio signal and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal, The steps include determining a second covariance matrix of the second parametrically encoded input audio signal based on the spatial parameters of the second set, A step of determining a composite core audio signal based on the first input core audio signal and the second input core audio signal, The steps include determining the output covariance matrix based on the first covariance matrix and the second covariance matrix determined above, A step of determining the modified set based on the determined output covariance matrix, wherein the modified set is different from the first set and the second set, The steps include determining the output core audio signal based on the composite core audio signal, A method relating to any one of EEE1 to 8, further including the above.

[0108] EEE10. The step of determining the output covariance matrix is: A step of calculating the sum of the determined first covariance matrix and the determined second covariance matrix, wherein the sum of the first covariance matrix and the second covariance matrix constitutes the output covariance matrix, or The step of determining the output covariance matrix as the one with the larger sum of the diagonal elements of the first covariance matrix and the second covariance matrix that has been determined, including, Methods related to EEE9.

[0109] EEE11. Prior to the step of determining the output covariance matrix, the steps of modifying the determined first covariance matrix based on the output bitstream presentation transformation data, and / or Before the step of determining the output covariance matrix, there is a step of modifying the determined second covariance matrix based on the output bitstream presentation transformation data. It further includes, The output bitstream presentation conversion data includes a set of signals intended for playback on a selected audio playback system. Methods related to EE9 or 10.

[0110] EEE12. The method according to EEE11, wherein the output bitstream presentation conversion data includes at least one of the following: downmixing conversion data for downmixing the first input bitstream, downmixing conversion data for downmixing the second input bitstream, remixing conversion data for remixing the first input bitstream, remixing conversion data for remixing the second input bitstream, headphone conversion data for converting the first input bitstream, or headphone conversion data for converting the second input bitstream, wherein the headphone conversion data includes a set of signals intended for playback headphones.

[0111] EEE13. A method relating to any one of EEE9 to 12, wherein at least two of the first parametrically encoded input audio signal, the second parametrically encoded input audio signal, and the parametrically encoded output audio signal use different spatial parametric encoding types.

[0112] EEE14. The method relating to EEE13, wherein the different spatial parametric coding types include at least two of MPEG parametric stereo parameterization, binaural cue coding, spatial audio reconstruction (SPAR), object parameterization in Joint Object Coding (JOC) or Advanced JOC (A-JOC), or Dolby AC-4 Advanced Coupling (A-CPL) parameterization.

[0113] EEE15. A method relating to one of EEE9 to 12, wherein the first parametrically encoded input audio signal and the second parametrically encoded input audio signal use different spatial parametric encoding types.

[0114] EEE16. A method according to any one of EEE9 to 12, wherein the first parametrically encoded input audio signal and the second parametrically encoded input audio signal use a spatial parametric encoding type different from the spatial parametric encoding type used by the parametrically encoded output audio signal.

[0115] EEE17. A method relating to any one of EEE9 to 16, wherein at least one of the first parametrically encoded input audio signal and the second parametrically encoded input audio signal represents sound captured from at least two different microphones.

[0116] EEE18. A step of receiving a second input bitstream for a mono audio signal, wherein the second input bitstream includes data representing the mono audio signal, The steps include determining a second covariance matrix based on the mono audio signal and a matrix containing desired spatial parameters for the second input bitstream, A step of determining a composite core audio signal based on the first input core audio signal and the mono audio signal, The steps include determining a composite covariance matrix based on the first covariance matrix and the second covariance matrix determined above, A step of determining the modified set based on the determined composite covariance matrix, wherein the modified set is different from the first set, The steps include determining the output core audio signal based on the composite core audio signal, A method relating to any one of EEE1 to 8, further including the above.

[0117] EEE19. One or more processors, A non-temporary computer-readable medium storing instructions configured to cause one or more processors to perform any one of the methods related to EEE1 to 18 during execution by the aforementioned one or more processors, A system equipped with these features.

[0118] EEE20. A non-temporary computer-readable medium storing instructions configured to cause one or more processors to perform one of the methods relating to EEE1 to 18 when executed by one or more processors.

Claims

1. Steps of receiving a first input bitstream for a first parametrically encoded input audio signal, wherein the first input bitstream is data representing a first input core audio signal and a first set of spatial parameters relating to the first parametrically encoded input audio signal, The steps include determining a first covariance matrix of the first parametrically encoded audio signal based on the spatial parameters of the first set, A step of receiving a second input bitstream for a second parametrically encoded input audio signal, wherein the second input bitstream includes data representing a second input core audio signal and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal. The steps include determining a second covariance matrix of the second parametrically encoded input audio signal based on the spatial parameters of the second set, A step of determining a composite core audio signal based on the first input core audio signal and the second input core audio signal, The steps include determining the output covariance matrix based on the first covariance matrix and the second covariance matrix determined above, A step of determining a modified set based on the determined output covariance matrix, wherein the modified set is different from the first set and the second set. A step of generating an output bitstream for a parametrically encoded output audio signal, wherein the output bitstream includes the synthesized core audio signal and data representing the modified set, A method that includes this.

2. One or more processors, When executed by the one or more processors, the one or more processors Steps of receiving a first input bitstream for a first parametrically encoded input audio signal, wherein the first input bitstream is data representing a first input core audio signal and a first set of spatial parameters relating to the first parametrically encoded input audio signal, The steps include determining a first covariance matrix of the first parametrically encoded audio signal based on the spatial parameters of the first set, A step of receiving a second input bitstream for a second parametrically encoded input audio signal, wherein the second input bitstream includes data representing a second input core audio signal and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal. The steps include determining a second covariance matrix of the second parametrically encoded input audio signal based on the spatial parameters of the second set, A step of determining a composite core audio signal based on the first input core audio signal and the second input core audio signal, The steps include determining the output covariance matrix based on the first covariance matrix and the second covariance matrix determined above, A step of determining a modified set based on the determined output covariance matrix, wherein the modified set is different from the first set and the second set. A step of generating an output bitstream for a parametrically encoded output audio signal, wherein the output bitstream includes the synthesized core audio signal and data representing the modified set, A non-temporary computer-readable medium storing instructions configured to execute, A system equipped with these features.

3. When executed by one or more processors, the one or more processors Steps of receiving a first input bitstream for a first parametrically encoded input audio signal, wherein the first input bitstream is data representing a first input core audio signal and a first set of spatial parameters relating to the first parametrically encoded input audio signal, The steps include determining a first covariance matrix of the first parametrically encoded audio signal based on the spatial parameters of the first set, A step of receiving a second input bitstream for a second parametrically encoded input audio signal, wherein the second input bitstream includes data representing a second input core audio signal and a second set of data including at least one spatial parameter relating to the second parametrically encoded input audio signal. The steps include determining a second covariance matrix of the second parametrically encoded input audio signal based on the spatial parameters of the second set, A step of determining a composite core audio signal based on the first input core audio signal and the second input core audio signal, The steps include determining the output covariance matrix based on the first covariance matrix and the second covariance matrix determined above, A step of determining a modified set based on the determined output covariance matrix, wherein the modified set is different from the first set and the second set. A step of generating an output bitstream for a parametrically encoded output audio signal, wherein the output bitstream includes the synthesized core audio signal and data representing the modified set, A computer program containing instructions configured to perform a method that includes the following.

Images