Parameter encoding and decoding

Abstract

Several examples of encoding and decoding techniques are disclosed. In particular, an audio synthesizer (300) for generating a synthesized signal (336, 340, yR) from a downmix signal (246, x), comprising: an input interface (312) for receiving the downmix signal (246, x), the downmix signal (246, x) having a plurality of downmix channels and side information (228), the side information (228) comprising channel levels and related information (314, xi, chi) of an original signal (212, y), the original signal (212, y) having a plurality of original channels; and a synthesis processor (404) for generating a synthesized signal (336, 340, yR) according to at least one mixing rule using: a channel level of the original signal (212, y) and related information (220, 314, zeta, chi); and covariance information (Cx) associated with the downmix signal (324, 246, x).

Description

[0001] This application is a divisional application whose applicant is Fraunhofer Gesellschaft, whose application date is June 15, 2020, whose application number is: 202080057545.X, and whose invention name is “Parameter Coding and Decoding”. Technical Field

[0002] 1. Introduction

[0003] Herein, several examples of encoding and decoding techniques are disclosed. In particular, an invention is directed to encoding and decoding multi-channel audio content at a low bit rate, for example using the DirAC framework. This approach can achieve high-quality output while using a low bit rate. This can be used for many applications, including artwork, communications, and virtual reality. Background Art

[0004] 1.1 Prior art

[0005] This section briefly describes the prior art.

[0006] 1.1.1 Discrete Coding of Multichannel Content

[0007] The most direct way to encode and transmit multi-channel content is to directly quantize and encode the waveform of the multi-channel audio signal without any prior processing or assumptions. Although said method can work perfectly in theory, there is a major drawback, namely the bit consumption required to encode the multi-channel content. Therefore, other methods to be described (and the proposed invention) are so-called "parametric methods" because they use meta-parameters to describe and transmit the multi-channel audio signal instead of the original audio multi-channel signal itself.

[0008] 1.1.2MPEG Surround

[0009] MPEG Surround is an ISO / MPEG standard for parametric coding of multi-channel audio, finalized in 2006.[1] The method relies primarily on two parameter sets:

[0010] - Interchannel coherences (ICC), which describes the coherence between each channel of a given multi-channel audio signal.

[0011] - Channel Level Difference (CLD), corresponding to the level difference between two input channels of a multi-channel audio signal.

[0012] One particularity of MPEG Surround is the use of so-called "tree structures" which allow "two input channels to be described by a single output channel" (cited from [1]).

[0013] As an example, an encoder scheme for a 5.1 multi-channel audio signal using MPEG Surround can be found below. In this figure, the six input channels (marked as "L", "L S ”, “R”, “R S , "C" and "LFE") are processed sequentially through tree structure elements (labeled "R_OTT" in the figure). Each of these tree structure elements will produce a parameter set such as the previously mentioned ICC and CLD and a residual signal, which will be processed again through another tree structure and produce another set of parameters. Once the end of the tree is reached, the different parameters previously calculated are transmitted to the decoder, like the downmix signal. These elements are used by the decoder to produce an output multi-channel signal, and the decoder processing is basically the inverse tree structure used by the encoder.

[0014] The main advantage of MPEG Surround is the use of this structure and the parameters mentioned above. However, one of the disadvantages of MPEG Surround is the lack of flexibility due to the tree structure. Also due to the peculiarities of the processing, quality degradation may occur on certain projects.

[0015] See Fig. 7, among others, which shows an overview of an MPEG surround encoder for 5.1 signals extracted from [1].

[0016] 1.2 Directional Audio Coding

[0017] Directional Audio Coding (DirAC) [2] is also a parametric method for reproducing spatial audio, developed by Ville Pulkki at Aalto University in Finland. DirAC relies on frequency band processing that uses two sets of parameters to describe spatial sound:

[0018] - Direction of Arrival (DOA), which is an angle, measured in degrees, describing the direction of arrival of the predominant sound in the audio signal.

[0019] - Diffusivity, a value between 0 and 1 describing how "diffuse" the sound is. If the value is 0, the sound is non-diffuse and can be assimilated to a point source coming from a precise angle; if the value is 1, the sound is completely diffuse and is assumed to come from "every" angle.

[0020] To synthesize the output signal, DirAC assumes that it is decomposed into diffuse and non-diffuse parts, the diffuse sound synthesis aiming to produce the perception of an ambient sound, while the direct sound synthesis aims to produce the dominant sound.

[0021] Given that DirAC provides high-quality output, it has one major drawback: it is not suitable for multi-channel audio signals. Therefore, the DOA and diffuseness parameters are not well suited to describe multi-channel audio input, and therefore, the output quality suffers.

[0022] 1.3 Binaural cue encoding

[0023] Binaural Cue Coding (BCC) [3] is a parameterized method developed by Christof Faller. This method relies on a similar set of parameters as those described for MPEG Surround (see 1.1.2), namely:

[0024] - Inter-channel Level Difference (ICLD), which is a measure of the energy ratio between two channels of a multi-channel input signal.

[0025] - Inter-Channel Time Difference (ICTD), which is a measure of the delay between two channels of a multi-channel input signal.

[0026] - Inter-channel correlation (ICC), which is a measure of the correlation between two channels of a multi-channel input signal.

[0027] Compared to the novel invention to be described later, the BCC method has very similar characteristics as far as the calculation of the transmitted parameters is concerned, but it lacks the flexibility and scalability of the transmitted parameters.

[0028] 1.4 MPEG Spatial Audio Object Coding

[0029] Spatial Audio Object Coding[4] will be briefly mentioned here. This is an MPEG standard for encoding so-called audio objects, which are somewhat related to multi-channel signals. It uses similar parameters as MPEG Surround. Summary of the invention

[0030] 1.5 Incentives / Disadvantages of Existing Technologies

[0031] 1.5.1 Causes

[0032] 1.5.1.1 Using the DirAC framework

[0033] One aspect of the present invention that must be mentioned is that the current invention must be adapted to the DirAC framework. Nevertheless, it has been mentioned before that the parameters of DirAC are not applicable to multi-channel audio signals. More explanation should be given on this topic.

[0034] The original DirAC process uses microphone signals or ambisonics signals. From these signals, parameters are calculated, namely direction of arrival (DOA) and diffuseness.

[0035] The first approach that was tried to use DirAC with multi-channel audio signals was to convert the multi-channel signals into ambiguous content using a method proposed by Ville Pulkki, as described in [5]. Then, once these ambiguous signals were derived from the multi-channel audio signal, conventional DirAC processing was performed using DOA and diffuseness. The result of this first attempt was that the quality and spatial characteristics of the output multi-channel signal deteriorated and did not meet the requirements of the target application.

[0036] Therefore, the driving force behind the novel invention is to use parameter sets that efficiently describe the multi-channel signal and also use the DirAC framework, further explanation will be given in Section 1.1.2.

[0037] 1.5.1.2 Providing Systems Operating at Low Bit Rates

[0038] One of the purposes and objectives of the present invention is to propose a method that allows low bit rate applications. This requires finding the best data set to describe the multi-channel content between the encoder and the decoder. This also requires finding the best trade-off in terms of the number of transmission parameters and output quality.

[0039] 1.5.1.3 Provide a flexible system

[0040] Another important object of the invention is to propose a flexible system which can accept any multi-channel audio format intended to be reproduced on any loudspeaker setup.Depending on the input setup, the output quality should not be compromised.

[0041] 1.5.2 Disadvantages of Existing Technologies

[0042] Several disadvantages of the prior art mentioned above are listed in the following table.

[0043]

[0044] 2. Description of the invention 2.1 SUMMARY OF THE INVENTION

[0046] According to one aspect, an audio synthesizer (encoder) is provided for generating a synthesized signal from a downmix signal, the synthesized signal having a plurality of synthesized channels, the audio synthesizer comprising:

[0047] an input interface configured to receive the downmix signal, the downmix signal having a plurality of downmix channels and side information, the side information comprising channel levels and related information of an original signal, the original signal having a plurality of original channels; and

[0048] A synthesis processor is configured to generate the synthesized signal according to at least one mixing rule using:

[0049] Channel levels and related information of the original signal; and

[0050] Covariance information associated with the downmix signal.

[0051] The audio synthesizer may include:

[0052] a prototype signal calculator configured to calculate a prototype signal from the downmix signal, the prototype signal having the plurality of synthesized channels;

[0053] A blending rule calculator configured to calculate at least one blending rule using:

[0054] The channel levels and related information of the original signal; and

[0055] the covariance information associated with the downmix signal;

[0056] The synthesis processor is configured to generate the synthesized signal using the prototype signal and the at least one mixing rule.

[0057] The audio synthesizer may be configured to reconstruct target covariance information of the original signal.

[0058] The audio synthesizer may be configured to reconstruct the target covariance information adapted to the number of channels of the synthesized signal.

[0059] The audio synthesizer may be configured to reconstruct the covariance information adapted to the number of channels of the synthesized signal by assigning an original channel group to a single synthesized channel, or vice versa, such that reconstructed target covariance information is reported to the plurality of channels of the synthesized signal.

[0060] The audio synthesizer may be configured to reconstruct the covariance information adapted to the number of channels of the synthesized signal by generating target covariance information for the number of original channels and then applying a downmix rule or an upmix rule and energy compensation to derive the target covariance for the synthesized channels.

[0061] The audio synthesizer may be configured to reconstruct a target version of the covariance information based on an estimated version of the original covariance information, wherein the estimated version of the original covariance information is reported to the plurality of synthesized channels or the plurality of original channels.

[0062] The audio synthesizer may be configured to obtain the estimated version of the original covariance information from covariance information associated with the downmix signal.

[0063] The audio synthesizer may be configured to obtain the estimated version of the original covariance information by applying an estimation rule to the covariance information associated with the downmix signal, the estimation rule being a prototype rule for calculating the prototype signal associated with a prototype rule for calculating the prototype signal.

[0064] The audio synthesizer may be configured to convert the original covariance information (C y ) Normalized to the square root of the level of the channel in the pair.

[0065] The audio synthesizer may be configured to construct a matrix using a normalized estimated version of the original covariance information.

[0066] The audio synthesizer may be configured to complete the matrix by inserting entries obtained in the side information of the bitstream.

[0067] The audio synthesizer may be configured to de-normalize the matrix by scaling the estimated version of the original covariance information by the square root of the levels of the channels forming the channel pair.

[0068] The audio synthesizer may be configured to retrieve channel levels and related information among the side information of the downmix signal, the audio synthesizer being further configured to reconstruct the target version of the covariance information by estimated versions of the original channel levels and related information from:

[0069] covariance information for at least one first channel or channel pair; and

[0070] Channel levels and related information for at least one second channel or channel pair.

[0071] The audio synthesizer may be configured to prefer the channel levels and related information describing the channel or channel pair obtained from the side information of the bitstream over the covariance information reconstructed from the downmix signal for the same channel or channel pair.

[0072] The reconstructed target version of the original covariance information may be understood to describe an energy relationship between a pair of channels, or to be based at least in part on levels associated with each channel of the pair of channels.

[0073] The audio synthesizer may be configured to obtain a frequency domain FD version of the downmix signal, the frequency domain version of the downmix signal being divided into frequency bands or frequency band groups, wherein different channel levels and related information are associated with different frequency bands or frequency band groups,

[0074] The audio synthesizer is configured to operate differently for different frequency bands or frequency band groups to obtain different mixing rules for different frequency bands or frequency band groups.

[0075] The downmix signal is divided into time slots, wherein different channel levels and related information are associated with different time slots, and the audio synthesizer is configured to operate differently for different time slots to obtain different mixing rules for different time slots.

[0076] The downmix signal is divided into frames and each frame is divided into time slots, wherein when the presence and position of a transient in a frame is signaled as being in a transient time slot, the audio synthesizer is configured to:

[0077] associating current channel levels and related information with the transient time slot and / or a time slot subsequent to the transient time slot of the frame; and

[0078] A time slot preceding the transient time slot of the frame is associated with the channel levels and related information of the preceding time slot.

[0079] The audio synthesizer may be configured to select a prototype rule configured for calculating a prototype signal on the basis of the plurality of synthesized channels.

[0080] The audio synthesizer may be configured to select the prototype rule among a plurality of pre-stored prototype rules.

[0081] The audio synthesizer may be configured to define prototype rules on a manually selected basis.

[0082] The prototype rule may be based on or include a matrix having a first dimension and a second dimension, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesis channels.

[0083] The audio synthesizer may be configured to operate at a bit rate equal to or lower than 160 kbit / s.

[0084] The audio synthesizer may further comprise an entropy decoder for obtaining the downmix signal with the side information.

[0085] The audio synthesizer also includes a decorrelation module to reduce the amount of correlation between different channels.

[0086] The prototype signal may be provided directly to the synthesis processor without performing decorrelation.

[0087] At least one of the channel levels and related information of the original signal, the at least one mixing rule, and the covariance information associated with the downmix signal is in a matrix form.

[0088] The side information includes an identification of the original channel;

[0089] The audio synthesizer may be further configured to calculate the at least one mixing rule using at least one of the channel levels and related information of the original signal, covariance information associated with the downmix signal, the identification of the original channels, and the identification of the synthesized channels.

[0090] The audio synthesizer may be configured to calculate at least one mixing rule by singular value decomposition (SVD).

[0091] The downmix signal may be divided into frames, the audio synthesizer being configured to smooth the received parameters, estimated or reconstructed values ​​or mixing matrices using a linear combination with parameters, estimated or reconstructed values ​​or mixing matrices obtained for previous frames.

[0092] The audio synthesizer may be configured to deactivate the smoothing of the received parameters, the estimated or reconstructed values ​​or the mixing matrix when the presence and / or position of a transient in a frame is signaled.

[0093] The downmix signal may be divided into frames, and the frames into time slots, wherein the channel levels and related information of the original signal are obtained from side information of a bitstream in a frame-by-frame manner, the audio synthesizer being configured to use a mixing matrix (or mixing rule) for a current frame, the audio synthesizer being configured to use a mixing rule for the current frame, the mixing rule being obtained by scaling the mixing matrix (or mixing rule) calculated for the current frame by a coefficient increased along a subsequent time slot of the current frame, and by adding a mixing matrix (or mixing rule) for a previous frame with a version scaled by a coefficient decreased along the subsequent time slot of the current frame.

[0094] The number of the synthesized channels may be greater than the number of the original channels. The number of the synthesized channels may be less than the number of the original channels. The number of the synthesized channels and the number of the original channels may be greater than the number of the downmixed channels.

[0095] At least one of the number of the synthesized channels, the number of the original channels, and the number of the downmixed channels, or all of the numbers are a plural number.

[0096] The at least one mixing rule may include a first mixing matrix and a second mixing matrix, and the audio synthesizer includes:

[0097] The first path includes:

[0098] A first mixing matrix block is configured to synthesize a first component of the synthesized signal according to the first mixing matrix calculated from:

[0099] a covariance matrix associated with the synthesized signal, the covariance matrix being reconstructed from the channel levels and related information; and

[0100] a covariance matrix associated with the downmix signal,

[0101] The second path is used to synthesize a second component of the synthesized signal, wherein the second component is a residual component.

[0102] The second path includes:

[0103] a prototype signal block configured to upmix the downmix signal from the number of downmix channels to the number of synthesis channels;

[0104] a decorrelator configured to decorrelate the upmixed prototype signal;

[0105] a second mixing matrix block configured to synthesize the second component of the synthesized signal from a decorrelated version of the downmix signal according to a second mixing matrix, the second mixing matrix being a residual mixing matrix,

[0106] wherein the audio synthesizer is configured to estimate the second mixing matrix from:

[0107] a residual covariance matrix provided by said first mixing matrix block; and

[0108] an estimate of the covariance matrix of the decorrelated prototype signal obtained from said covariance matrix associated with said downmix signal,

[0109] The audio synthesizer further comprises an adder block for summing the first component of the synthesized signal and the second component of the synthesized signal.

[0110] According to one aspect, an audio synthesizer is provided for generating a synthesized signal from a downmix signal having a plurality of downmix channels, the synthesized signal having a plurality of synthesized channels, the downmix signal being a downmixed version of an original signal having a plurality of original channels, the audio synthesizer comprising:

[0111] The first path includes:

[0112] A first mixing matrix block is configured to synthesize a first component of the synthesized signal according to a first mixing matrix calculated from:

[0113] a covariance matrix associated with the composite signal; and

[0114] a covariance matrix associated with the downmix signal;

[0115] The second path is used to synthesize a second component of the synthesized signal, wherein the second component is a residual component.

[0116] The second path includes:

[0117] a prototype signal block configured to upmix the downmix signal from the number of downmix channels to the number of synthesis channels;

[0118] a decorrelator configured to decorrelate the upmixed prototype signal (613c);

[0119] a second mixing matrix block configured to synthesize a second component of the synthesized signal from a decorrelated version of the downmix signal according to a second mixing matrix, the second mixing matrix being a residual mixing matrix,

[0120] wherein the audio synthesizer is configured to calculate the second mixing matrix from:

[0121] the residual covariance matrix provided by the first mixing matrix block; and

[0122] an estimate of the covariance matrix of the decorrelated prototype signal obtained from the covariance matrix associated with the downmix signal,

[0123] The audio synthesizer further comprises an adder block for summing the first component of the synthesized signal and the second component of the synthesized signal.

[0124] The residual covariance matrix is ​​obtained by subtracting a matrix obtained by applying the first mixing matrix to the covariance matrix associated with the downmix signal from the covariance matrix associated with the synthesized signal.

[0125] The audio synthesizer may be configured to define the second mixing matrix from:

[0126] a second matrix obtained by decomposing the residual covariance matrix associated with the composite signal;

[0127] A first matrix which is an inverse matrix or a regularized inverse matrix of a diagonal matrix obtained from the estimate of the covariance matrix of the decorrelated prototype signal.

[0128] The diagonal matrix may be obtained by applying the square root function to the main diagonal elements of the covariance matrix of the decorrelated prototype signal.

[0129] The second matrix may be obtained by applying singular value decomposition to the residual covariance matrix associated with the composite signal.

[0130] The audio synthesizer may be configured to define the second mixing matrix by multiplying the second matrix with the inverse matrix of the diagonal matrix obtained from the estimate of the covariance matrix of the decorrelated prototype signal or the regularized inverse matrix and a third matrix.

[0131] The audio synthesizer can be configured to obtain the third matrix by applying singular value decomposition to a matrix obtained from a normalized version of the covariance matrix of the decorrelated prototype signal, wherein the normalization is performed relative to the residual covariance matrix and the main diagonal of the diagonal matrix and the second matrix.

[0132] The audio synthesizer may be configured to define the first mixing matrix from a second matrix and an inverse matrix or a regularized inverse matrix of the second matrix,

[0133] The second matrix is ​​obtained by decomposing the covariance matrix associated with the downmix signal, and the second matrix is ​​obtained by decomposing a reconstructed target covariance matrix associated with the downmix signal.

[0134] The audio synthesizer may be configured to estimate the covariance matrix of the decorrelated prototype signal from the diagonal entries of the matrix, the matrix being obtained by applying a prototype rule used at the prototype block for upmixing the downmix signal from the number of the downmix channels to the number of synthesized channels to the covariance matrix associated with the downmix signal.

[0135] The frequency bands are aggregated with each other into aggregated frequency band groups, wherein information about the aggregated frequency band groups is provided in side information of the bitstream, wherein the channel levels and related information of the original signal are provided per frequency band group to calculate the same at least one mixing matrix for different frequency bands of the same aggregated frequency band group.

[0136] According to one aspect, an audio encoder is provided for generating a downmix signal from an original signal, wherein the original signal has a plurality of original channels, and the downmix signal has a plurality of downmix channels, the audio encoder comprising:

[0137] a parameter estimator configured to estimate channel levels and related information of the original signal, and

[0138] A bitstream writer is used for encoding the downmix signal into a bitstream, so that the downmix signal is encoded in the bitstream to have side information, wherein the side information includes channel levels and related information of the original signal.

[0139] The audio encoder may be configured to provide the channel levels and related information of the original signal as normalized values.

[0140] The channel levels and related information of the original signal encoded in the side information represent at least channel level information associated with a total number of the original channels.

[0141] The channel levels and related information of the original signal encoded in the side information represent at least related information describing an energy relationship between at least one pair of different original channels, but less than a total number of the original channels.

[0142] The channel levels and related information of the original signal include at least one coherence value describing the coherence between two channels in a pair of original channels.

[0143] The coherence value may be normalized. The coherence value may be

[0144]

[0145] in is the covariance between channels i and j, and are the levels associated with channels i and j respectively.

[0146] The channel levels and related information of the original signal include at least one inter-channel level difference (ICLD).

[0147] The at least one ICLD may be provided as a logarithmic value. The at least one ICLD is normalized. The at least one ICLD may be

[0148]

[0149] in

[0150] -x i is the inter-channel level difference for channel i,

[0151] -P i is the power of the current channel i,

[0152] -P dmx,i is a linear combination of the values ​​of the covariance information of the downmix signal.

[0153] The audio encoder may be configured to select whether to encode or not encode at least a portion of the channel levels and related information of the original signal based on state information to include an increased amount of channel levels and related information in the side information when the payload is relatively low.

[0154] The audio encoder may be configured to select which part of the channel levels and related information of the original signal to encode in the side information based on a metric about a channel to include channel levels and related information associated with a more sensitive metric in the side information.

[0155] The channel levels and related information of the original signal may be in the form of entries of a matrix.

[0156] The matrix may be a symmetric matrix or a Hermitian matrix, wherein entries of the channel levels and related information are provided for all or less than the total number of entries in the diagonal of the matrix and / or for less than half of the off-diagonal elements of the matrix.

[0157] The bitstream writer is configured to encode an identification of at least one channel.

[0158] The original signal or a processed version thereof may be divided into a plurality of subsequent frames of equal time length.

[0159] The audio encoder may be configured to encode channel levels and related information of the original signal specific to each frame in the side information.

[0160] The audio encoder may be configured to encode, in the side information, the same channel levels and related information of the original signal commonly associated with a plurality of consecutive frames.

[0161] The audio encoder may be configured to select a number of consecutive frames for which the same channel levels and related information of the original signal are selected such that:

[0162] A relatively higher bit rate or a higher payload implies an increase in the number of consecutive frames to which the same channel levels and related information of the original signal are associated, and vice versa.

[0163] The audio encoder may be configured to reduce the number of consecutive frames of the original signal to which the same channel levels and related information are associated when a transient is detected.

[0164] Each frame may be subdivided into an integer number of consecutive time slots.

[0165] The audio encoder may be configured to estimate the channel levels and related information for each time slot, and encode a sum or an average or another predetermined linear combination of the channel levels and related information estimated for different time slots in the side information.

[0166] The audio encoder may be configured to perform a transient analysis on the time domain version of the frame to determine occurrences of transients within the frame.

[0167] The audio encoder may be configured to determine in which time slot of the frame the transient has occurred, and:

[0168] encoding the channel levels and related information of the original signal associated with a time slot in which the transient has occurred and / or a subsequent time slot in the frame,

[0169] The channel levels and related information of the original signal associated with the time slot preceding the transient are not encoded.

[0170] The audio encoder may be configured to signal in the side information that the occurrence of the transient occurs in a time slot of the frame.

[0171] The audio encoder may be configured to signal in the side information in which time slot of the frame the transient has occurred.

[0172] The audio encoder may be configured to estimate channel levels and related information of the original signal associated with a plurality of time slots of the frame and sum them or average them or linearly combine them to obtain channel levels and related information associated with the frame.

[0173] The original signal may be converted into a frequency domain signal, wherein the audio encoder is configured to encode the channel levels and related information of the original signal in the side information in a frequency band-by-frequency band manner.

[0174] The audio encoder may be configured to aggregate a plurality of frequency bands of the original signal into a further reduced number of frequency bands so as to encode the channel levels and related information of the original signal in the side information in an aggregated frequency band-by-aggregated frequency band manner.

[0175] The audio encoder may be configured to further aggregate the frequency bands in case a transient in the frame is detected, such that:

[0176] The number of frequency bands is reduced; and / or

[0177] The width of at least one frequency band is increased by aggregating with another frequency band.

[0178] The audio encoder may further be configured to encode at least one channel level and related information for a frequency band in the bitstream as a delta with respect to previously encoded channel level and related information.

[0179] The audio encoder may be configured to encode in the side information of the bitstream an incomplete version of the channel levels and related information relative to the channel levels and related information estimated by the estimator.

[0180] The audio encoder may be configured to adaptively select selected information to be encoded in the side information of the bitstream among the overall channel levels and related information estimated by the estimator such that remaining non-selected information of the channel levels and / or related information estimated by the estimator is not encoded.

[0181] The audio encoder may be configured to reconstruct the channel levels and related information from the selected channel levels and related information, thereby simulating estimates of the non-selected channel levels and related information at the decoder, and calculating error information between:

[0182] the non-selected channel levels and related information estimated by the encoder; and

[0183] said non-selected channel levels and related information reconstructed by simulating estimates of unencoded channel levels and related information at said decoder; and

[0184] So that a distinction is made on the basis of the calculated error information:

[0185] Correctly reconstructed channel levels and related information; and

[0186] Channel levels and related information cannot be correctly reconstructed,

[0187] To determine:

[0188] selecting the incorrectly reconstructible channel levels and related information to be encoded in the side information of the bitstream; and

[0189] The correctly reconstructible channel levels and related information are not selected, thereby avoiding encoding the correctly reconstructible channel levels and related information in the side information of the bitstream.

[0190] The channel levels and related information may be indexed according to a predetermined ordering, wherein the encoder is configured to signal an index associated with the predetermined ordering in the side information of the bitstream, the index indicating which of the channel levels and related information is encoded. The index is provided via a bitmap. The index is defined according to a combined numbering system associating a one-dimensional index with an entry of a matrix.

[0191] The audio encoder can be configured to select between:

[0192] adaptive provision of said channel levels and related information, wherein an index associated with said predetermined ordering is encoded in said side information of said bitstream; and

[0193] The fixed provision of the channel levels and related information enables the encoded channel levels and related information to be predetermined and sorted according to a predetermined fixed order without providing an index.

[0194] The audio encoder may be configured to signal in the side information of the bitstream whether the channel levels and related information are provided according to an adaptive provision or according to a fixed provision.

[0195] The audio encoder may be further configured to encode current channel levels and related information in the bitstream as deltas relative to previous channel levels and related information.

[0196] The audio encoder may also be configured to generate the downmix signal according to a static downmix.

[0197] According to one aspect, there is provided a method for generating a synthesis signal from a downmix signal, the synthesis signal having a plurality of synthesis channels, the method comprising:

[0198] A downmix signal and side information are received, wherein the downmix signal has a plurality of downmix channels, and the side information comprises:

[0199] Channel levels and related information of an original signal, the original signal having a plurality of original channels;

[0200] The synthesized signal is generated using channel levels and related information of the original signal and covariance information associated with the signal.

[0201] The method may include:

[0202] calculating a prototype signal from the downmix signal, the prototype signal having the plurality of synthesized channels;

[0203] calculating a mixing rule using channel levels and related information of the original signal and covariance information associated with the downmix signal; and

[0204] The synthesized signal is generated using the prototype signal and the mixing rule.

[0205] According to one aspect, a method for generating a downmix signal from an original signal is provided, wherein the original signal has a plurality of original channels, and the downmix signal has a plurality of downmix channels, the method comprising:

[0206] estimating channel levels and related information of the original signal,

[0207] The downmix signal is encoded into a bitstream such that the downmix signal is encoded in the bitstream to have side information including channel levels and related information of the original signal.

[0208] According to one aspect, a method for generating a synthesized signal from a downmix signal having a plurality of downmix channels, the synthesized signal having a plurality of synthesized channels, the downmix signal being a downmixed version of an original signal having a plurality of original channels, the method comprising the following stages:

[0209] The first phase includes:

[0210] The first component of the composite signal is synthesized according to a first mixing matrix calculated from:

[0211] a covariance matrix associated with the composite signal; and

[0212] a covariance matrix associated with the downmix signal,

[0213] The second stage is used to synthesize the second component of the synthesized signal, wherein the second component is a residual component.

[0214] The second phase includes:

[0215] a prototype signal step of upmixing the downmix signal from the number of downmix channels to the number of synthesis channels;

[0216] a decorrelator step for decorrelating the upmixed prototype signal;

[0217] a second mixing matrix step of synthesizing the second component of the synthesized signal from a decorrelated version of the downmix signal according to a second mixing matrix, the second mixing matrix being a residual mixing matrix,

[0218] The method calculates the second mixing matrix from:

[0219] the residual covariance matrix provided by said first mixing matrix step; and

[0220] an estimate of the covariance matrix of the decorrelated prototype signal obtained from the covariance matrix associated with the downmix signal,

[0221] The method further comprises an adder step of summing the first component of the composite signal and the second component of the composite signal to obtain the composite signal.

[0222] According to one aspect, an audio synthesizer for generating a synthesized signal from a downmix signal is provided, wherein the synthesized signal has a number of synthesized channels, the number of synthesized channels being greater than one or greater than two, the audio synthesizer comprising at least one of the following:

[0223] An input interface is configured to receive the downmix signal, the downmix signal having at least one downmix channel and

[0224] Side information, the side information including at least one of the following:

[0225] Channel levels and related information of an original signal, wherein the original signal has a plurality of original channels, and the number of the original channels is greater than one or greater than two;

[0226] a component, such as a prototype signal calculator [e.g., "prototype signal calculation"], configured to calculate a prototype signal from said downmix signal, said prototype signal having said number of synthesis channels;

[0227] means, such as a mixing rule calculator [e.g. "parameter reconstruction"], configured to calculate one (or more) mixing rules using channel levels and related information of said original signal, covariance information associated with said downmix signal; and

[0228] A component, such as a synthesis processor [eg, a "synthesis engine"], is configured to generate the synthesized signal using the prototype signal and the mixing rule.

[0229] The number of the synthesized channels may be greater than the number of the original channels. Alternatively, the number of the synthesized channels may be less than the number of the original channels.

[0230] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) can be configured to reconstruct a target version of the original channel levels and related information.

[0231] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) may be configured to reconstruct a target version of the original channel levels and related information adapted to the plurality of channels of the synthesized signal.

[0232] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) may be configured to reconstruct a target version of the original channel levels and related information that is based on an estimated version of the original channel levels and related information.

[0233] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) may be configured to derive the estimated version of the original channel levels and related information from covariance information associated with the downmix signal.

[0234] The audio synthesizer (in particular, in some aspects, the mixing rule calculator) can be configured to obtain, for the prototype signal, the estimated version of the original channel levels and related information by applying an estimation rule associated with a prototype rule used by the prototype signal calculator to the covariance information associated with the downmix signal.

[0235] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) may be configured to retrieve from the side information of the downmix signal both:

[0236] covariance information associated with the downmix signal, describing the level of a first channel or energy relationships between pairs of channels in the downmix signal; and

[0237] Channel levels and related information of the original signal, describing the level of the first channel in the original signal or the energy relationship between channel pairs,

[0238] so that the original channel levels and related information are reconstructed by using at least one of the following

[0239] Standard version:

[0240] covariance information of the original channels for at least one first channel or channel pair; and

[0241] Describing the channel levels and related information of the at least one first channel or channel pair.

[0242] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) may be configured to prefer the channel levels and related information describing the channel or channel pair over the covariance information of the original channels for the same channel or channel pair.

[0243] The reconstructed target version of the original channel levels and related information describes an energy relationship between a channel pair based at least in part on a level associated with each channel in the channel pair.

[0244] The downmix signal can be divided into frequency bands or frequency band groups; different channel levels and related information can be associated with different frequency bands or frequency band groups; the audio synthesizer (the prototype signal calculator, in particular, in some aspects, at least one of the mixing rule calculator and the synthesis processor) is configured to operate differently for different frequency bands or frequency band groups to obtain different mixing rules for different frequency bands or frequency band groups.

[0245] The downmix signal may be divided into time slots, wherein different channel levels and related information are associated with different time slots, and at least one component of the audio synthesizer (e.g., the prototype signal calculator, the mixing rule calculator, the synthesis processor or other elements of the synthesizer) is configured to operate differently for different time slots to obtain different mixing rules for different time slots.

[0246] The audio synthesizer (eg the prototype signal calculator) may be configured to select a prototype rule configured for calculating a prototype signal based on the number of synthesized channels.

[0247] The audio synthesizer (eg the prototype signal calculator) may be configured to select the prototype rule among pre-stored prototype rules.

[0248] The audio synthesizer (eg, the prototype signal calculator) may be configured to define prototype rules on a manually selected basis.

[0249] The prototype rule (eg the prototype signal calculator) may include a matrix having a first dimension and a second dimension, wherein the first dimension is associated with the number of the downmix channels and the second dimension is associated with the number of the synthesized channels.

[0250] The audio synthesizer (eg the prototype signal calculator) may be configured to operate at a bit rate equal to or lower than 160 kbit / s.

[0251] The side information may include the identifier of the original channel [eg, L, R, C, etc.].

[0252] The audio synthesizer (and in particular, in certain aspects, the mixing rule calculator) can be configured to calculate [e.g., a "parametric reconstruction"] mixing rule [e.g., a mixing matrix] using the channel levels and related information of the original signal, covariance information associated with the downmix signal, and the identities of the original channels, and the identities of the synthesized channels.

[0253] The audio synthesizer may select [e.g. by selection such as manual selection, or by pre-selection, or automatically such as by identifying the number of loudspeakers] a number of channels for the synthesized signal, the number of channels being independent of at least one of the channel levels of the original channels and the related information in the side information.

[0254] In some examples, the audio synthesizer may select different prototype rules for different selections.The mixing rule calculator may be configured to calculate the mixing rule.

[0255] According to one aspect, a method for generating a synthesized signal from a downmix signal is provided, wherein the synthesized signal has a plurality of synthesized channels, the number of the synthesized channels being greater than one or greater than two, the method comprising:

[0256] The downmix signal is received, the downmix signal having at least one downmix channel and side information, the side information comprising:

[0257] Channel levels and related information of an original signal, wherein the original signal has a plurality of original channels, and the number of the original channels is greater than one or greater than two;

[0258] Calculate a prototype signal from the downmix signal, the prototype signal having the number of synthesized signals;

[0259] computing a mixing rule using the channel levels and related information of the original signal, covariance information associated with the downmix signal; and

[0260] The synthesized signal is generated using the prototype signal and the mixing rule [eg, rule].

[0261] According to one aspect, an audio encoder is provided for generating a downmix signal from an original signal [e.g., y], wherein the original signal has at least two channels, and the downmix signal has at least one downmix channel, the audio encoder comprising at least one of the following:

[0262] a parameter estimator configured to estimate channel levels and related information of the original signal,

[0263] A bitstream writer is used to encode the downmix signal into a bitstream, so that the downmix signal is encoded in the bitstream to have side information, wherein the side information includes channel levels and related information of the original signal.

[0264] The channel levels and related information of the original signal encoded in the side information represent channel level information associated with a total number of channels that is smaller than the original signal.

[0265] The channel levels and related information of the original signal encoded in the side information represent related information describing energy relations between at least one different channel pair in the original channels but less than the total number of channels of the original signal.

[0266] The channel levels and related information of the original signal may include at least one coherence value describing the coherence between two channels of a channel pair.

[0267] The channel levels and related information of the original signal may include at least one inter-channel level difference (ICLD) between two channels of a channel pair.

[0268] The audio encoder may be configured to select whether to encode or not to encode at least a portion of the channel levels and related information of the original signal based on state information to include an increased amount of channel levels and related information in the side information if the payload is relatively low.

[0269] The audio encoder may be configured to select which part of the channel levels and related information of the original signal to be encoded in the side information based on a measure about the channel to include in the side information channel levels and related information associated with a more sensitive measure [e.g. a measure associated with a covariance that is more perceptually significant].

[0270] The channel levels and related information of the original signal may be in a matrix form.

[0271] The bitstream writer is configured to encode an identification of at least one channel.

[0272] According to one aspect, a method for generating a downmix signal from an original signal is provided, the original signal having at least two channels, the downmix signal having at least one downmix channel.

[0273] The method may include:

[0274] estimating channel levels and related information of the original signal,

[0275] The downmix signal is encoded into a bitstream such that the downmix signal is encoded in the bitstream to have side information including channel levels and related information of an original signal.

[0276] The audio encoder may be agnostic to the decoder. The audio synthesizer may be agnostic to the decoder.

[0277] According to one aspect, a system is provided, comprising the audio synthesizer as described above or below and one of the audio encoders as described above or below.

[0278] According to one aspect, a non-transitory storage unit storing instructions is provided. When the instructions are executed by a processor, the processor is caused to perform a method as described above or as described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0279] 3. Example

[0280] 3.1 Attached Figures

[0281] Figure 1 A simplified overview of the process according to the invention is shown.

[0282] Figure 2a An audio encoder according to the invention is shown.

[0283] Figure 2b Another view of an audio encoder according to the invention is shown.

[0284] Figure 2c Another view of an audio encoder according to the invention is shown.

[0285] Figure 2d Another view of an audio encoder according to the invention is shown.

[0286] Figure 3a An audio synthesizer (decoder) according to the invention is shown.

[0287] Figure 3b Another view showing an audio synthesizer (decoder) according to the invention.

[0288] Figure 3cAnother view showing an audio synthesizer (decoder) according to the invention.

[0289] Figures 4a-4d An example of covariance synthesis is shown.

[0290] Figure 5 An example of a filter bank for an audio encoder according to the present invention is shown.

[0291] Figure 6a-6c An example of the operation of an audio encoder according to the present invention is shown.

[0292] FIG. 7 shows an example of the prior art.

[0293] Figure 8a-8c An example of how to obtain covariance information according to the present invention is shown.

[0294] Figures 9a-9d An example of an inter-channel coherence matrix is ​​shown.

[0295] Figure 10a-10b An example of a frame is shown.

[0296] Fig.11 The scheme used by the decoder to obtain the mixing matrix is ​​shown. DETAILED DESCRIPTION

[0297] 3.2 About the concept of the present invention

[0298] It will be shown that the example is based on the encoder downmixing the signal 212 and providing the decoder with channel level and correlation information 220. The decoder can generate a mixing rule (e.g., a mixing matrix) from the channel level and correlation information 220. Information that is important for generating the mixing rule may include covariance information of the original signal 212 (e.g., the covariance matrix C y ) and the covariance information of the downmix signal (e.g., the covariance matrix C x ). Although the covariance matrix C x It can be directly estimated by the decoder by analyzing the downmix signal, but the covariance matrix C of the original signal 212 is y The covariance matrix C of the original signal 212 is easily estimated by the decoder. yIt is typically a symmetric matrix (e.g. a 5x5 matrix in the case of a 5 channel original signal 212): while the matrix shows the levels of each channel at the diagonal, it exhibits covariances between the channels at the non-diagonal entries. The matrix is ​​a diagonal matrix because the covariance between generic channels i and j is the same as the covariance between j and i. Therefore, in order to provide the decoder with the entire covariance information, it is necessary to signal to the decoder the 5 levels at the diagonal entries and the 10 covariances at the non-diagonal entries. However, it will be shown that it is feasible to reduce the amount of information to be encoded.

[0299] Furthermore, it will be shown that in some cases, instead of providing the level and covariance, standardized values ​​may be provided. For example, an inter-channel coherence value (ICC, also referred to as ξ) indicating an energy value may be provided. i,j Indicator) and inter-channel level difference (ICLD, inter channel level difference, also expressed as χ i Indication). ICC can be, for example, to provide correlation values ​​instead of the matrix C y The covariance of the off-diagonal terms of . Examples of relevant information could be In some examples, only ξ i,j part of the code that actually does the encoding.

[0300] In this way, an ICC matrix is ​​generated. The diagonal entries of the ICC matrix will in principle be equal to 1, so they do not have to be encoded in the bit stream. However, it is understood that it is feasible for the encoder to provide the ICLD to the decoder, for example in In some examples, all x i are actually encoded.

[0301] Figures 9a to 9d An example of an ICC matrix 900 is shown, where the diagonal values ​​"d" may be ICLDx i , while the off-diagonal values ​​are indicated by 902, 904, 905, 906, 907 (see below), which can be the ICCξ i,j .

[0302] In this document, the product between matrices is indicated without a sign. For example, the product between matrix A and matrix B is indicated by AB. The conjugate transpose of a matrix is ​​indicated by an asterisk (*).

[0303] When referring to a diagonal, it is the main diagonal.

[0304] 3.3 The present invention

[0305] Figure 1 An audio system 100 with an encoder side and a decoder side is shown. The encoder side can be implemented by an encoder 200, and an audio signal 212 can be obtained, for example, from an audio sensor unit (e.g., a microphone), or can be obtained from a storage unit or from a remote unit (e.g., via radio transmission). The decoder side can be implemented by an audio decoder (audio synthesizer) 300, which can provide audio content to an audio reproduction unit (e.g., a speaker). The encoder 200 and the decoder 300 can communicate with each other, for example, through a communication channel, which can be wired or wireless (e.g., through radio frequency waves, light or ultrasound waves, etc.). The encoder and / or decoder can therefore include or be connected to a communication unit (e.g., an antenna, a transceiver, etc.) for transmitting the encoded bit stream 248 from the encoder 200 to the decoder 300. In some cases, the encoder 200 can store the encoded bit stream 248 in a storage unit (e.g., a RAM memory, a FLASH memory, etc.) for future use. Similarly, the decoder 300 can read the bit stream 248 stored in the storage unit. In some examples, encoder 200 and decoder 300 may be the same device: after bitstream 248 has been encoded and stored, the device may need to read it to play back the audio content.

[0306] Figure 2a , 2b, 2c and 2d show examples of encoder 200. In some examples, Figure 2a and 2b The encoders of 2c and 2d may be identical and differ from each other only by the absence of certain elements in one and / or the other figure.

[0307] The audio encoder 200 may be configured to generate a downmix signal 246 from an original signal 212 (the original signal 212 having at least two (eg three or more) channels and a downmix signal 246 having at least one downmix channel).

[0308] The audio encoder 200 may include a parameter estimator 218 configured to estimate channel levels and related information 220 of the original signal 212. The audio encoder 200 may include a bitstream writer 226 for encoding the downmix signal 246 into a bitstream 248. Thus, the downmix signal 246 is encoded in the bitstream 248 in such a way that it has the side information 228 including the channel levels and related information of the original signal 212.

[0309] In particular, in some examples, the input signal 212 can be understood as a time domain audio signal, such as a time sequence of audio samples. The original signal 212 has at least two channels, which can correspond to different microphones (e.g., for stereo audio positions, or multi-channel audio positions), or correspond to different speaker positions of an audio reproduction unit, for example. The input signal 212 can be downmixed at the downmixer calculation block 244 to obtain a downmixed version 246 (also represented as x) of the original signal 212. This downmixed version of the original signal 212 is also referred to as a downmixed signal 246. The downmixed signal 246 has at least one downmixed channel. The downmixed signal 246 has fewer channels than the original signal 212. The downmixed signal 212 can be in the time domain.

[0310] The downmix signal 246 is encoded in a bitstream 248 by a bitstream writer 226 (e.g., including an entropy encoder, or a multiplexer, or a core encoder) for storing the bitstream or transmitting it to a receiver (e.g., associated with a decoder side). The encoder 200 may include a parameter estimator (or parameter estimation block) 218. The parameter estimator 218 may estimate channel levels and related information 220 associated with the original signal 212. The channel levels and related information 220 may be encoded in the bitstream 248 as side information 228. In an example, the channel levels and related information 220 are encoded by the bitstream writer 226. In an example, even if Figure 2b The bitstream writer 226 is not shown downstream of the downmix calculation block 235, but may still be present. Figure 2c , it is shown that the bitstream writer 226 may include a core encoder 247 to encode the downmix signal 246 to obtain an encoded version of the downmix signal 246 . Figure 2c It is also shown that the bitstream writer 226 may include a multiplexer 249 which encodes in the bitstream 228 both the encoded downmix signal 246 and the channel levels and related information 220 in the side information 228 (eg as encoded parameters).

[0311] like Figure 2b As shown (in Figure 2a and 2c ), the original signal 212 may be processed (eg, by a filter bank 214, see below) to obtain a frequency domain version 216 of the original signal 212.

[0312] An example of parameter estimation is shown in Figure 6c , where the parameter estimator 218 defines the parameter ξ i,j and χ i(e.g., standardized parameters) to be subsequently encoded in the bitstream. The covariance estimators 502 and 504 estimate the covariance C for the downmix signal 246 to be encoded and the input signal 212, respectively. x and C y Then, in ICLD block 506, the ICLD parameter x i is calculated and provided to the bitstream writer 246. At the covariance-to-coherence block 510, ICCξ i,j (412) is obtained. At block 250, only some ICCs are selected to be encoded.

[0313] Parameter quantization block 222 ( Figure 2b ) may allow obtaining channel levels and related information 220 in a quantized version 224.

[0314] The channel levels and related information 220 of the original signal 212 may generally include information about the energy (or level) of the channels of the original signal 212. Additionally or alternatively, the channel levels and related information 220 of the original signal 212 may include related information between channel pairs, such as correlation between two different channels. The channel levels and related information may include information related to the covariance matrix C y The associated information (e.g., in its normalized form, such as correlation or ICC), where each column and each row is associated with a particular channel of the original signal 212, and is represented by the matrix C y The diagonal elements and related information are used to describe the channel level, and through the matrix C y The non-diagonal elements of are used to describe the relevant information. y Can be a symmetric matrix (i.e. it is equal to its transpose) or a Hermitian matrix (i.e. it is equal to its conjugate transpose). y is typically positive semidefinite. In some examples, the correlation may be replaced by the covariance (and the correlation information replaced by the covariance information). It is understood that it is feasible to encode information associated with less than the total number of channels of the original signal 212 in the side information 228 of the bitstream 248. For example, it is not necessary to provide channel levels and related information for all channels or all channel pairs. For example, only a reduced set of information about the correlation between channel pairs of the downmix signal 212 may be encoded in the bitstream 248, while the remaining information may be estimated at the decoder side. Typically, the comparison C y It is feasible to encode the elements with fewer diagonal elements, and compared with C y It is feasible to encode elements with few elements outside the diagonal.

[0315] For example, the channel levels and related information may include the covariance matrix C of the original signal 212 y (channel levels and related information 220 of the original signal) and / or the covariance matrix C of the downmix signal 246 x (covariance information of the downmix signal), for example in a normalized form. For example, the covariance matrix may associate each row and each column with each channel to represent the covariance between different channels, and the diagonal of the matrix represents the level of each channel. In some examples, the channel level and related information 220 of the original signal 212 encoded in the side information 228 may include only the channel level information (e.g., only the correlation matrix C y The diagonal values ​​of ) or only related information (for example, only the correlation matrix C y The same applies to the covariance information of the downmix signal.

[0316] As will be shown later, the channel level and related information 220 may include at least one coherence value (ξ i,j ), describing the coherence between two channels i and j in a channel pair i, j. Additionally or alternatively, the channel levels and related information 220 may include at least one inter-channel level difference ICLD(X i ). In particular, it is feasible to define a matrix with ICLD values ​​or ICC values. Therefore, the above matrix C y and C x The example of transmission of elements of may be generalized for other values ​​to be encoded (eg, transmitted) for implementing channel level and related information 220 and / or coherent information of downmix channels.

[0317] The input signal 212 may be subdivided into a plurality of frames. Different frames may have, for example, the same time length (e.g., each frame may be constructed by the same number of samples in the time domain during the time of one frame). Therefore, different frames generally have equal time lengths. In the bitstream 248, the downmix signal 246 (which may be a time domain signal) may be encoded in a frame-by-frame manner (or in any case, it may be determined by the decoder to be subdivided into frames). As encoded as side information 228 in the bitstream 248, channel levels and related information 220 may be associated with each frame (e.g., parameters of channel levels and related information 220 may be provided for each frame or for a plurality of consecutive frames). Accordingly, for each frame of the downmix signal 246, the associated side information 228 (e.g., parameters) may be encoded in the side information 228 of the bitstream 248. In some cases, a plurality of consecutive frames may be associated with the same channel level and related information 220 (e.g., with the same parameters) as encoded in the side information 228 of the bitstream 248. Accordingly, a parameter may result in being associated with a plurality of consecutive frames in common. In some examples, this may happen when two consecutive frames have similar properties, or when the bit rate needs to be reduced (e.g. due to the necessity of reducing the payload). For example:

[0318] In case of high payload, the number of consecutive frames associated with the same specific parameters is increased to reduce the number of bits written to the bitstream;

[0319] In case of low payload, the number of consecutive frames associated with the same specific parameters is reduced to improve the mixing quality.

[0320] In other cases, when the bit rate is reduced, the number of consecutive frames associated with the same specific parameters is increased to reduce the number of bits written to the bitstream, and vice versa.

[0321] In some cases, it is feasible to use a linear combination of parameters (or reconstructed or estimated values, such as covariance) from previous frames to the current frame, for example by adding, averaging, etc., to smooth the parameters (or reconstructed or estimated values, such as covariance).

[0322] In some examples, a frame may be divided between multiple subsequent time slots. Fig.10a Frame 920 is shown (subdivided into four consecutive time slots 921 to 924), Fig.10b A frame 930 is shown (subdivided into four consecutive time slots 931 to 934). The time length of the different time slots may be the same. If the length of the frame is 20 ms and a slot size of 1.25 ms, there are 16 time slots in one frame (20 / 1.25=16).

[0323] The time slot subdivision may be performed in a filter bank (eg, 214), as discussed below.

[0324] In the example, the filter bank is a complex modulated low delay filter bank (CLDFB), the frame size is 20ms, the slot size is 1.25ms, resulting in 16 filter banks per frame and the number of frequency bands per time slot depends on the input sampling frequency, and wherein the frequency band has a width of 400 Hertz (Hz). Thus, for example, for an input sampling frequency of 48 kHz, the length of the frame in samples is 960, the slot length is 60 samples, and the number of filter bank samples per time slot is also 60.

[0325]

[0326]

[0327] Even though each frame (and each time slot) may be encoded in the time domain, a band-by-band analysis may be performed. In an example, multiple bands are analyzed for each frame (or time slot). For example, a filter bank may be applied to the time signal and the resulting sub-band signals may be analyzed. In some examples, the channel levels and related information 220 are also provided in a band-by-band manner. For example, for each frequency band of the input signal 212 or the downmix signal 246, the associated channel levels and related information 220 (e.g., C y or ICC matrix) can be provided. In some examples, the number of frequency bands can be modified based on properties of the signal and / or properties of the requested bit rate, or properties of measurements on the current payload. In some examples, more time slots are required, and fewer frequency bands are used to maintain a similar bit rate.

[0328] Since the size of a time slot is smaller than the size of a frame (in time length), the time slot can be used appropriately in case a transient in the original signal 212 is detected within a frame: the encoder (in particular the filter bank 214) can recognize the presence of a transient, signal its presence in the bitstream and indicate in the side information 228 of the bitstream 248 in which time slot of the frame the transient has occurred. Furthermore, the parameters of the channel levels and related information 220 encoded in the side information 228 of the bitstream 248 can thus be associated only with the time slots subsequent to the transient and / or with the time slots in which the transient has occurred. Thus, the decoder will determine the presence of a transient and associate the channel levels and related information 220 only with the time slots subsequent to the transient and / or with the time slots in which the transient has occurred (for the time slots before the transient, the decoder will use the channel levels and related information 220 of the previous frame). Fig.10a , no transient has occurred, and the parameters 220 encoded in the side information 228 can therefore be understood as being associated with the entire frame 920. Fig.10b, the transient has occurred at time slot 932: therefore, the parameters 220 encoded in the side information 228 will refer to time slots 932, 933 and 934, while the parameters associated with time slot 931 will be assumed to be the same as the parameters of the frame preceding frame 930.

[0329] In view of the above, for each frame (or time slot) and each frequency band, a specific channel level and related information 220 related to the original signal 212 can be defined. For example, the covariance matrix C can be estimated for each frequency band y elements of (e.g. covariance and / or level).

[0330] If the detection of a transient occurs while multiple frames are commonly associated with the same parameter, it may be feasible to reduce the number of frames commonly associated with the same parameter, thereby increasing the quality of the mix.

[0331] Fig.10a A frame 920 (here indicated as a "normal frame") is shown for which eight frequency bands are defined in the original signal 212 (eight frequency bands 1 ... 8 are shown on the ordinate and time slots 921 to 924 are shown on the abscissa). Parameters of the channel levels and related information 220 can theoretically be encoded in the side information 228 of the bitstream 248 in a band-by-band manner (e.g. there will be one covariance matrix for each original frequency band). However, in order to reduce the amount of side information 228, the encoder may aggregate a plurality of original frequency bands (e.g. contiguous frequency bands) to obtain at least one aggregated band formed of a plurality of original frequency bands. For example, in Fig.10a , eight original bands are grouped to obtain four aggregated bands (aggregated band 1 is associated with original band 1; aggregated band 2 is associated with original band 2; aggregated band 3 groups original bands 3 and 5; aggregated band 4 groups original bands 5…8). Matrices of covariance, correlation, ICC, etc. may be associated with each of the aggregated bands. In some examples, encoded in the side information 228 of the bitstream 248 are parameters obtained from the sum (or average or another linear combination) of the parameters associated with each aggregated band. Thus, the size of the side information 228 of the bitstream 248 is further reduced. In the following, “aggregated band” is also referred to as “parameter band” because it means those bands that are used to determine the parameters 220.

[0332] Fig.10bA frame 931 (subdivided into four consecutive time slots 931 to 934, or another integer) is shown in which a transient occurs. Here, the transient occurs in the second time slot 932 (the "transient slot"). In this case, the decoder may determine to direct the parameters of the channel levels and related information 220 only to the transient time slot 932 and / or the subsequent time slots 933 and 934. The channel levels and related information 220 of the previous time slot 931 will not be provided: it is understood that the channel levels and related information of time slot 931 will in principle be different from the channel levels and related information of the time slot, but may be more similar to the channel levels and related information of the frame before frame 930. Therefore, the decoder applies the channel levels and related information of the frame before frame 930 to time slot 931, and the channel levels and related information of frame 930 are only applied to time slots 932, 933 and 934.

[0333] Since the presence and location of the time slot 931 with transients can be signaled in the side information 228 of the bitstream 248 (e.g., in 261, as shown later), a technique has been developed to avoid or reduce the size increase of the side information 228: the grouping between the aggregated bands can be changed: for example, aggregated band 1 groups the original bands 1 and 2, and aggregated band 2 groups the original bands 3 ... 8. Thus, relative to Fig.10a In the case of , the number of bands is further reduced and parameters will be provided for only two aggregated bands.

[0334] Figure 6a A parameter estimation block (parameter estimator) 218 ​​is shown capable of retrieving a certain number of channel levels and related information 220 .

[0335] Figure 6a The parameter estimator 218 is shown to be able to retrieve a certain number of parameters (channel levels and related information 220), which may be Figures 9a to 9d The ICC of the matrix is ​​900.

[0336] However, only a portion of the estimated parameters are actually submitted to the bitstream writer 226 to encode the side information 228. This is because the encoder 200 can be configured to select (in Figures 1 to 5 2 and 3. In the embodiment of the present invention, it is determined whether at least a portion of the channel levels and related information 220 of the original signal 212 are encoded (not shown in the determination block 250).

[0337] This is Figure 6a 254s, which are controlled by selections (commands) 254 from the determination block 250. If each of the outputs 220 of the block parameter estimation 218 is Fig.9cIf the ICC of the matrix 900 is coded, then not all parameters estimated by the parameter estimation block 218 are actually encoded in the side information 228 of the bitstream 248: in particular, while item 908 (ICC between channels: R and L; C and L; C and R; RS and CS) is actually encoded, item 907 is not (i.e., the determination block 250 may be used in conjunction with Figure 6c , can be considered as having opened the switch 254s for the uncoded item 907, but having closed the switch 254s for the item 908 to be coded in the side information 228 of the bitstream 248. It is to be noted that the information 254' (item 908) about which parameters have been selected to be coded can be coded (e.g. as a bitmap other information about which item 908 is coded). In fact, the information 254' (which can be an ICC map, for example) can include the index of the coded item 908 (in Figure 9d ). The information 254' may be in the form of a bitmap: for example, the information 254' may consist of fields of fixed length, each position being associated with an index according to a predetermined order, and the value of each bit providing information on whether the parameter associated with the index is actually provided.

[0338] In general, the determination block 250 may, for example, select whether to encode at least a portion of the channel levels and related information 220 (i.e. decide whether the entries of the matrix 900 are to be encoded), for example on the basis of the state information 252. The state information 252 may be based on the payload state: for example, in case the transmission is highly loaded, it will be possible to reduce the amount of side information 228 to be encoded in the bitstream 248. For example, and with reference to Fig.9c :

[0339] In case of high payload, the number of entries 908 of the matrix 900 that are actually written into the side information 228 of the bitstream 248 is reduced;

[0340] In the case of lower payloads, the number of entries 908 of the matrix 900 that are actually written into the side information 228 of the bitstream 248 is reduced.

[0341] Alternatively or additionally, the metrics 252 may be evaluated to determine which parameters 220 are to be encoded in the side information 228 (e.g., which entries of the matrix 900 are designated as the encoded entries 908, and which entries are to be discarded). In this case, only the parameters 220 (associated with more sensitive metrics, such as metrics associated with perceptually more important covariances, may be associated with the entries to be selected as the encoded entries 908) may be encoded in the bitstream.

[0342] It is to be noted that this process may be repeated for each frame (or multiple frames in case of downsampling) and for each frequency band.

[0343] Therefore, in addition to the state metrics, etc., the determination block 250 may also be determined by the parameter estimator 218 through Figure 6a Controlled by command 251.

[0344] In some examples (e.g. Figure 6b ), the audio encoder may be further configured to encode the current channel level and related information 220t in the bitstream 248 as a delta 220k relative to the previous channel level and related information 220(t-1). What the bitstream writer 226 encodes in the side information 228 may be the delta 220k associated with the current frame (or time slot) relative to the previous frame. Figure 6b The current channel level and related information 220t are provided to the storage element 270 so that the storage element 270 stores the value of the current channel level and related information 220t for the subsequent frame. At the same time, the current channel level and related information 220t can be compared with the previously obtained channel level and related information 220(t-1). (This is in Figure 6b 270 is shown as a subtractor 273 in the side information 228). Therefore, a subtraction result 220Δ may be obtained by the subtractor 273. The difference 220Δ may be used at the scaler 220s to obtain a relative increment 220k between the previous channel level and related information 220(t-1) and the current channel level and related information 220t. For example, if the current channel level and related information 220t is 10% greater than the previous channel level and related information 220(t-1), the increment 220 encoded in the side information 228 by the bitstream writer 226 will indicate information of the 10% increment. In some examples, instead of providing the relative increment 220k, the difference 220Δ may simply be encoded.

[0345] Among the parameters such as ICC and ICLD discussed above and below, the choice of the parameters to actually be encoded can be adapted to the specific situation. For example, in some examples:

[0346] For a first frame, select only Fig.9c The ICC 908 is to be encoded in the side information 228 of the bitstream 248, while the ICC 907 is not to be encoded in the side information 228 of the bitstream 248;

[0347] For the second frame, a different ICC is selected to be encoded, while a different non-selected ICC is not encoded.

[0348] For time slots and frequency bands (and for different parameters, such as ICLD), it may be equally effective. Therefore, the encoder (particularly block 250) can determine which parameter is to be encoded and which parameter is not encoded, thereby making the selection of the parameter to be encoded adapted to a particular situation (e.g., state, selection ...). Therefore, the "importance feature (feature for importance)" can be analyzed to select which parameter is to be encoded and which parameter is not encoded. The importance feature can be a measure associated with the result obtained in the simulation of the operation performed by the decoder, for example. For example, the encoder can simulate the reconstruction of the uncoded covariance parameters 907 by the decoder, and the feature of importance can be a measure indicating the absolute error between the uncoded covariance parameters 907 and the same parameters reconstructed by the decoder. By measuring the errors in different simulation scenarios (e.g. each simulation scenario is associated with the transmission of certain coded covariance parameters 908 and a measure of the errors affecting the reconstruction of the uncoded covariance parameters 907), it is possible to determine the simulation scenario least affected by the errors (e.g. a measure of all errors in the reconstruction in the simulation scenario) in order to distinguish the covariance parameters 908 to be coded from the uncoded covariance parameters 907 based on the least affected simulation scenario. In the case of the least affected scenario, the unselected parameters 907 are the parameters that are easiest to reconstruct, while the selected parameters 908 tend to be the parameters with the largest measure of errors associated with them.

[0349] The same can be done by simulating the reconstruction of the decoder or estimating the covariance, or by simulating the mixing characteristics or mixing results, instead of simulating parameters such as ICC and ICLD. It is worth noting that the simulation can be done for each frame or each time slot, and can be done for each frequency band or aggregated frequency bands.

[0350] One example could be to model the reconstruction of the covariance using equation (4) or (6) (see below), starting from the parameters encoded in the side information 228 of the bitstream 248 .

[0351] More generally, it is possible to reconstruct channel levels and related information from selected channel levels and related information, thereby simulating channel levels and related information (220, C) not selected at the decoder (300). y ) and calculate the error information between:

[0352] unselected channel levels and related information estimated by the encoder (220); and

[0353] unselected channel levels and related information reconstructed by simulating estimates of channel levels and related information (220) not encoded at the decoder (300); and

[0354] So that on the basis of the calculated error information:

[0355] Correctly reconstructed channel levels and related information; and

[0356] Channel levels and related information cannot be correctly reconstructed,

[0357] To decide:

[0358] selecting incorrectly reconstructible channel levels and related information to be encoded in the side information (228) of the bitstream (248); and

[0359] Correctly reconstructable channel levels and related information are not selected, thereby avoiding encoding the correctly reconstructable channel levels and related information in the side information (228) of the bitstream (248).

[0360] In general, the encoder can simulate any operation of the decoder and evaluate the error metric based on the simulation results.

[0361] In some examples, the importance feature may be different from (or may include other metrics that are different from) the evaluation of the metric associated with the error. In some cases, the importance feature may be associated with a manual selection, or based on importance based on psychoacoustic criteria. For example, the most important channel pairs may be selected to be encoded (908), even without simulation.

[0362] Now, some additional discussion is provided for explaining how the encoder signals which parameters 908 are actually encoded in the side information 220 of the bitstream 248.

[0363] refer to Figure 9d , the parameters on the diagonal of the ICC matrix 900 are associated with ordered indices 1...10 (the order is predetermined and known to the decoder). Fig.9c In FIG. 1 , it is shown that the selected parameter 908 to be encoded is the ICC for LR, LC, RC, LS-RS indexed by indices 1, 2, 5, 10, respectively. Therefore, in the side information 228 of the bitstream 248, an indication of the indices 1, 2, 5, 10 will also be provided (e.g., in Figure 6ainformation 254' of the side information 228). Accordingly, with the help of the information about the indices 1, 2, 5, 10 provided by the encoder in the side information 228, the decoder will understand that the four ICCs provided in the side information 228 of the bitstream 248 are LR, LC, RC, LS-RS. The index can be provided, for example, by associating the position of each bit in the bitmap with a predetermined position. For example, in order to signal the index 1, 2, 5, 10, "1100100001" can be written (in the field 254' of the side information 228), because the first, second, fifth and tenth bits refer to the indexes 1, 2, 5, 10 (other possibilities are at the disposal of the skilled person). This is a so-called one-dimensional index, but other indexing strategies are also possible. For example, a combinatorial number technique, according to which a number N is encoded (in the field 254' of the side information 228), the number N is unambiguously associated with a specific channel pair (see also https: / / en.wikipedia.org / wiki / Combinatorial_number_system). When the bitmap points to ICC, it can also be called an ICC map.

[0364] It should be noted that in some cases, non-adaptive (fixed) parameter provisioning is used. This means that Figure 6a In the example of , the selection 254 among the parameters to be encoded is fixed and the selected parameter does not need to be indicated in the field 254'. Figure 9b An example of fixed provision of parameters is shown: the selected ICCs are LC, L-LS, RC, C-RS, and no signaling of their indices is necessary, since the decoder already knows which ICCs are encoded in the side information 228 of the bitstream 248 .

[0365] However, in some cases, the encoder may choose between fixed provision of parameters and adaptive provision of parameters. The encoder may signal the choice in the side information 228 of the bitstream 248 so that the decoder may know which parameters are actually encoded.

[0366] In some cases, at least some parameters can be provided without modification: for example:

[0367] ICDLs can be encoded in any case without indicating them in the bitmap; and

[0368] ICCs may be subject to adaptive provision.

[0369] The explanation relates to each frame, or time slot, or frequency band. For subsequent frames, time slots, or frequency bands, different parameters 908 are provided to the decoder, and different indices are associated with the subsequent frames, time slots, or frequency bands; and different selections (e.g., fixed and adaptive) can be made. Figure 5 2 shows an example of a filter bank 214 of the encoder 200, which can be used to process the original signal 212 to obtain a frequency domain signal 216. Figure 5 As can be seen, the time domain (TD) signal 212 can be analyzed by the transient analysis block 258 (transient detector). In addition, a conversion of a frequency domain (FD) version 264 of the input signal 212 in multiple frequency bands is provided by a filter 263 (which can implement, for example, a Fourier filter, a short Fourier filter, an orthogonal mirror, etc.). The frequency domain version 264 of the input signal 212 can be analyzed, for example at a band analysis block 267, which can decide (command 268) a specific group of frequency bands to be performed at the partition grouping block 265. Thereafter, the FD signal 216 will be a signal with a reduced number of aggregated frequency bands. The aggregation of frequency bands has been described above with respect to Fig.10a and Fig.10b for explanation. The partition grouping block 267 may also be adjusted by transient analysis performed by the transient analysis block 258. As described above, in the case of transients, it is possible to further reduce the number of aggregated bands: therefore, the information 260 about transients may adjust the partition grouping. Additionally or alternatively, information 261 about transients is encoded in the side information 228 of the bitstream 248. When the information 261 is encoded in the side information 228, the information 261 may include, for example, a flag indicating whether a transient has occurred (such as: "1", meaning "there is a transient in the frame", and "0" meaning: "there is no transient in the frame") and / or an indication of the position of the transient in the frame (such as a field indicating in which time slot the transient has been observed). In some examples, when the information 261 indicates that there is no transient in the frame ("0"), an indication of the position of the transient is encoded in the side information 228 to reduce the size of the bitstream 248. The information 261 is also referred to as a "transient parameter", and as shown in FIG. Figure 2d and 6b As shown, it is encoded in the side information 228 of the bitstream 246.

[0370] In some examples, the partition grouping at block 265 may also be conditioned by external information 260', such as information about the state of the transmission (e.g., measurements associated with the transmission, error rates, etc.). For example, the higher the payload (or the greater the error rate), the greater the aggregation (the tendency for less aggregated bandwidth to be wider), thereby causing a smaller amount of side information 228 to be encoded in the bitstream 248. In some examples, the information 260' may be similar to Figure 6a information or measurement 252.

[0371] It is usually not feasible to send parameters for each frequency band / time slot combination, but filter bank samples are grouped both over multiple time slots and multiple frequency bands to reduce the number of parameter sets sent per frame. Along the frequency axis, grouping the frequency bands into parameter bands uses a non-constant division in the parameter bands, where the number of bands in a parameter band is not constant, but rather attempts to follow a psychoacoustically motivated parameter band resolution, i.e., at lower frequency bands, the parameter bands include only one or a small number of filter bank bands, and for higher parameter bands a larger (and steadily increasing) number of filter bank bands are grouped into one parameter band.

[0372] Thus, for example, for an input sampling rate of 48kHz and the number of parameter bands set to 14, the vector grp below 14 Describes the filter bank index which gives the band boundaries for the parameter band (index starts at 0):

[0373] grp 14 =[0,1,2,3,4,5,6,8,10,13,16,20,28,40,60]

[0374] The parameter band j includes the filter bank band [grp 14 [j],grp 14 [j+1][

[0375] Note that the bands grouped at 48 kHz can also be used directly for other possible sampling rates by simply truncating the bands, since the groupings all follow the psychoacoustically motivated frequency scale and have certain band boundaries corresponding to the number of bands per sampling frequency (Table 1).

[0376] If the frame is non-transient, or no transient processing is implemented, the grouping along the time axis will traverse all time slots in the frame so that one parameter set is available per parameter band.

[0377] Nevertheless, the number of parameter sets is still large, but the temporal resolution can be lower than a 20ms frame (40ms on average). Therefore, in order to further reduce the number of parameter sets sent per frame, only a subset of the parameter bands is used to determine and encode the parameters for sending to the decoder in the bitstream. The subsets are fixed and known to both the encoder and the decoder. The specific subset sent in the bitstream is signaled by a field in the bitstream to indicate to the decoder which subset of parameter bands the transmitted parameters belong to, and the decoder then replaces the parameters for this subset with the transmitted parameters (ICC, ICLD) and keeps the parameters (ICC, ICLD) from the previous frame for all parameter bands that are not in the current subset.

[0378] In an example, the parameter bands may be divided into two subsets that include roughly half of the total parameter bands and a contiguous subset for the lower parameter band and one contiguous subset for the upper parameter band. Since we have two subsets, the bitstream field used to signal the subsets is a single bit, and an example of a subset for 48kHz and 14 parameter bands is:

[0379] s 14 =[1,1,1,1,1,1,1,0,0,0,0,0,0,0]

[0380] where s 14 [j] indicates which subset of parameter band j it belongs to.

[0381] It is to be noted that the downmix signal 246 may be a signal in the time domain actually encoded in the bitstream 248: simply, the subsequent parameter estimator 218 will estimate the parameters 220 (eg ξ i,j and / or x i ) (and the decoder 300 will use the parameters 220 for preparing a mixing rule (eg, a mixing matrix) 403, which will be explained below.

[0382] Figure 2d An example of an encoder 200 is shown, which may be one of the encoders described above or may include elements of the encoders discussed previously. A TD input signal 212 is input to the encoder, and a bitstream 248 is output, which includes a downmix signal 246 (e.g., encoded by a core encoder 247) and correlation and level information 220 encoded in side information 228.

[0383] from Figure 2d It can be seen that the filter bank 214 (in Figure 5An example of a filter bank is provided in FIG. 2 ). A frequency domain (FD) conversion (frequency domain DMX) is provided in block 263 to obtain a FD signal 264, which is a FD version of the input signal 212. The FD signal 264 is obtained in a plurality of frequency bands (also indicated by X). A frequency band / time slot grouping block 265 (which may be implemented as Figure 5 265) to obtain the FD signal 216 in the aggregated frequency band. In some examples, the FD signal 216 may be a version of the FD signal 264 in fewer frequency bands. Next, the signal 216 may be provided to the parameter estimator 218, which includes the covariance estimation blocks 502, 504 (shown here as a single block), and downstream parameter estimation and encoding blocks 506, 510 (an embodiment of elements 502, 504, 506 and 510 is shown in FIG. Figure 6c ). The parameter estimation encoding blocks 506, 510 may also provide the parameters 220 to be encoded in the side information 228 of the bitstream 248. The transient detector 258 (which may be implemented as Figure 5 The transient analysis block 258 of the present invention can find out transients and / or the position of transients within a frame (e.g., in which time slot the transient has been identified). Therefore, information 261 about the transients (e.g., transient parameters) can be provided to the parameter estimator 218 (e.g., to decide which parameters to be encoded). The transient detector 258 can also provide information or commands (268) to the block 265 to perform grouping by taking into account the presence and / or position of transients in the frame.

[0384] Figure 3a , 3b, 3c show an example of an audio decoder 300 (also called an audio synthesizer). In the example, Figure 3a , 3b, 3c may be the same decoder, but with some differences to avoid different elements. In the example, decoder 300 may be the same as Figure 1 The decoder 300 is the same as the decoder 4. In an example, the decoder 300 may also be the same device as the encoder 200.

[0385] The decoder 300 may be configured to generate a synthesized signal (336, 340, y) from the downmix signal x in the TD (246) or the FD (314). R). The audio synthesizer 300 may include an input interface 312 configured to receive a downmix signal 246 (e.g., the same downmix signal encoded by the encoder 200) and side information 228 (e.g., encoded in the bitstream 248). As explained above, the side information 228 may include channel levels and related information (220, 314) of the original signal (which may be the original input signal 212, y at the encoder side), such as ξ, χ, etc. or at least one of their elements (as will be explained below). In some examples, all ICLD (χ) and some (but not all) items 906 or 908 (ICC or ξ values) outside the diagonal of the ICC matrix 900 are obtained by the decoder 300.

[0386] The decoder 300 may be configured (eg, via a prototype signal calculator or prototype signal calculation module 326 ) to calculate a prototype signal 328 having multiple channels (greater than one) of the synthesized signal 336 from the downmix signal ( 324 , 246 , x).

[0387] The decoder 300 may be configured (eg, via the mixing rule calculator 402) to calculate the mixing rule 403 using at least one of the following:

[0388] The channel levels and related information (e.g., 314, C y , ξ, χ or elements thereof); and

[0389] Covariance information (eg, C x or its elements).

[0390] The decoder 300 may include a synthesis processor 404 configured to use the prototype signal 328 and the mixing rule 403 to generate a synthesized signal (336, 340, y R ).

[0391] The synthesis processor 404 and the blending rule calculator 402 may be collected in a synthesis engine 334. In some examples, the blending rule calculator 402 may be external to the synthesis engine 334. Figure 3a Mixed Rule Calculator 402 with Figure 3b The parameter reconstruction module 316 can be integrated.

[0392] Synthetic signal (336, 340, y R) is greater than 1 (in some cases greater than 2 or greater than 3), and may be greater than, less than, or equal to the number of original channels of the original signal (212, y), which is also greater than 1 (in some cases greater than 2 or greater than 3). The number of channels of the downmix signal (246, 216, x) is at least one or two, and is less than the number of original channels of the original signal (212, y) and the number of channels of the synthesized signal (336, 340, y). R ) of the synthesized channels.

[0393] The input interface 312 may read an encoded bitstream 248 (e.g., the same bitstream 248 encoded by the encoder 200). The input interface 312 may be or include a bitstream reader and / or an entropy decoder. As described above, the bitstream 248 may encode the downmix signal (246, x) and the side information 228 as described above. The side information 228 may, for example, include the original channel levels and related information 220 in a form output by the parameter estimator 218 or any element downstream of the parameter estimator 218 (e.g., the parameter quantization block 222, etc.). The side information 228 may include encoded values ​​or indexed values ​​or both. Even in Figure 3b The input interface 312 is not shown for the downmix signal (346, x), but the input interface 312 may also be as shown in FIG. Figure 3a In some examples, the input interface 312 may quantize the parameters obtained from the bitstream 248 .

[0394] Therefore, decoder 300 can obtain downmix signal (246, x), and downmix signal (246, x) can be in time domain.As mentioned above, downmix signal 246 can be divided into frame and / or time slot (see above).In an example, filter bank 320 can convert downmix signal 246 in time domain to obtain version 324 of downmix signal 246 in frequency domain.As mentioned above, the frequency band of frequency domain version 324 of downmix signal 246 can be grouped into frequency band group.In an example, the same grouping (see above) carried out at filter bank 214 can be performed.Parameters for grouping (such as which frequency bands and / or how many frequency bands are to be grouped ...) can be based on the signal notification of partition grouper 265 or frequency band analysis block 267, for example, and the signal notification is encoded in side information 228.

[0395] The decoder 300 may include a prototype signal calculator 326. The prototype signal calculator 326 may calculate a prototype signal 328 from the downmix signal (e.g., one of the versions 324, 246, x), for example, by applying a prototype rule (e.g., a matrix Q). The prototype rule may be implemented by a prototype matrix (Q) having a first dimension and a second dimension, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesized channels. Thus, the prototype signal has a plurality of channels of the synthesized signal 340 that is ultimately to be generated.

[0396] The prototype signal calculator 326 may apply a so-called upmix to the downmix signal (324, 246, x) in that it simply generates a version of the downmix signal (324, 246, x) with an increased number of channels (the number of channels of the synthesized signal to be generated), but without applying too much "intelligence". In an example, the prototype signal calculator 326 may simply apply a fixed predetermined prototype matrix (identified as "Q" in this document) to the FD version 324 of the downmix signal 246. In an example, the prototype signal calculator 326 may apply different prototype matrices to different frequency bands. For example, based on a specific number of downmix channels and a specific number of synthesized channels, a prototype rule (Q) may be selected from a plurality of pre-stored prototype rules.

[0397] The prototype signal 328 may be decorrelated at the decorrelation module 330 to obtain a decorrelated version 332 of the prototype signal 328. However, in some examples, the decorrelation module 330 is advantageously absent, as the present invention has proven to be sufficiently effective to allow its avoidance.

[0398] The prototype signal (in any of its versions 328, 332) may be input to the synthesis engine 334 (and in particular the synthesis processor 404). There, the prototype signal (328, 332) is processed to obtain the synthesized signal (336, R ). The synthesis engine 334 (and in particular the synthesis processor 404) may apply a mixing rule 403 (in some examples, discussed below, the mixing rule is two, e.g., one for the principal component of the synthesized signal and one for the residual component). The mixing rule 403 may be implemented, for example, by a matrix. The matrix 403 may be generated, for example, by a mixing rule calculator 402 based on the channel levels of the original signal (212, y) and related information (314, such as ξ, χ or elements thereof).

[0399] The synthesized signal 336 output by the synthesis engine 334 (particularly by the synthesis processor 404) may be optionally filtered at a filter bank 338. Additionally or alternatively, the synthesized signal 336 may be converted to the time domain at the filter bank 338. Thus, a version 340 of the synthesized signal 336 (in the time domain or after filtering) may be used for audio reproduction (e.g., through a speaker).

[0400] In order to obtain a mixing rule (eg, a mixing matrix) 403, the channel levels and related information (eg, C y , etc.) and covariance information associated with the downmix signal (e.g., C x ) may be provided to the mixing rule calculator 402. For this purpose, it is feasible to encode the channel levels and related information 220 in the side information 228 by means of the encoder 200.

[0401] However, in some cases, in order to reduce the amount of information encoded in the bitstream 248, not all parameters are encoded by the encoder 200 (e.g., not the entire channel level and related information of the original signal 212 and / or not the entire covariance information of the downmix signal 246). Therefore, some parameters 318 will be estimated at the parameter reconstruction module 316.

[0402] The parameter reconstruction module 316 may, for example, be fed with at least one of:

[0403] a copy 322 of the downmix signal 246(x), which may be, for example, a filtered version or an FD version of the downmix signal 246; and

[0404] Side information 228 (including channel levels and related information 228).

[0405] The side information 228 may include a correlation matrix C with the original signal (212, y) y Correlated information (as input signal level and related information): However, in some cases, it is not the correlation matrix C y All elements of are actually encoded. Therefore, estimation and reconstruction techniques have been developed to reconstruct the correlation matrix C y Version (e.g. by obtaining an estimated version intermediate step).

[0406] The parameters 314 provided to the module 316 may be obtained by the entropy decoder 312 (input interface) and may, for example, be quantized.

[0407] Figure 3c An example of a decoder 300 is shown, which may be Figures 1 to 3bHere, the decoder 300 comprises an input interface 312 represented by a demultiplexer. The decoder 300 outputs a composite signal 340, which may be, for example, in TD (signal 340), to be played back by a loudspeaker or in FD (signal 336). Figure 3c The decoder 300 may include a core decoder 347, which may also be part of the input interface 312. The core decoder 347 may thus provide a downmix signal x, 246. The filter bank 320 may convert the downmix signal 246 from TD to FD. The FD version of the downmix signal x, 246 is indicated at 324. The FD downmix signal 324 may be provided to a covariance synthesis block 388. The covariance synthesis block 388 may provide a synthesized signal 336 (Y) in FD. The inverse filter bank 338 may convert the audio signal 314 in its TD version 340. The FD downmix signal 324 may be provided to a band / slot grouping block 380. The band / slot grouping block 380 may perform the operations performed in the encoder by Figure 5 and 2d The same operation has been performed by the partition grouping block 265. In the encoder, as Figure 5 and 2d The frequency bands of the downmix signal 216 have been grouped or aggregated in a few frequency bands (with a wider width), and the parameters 220 (ICC, ICLD) have been associated with the aggregated frequency band groups. Now it is necessary to aggregate the decoded downmix signal in the same way, assigning each aggregated frequency band to the associated parameters. Therefore, reference numeral 385 refers to the aggregated downmix signal X B It is to be noted that the filter provides an unaggregated FD representation so that the bands / slots can be grouped in the decoder (380) to process the parameters in the same way as in the encoder, performing the same aggregation on the bands / slots as the encoder to provide an aggregated downmix X B .

[0408] The band / slot grouping block 380 may also aggregate over different slots in a frame so that the signal 385 is also aggregated with a slot size similar to that of the encoder. The band / slot grouping block 380 may also receive information 261 encoded in the side information 228 of the bitstream 248 indicating the presence of a transient and, if appropriate, the position of the transient within the frame.

[0409] At the covariance estimation block 384, the covariance C of the downmix signal 246 (324) is estimated. x At the covariance calculation block 386, the covariance C is obtained. y , for example by utilizing equations (4) to (8) can be used for this purpose. Figure 3cThe "multi-channel parameters" are shown, which may be, for example, the parameters 220 (ICC and ICLD). Then the covariance C y and C x Provided to covariance synthesis block 388 to synthesize a synthesized signal 388. In some examples, when blocks 384, 386 and 388 are implemented together, both parameter reconstruction 316 and mixing will be calculated 402 and synthesis processor 404 will be as discussed above and below.

[0410] 4Discussion

[0411] 4.1 Overview

[0412] The novel method of this example is particularly intended to encode and decode multi-channel content at a low bit rate (meaning equal to or lower than 160kbits / sec) while keeping the sound quality as close to the original signal as possible and preserving the spatial characteristics of the multi-channel signal. A function of the novel method is also to fit into the DirAC framework mentioned above. The output signal can be rendered on the same speaker setup as the input 212, or it can be rendered on a different speaker setup (which can be larger or smaller in terms of speakers). Similarly, the output signal can be rendered on speakers using binaural rendering.

[0413] The present section will provide an in-depth description of the present invention and the different modules that make up the present invention.

[0414] The proposed system consists of two main parts:

[0415] 1. Encoder 200, which derives necessary parameters 220 from input signal 212, quantizes them (at 222)

[0416] and encodes them (at 226). The encoder 200 may also compute a downmix signal 246 to be encoded in a bitstream 248 (and may be sent to the decoder 300).

[0417] 2. A decoder 300 which uses the encoded (eg transmitted) parameters and the downmix signal 246 to produce a multi-channel output with a quality as close as possible to the original signal 212 .

[0418] Figure 1 An overview of the novel approach proposed according to examples is shown. Note that some examples will use only a subset of the building blocks shown in the general figure and discard certain processing blocks depending on the application scenario.

[0419] The input 212(y) of the present invention is a multi-channel audio signal 212 (also referred to as a "multi-channel stream") (eg, signal 216) in the time domain or time-frequency domain, for example, a set of audio signals generated by a set of speakers or intended to be played.

[0420] The first part of the process is the encoding part; from the multi-channel audio signal, a so-called "downmix" signal 246 (see 4.2.6) is calculated together with a parameter set or side information 228 (see 4.2.2 and 4.2.3 ) which are derived from the input signal 212 in the time domain or frequency domain. These parameters will be encoded (see 4.2.5) and sent to the decoder 300 as appropriate.

[0421] The downmix signal 246 and the encoding parameters 228 may then be sent to a core encoder and a transmission canal that links the encoder and decoder sides of the process.

[0422] At the decoder side, the downmix signal is processed (4.3.3 and 4.3.4) and the transmitted parameters are decoded (see 4.3.2). The decoded parameters will be used for synthesis of the output signal using covariance synthesis (see 4.3.5), which will result in the final multi-channel output signal in the time domain.

[0423] Before going into detail, some general characteristics need to be established, at least one of which is valid:

[0424] - The described processing can be used with any loudspeaker setup. Keep in mind that when increasing the number of loudspeakers, the complexity of the described processing and the bits required to encode the transmitted parameters will also increase.

[0425] - The entire processing may be done on a frame basis, ie the input signal 212 may be divided into frames which are processed independently.

[0426] On the encoder side, each frame will generate a set of parameters, which will be transmitted to the decoder side to be processed.

[0427] - A frame can also be divided into time slots; these time slots then exhibit statistical properties that cannot be obtained at the frame scale. A frame can be divided into, for example, eight time slots, and the length of each time slot will be equal to 1 / 8 of the frame length.

[0428] 4.2 Encoder

[0429] The purpose of the encoder is to extract appropriate parameters 220 to describe the multi-channel signal 212, quantize them (at 222), encode them as side information 228 (at 226), and then send them to the decoder side as appropriate. Here, the parameters 220 and how to calculate them will be described in detail.

[0430] A more detailed scheme of the encoder 200 can be found in Figures 2a to 2d This overview highlights the two main outputs 228 and 246 of the encoder.

[0431] The first output of the encoder 200 is a downmix signal 228 computed from the multichannel audio input 212; the downmix signal 228 is a representation of the original multichannel stream (signal) on fewer channels than the original content (212). For more information on its computation, see Section 4.2.6.

[0432] The second output of the encoder 200 is the encoded parameters 220 represented as side information 228 in the bitstream 248; these parameters 220 are the key point of this example: they are the parameters that will be used to efficiently describe the multi-channel signal at the decoder side. These parameters 220 provide a good trade-off between quality and the number of bits required to encode them in the bitstream 248. On the encoder side, the parameter calculation can be done in several steps; the process will be described in the frequency domain, but it can also be done in the time domain. The parameters 220 are first estimated from the multi-channel input signal 212, then they are quantized at the quantizer 222, and then they can be converted into a digital bitstream 248 as side information 228. For more information on these steps, see Section 4.2.2, 4.2.3 and Section 4.2.5.

[0433] 4.2.1 Filter Banks and Partition Grouping

[0434] Filter banks are discussed with respect to either the encoder side (eg, filter bank 214) or the decoder side (eg, filter banks 320 and / or 338).

[0435] The present invention may use filter banks at various points during processing. These filter banks may convert signals from the time domain to the frequency domain (so-called aggregate bands or parameter bands), in which case they are called "analysis filter banks," or from the frequency domain to the time domain (e.g., 338), in which case they are called "synthesis filter banks."

[0436] The choice of filter bank must meet the required performance and optimization requirements, but the rest of the processing can be performed independently of the specific selected filter bank. For example, a filter bank based on orthogonal mirror filters or a filter bank based on short-time Fourier transform is used.

[0437] Reference Figure 5 , the output of the filter bank 214 of the encoder 200 will be a signal 216 in the frequency domain represented on a certain number of frequency bands (266 relative to 264). Performing the remaining processing for all frequency bands (264) can be understood to provide better quality and better frequency resolution, but a more significant bit rate is also required to transmit all information. Therefore, so-called "partition grouping" (265) is performed together with the filter bank processing, which corresponds to grouping certain frequencies together so as to represent the information 266 on smaller groups of frequency bands.

[0438] For example, the output 264 of the filter 263 ( Figure 5 ) can be represented over 128 frequency bands, and the partition grouping at 265 can result in signal 266 (216) having only 20 frequency bands. There are several ways to group the frequency bands together, and one interesting approach can be, for example, to try to approximate an equivalent rectangular bandwidth. The equivalent rectangular bandwidth is a psycho-acoustically motivated frequency band division that attempts to model how the human auditory system processes audio events, i.e., the goal is to group the filter banks in a way that is suitable for human hearing.

[0439] 4.2.2 Parameter Estimation (eg, Estimator 218)

[0440] Aspect 1: Using covariance matrices to describe and synthesize multi-channel content

[0441] The parameter estimation at 218 is one of the key points of the present invention; they are used at the decoder side to synthesize the output multi-channel audio signal. Those parameters 220 (encoded as side information 228) have been chosen because they efficiently describe the multi-channel input stream (signal) 212 and they do not require a lot of data to be transmitted. These parameters 220 are calculated at the encoder side and later used together with the synthesis engine at the decoder side to calculate the output signal.

[0442] Here, the covariance matrix can be calculated between the multi-channel audio signal and the channels of the downmix signal. That is:

[0443] -C y : covariance matrix of the multichannel stream (signal), and / or

[0444] -C x : Covariance matrix of the downmixed stream (signal) 246

[0445] The processing may be performed on a parameter band basis, so one parameter band is independent of another parameter band, and the formulas may be described for a given parameter band without loss of generality.

[0446] For a given parameter band, the covariance matrix is ​​defined as follows:

[0447]

[0448] in

[0449] - Represents the real part operator.

[0450] - Instead of the real part, it may be any other operation that produces a real value that is related to the complex value from which it is derived (eg the absolute value).

[0451] -* represents the conjugate transpose operator.

[0452] -B represents the relationship between the original multiple frequency bands and the grouped frequency bands (for partition grouping, see 4.2.1).

[0453] - Y and X are the original multi-channel signal 212 and the downmix signal 246 in the frequency domain, respectively.

[0454] C y (or its elements, or from C y or values ​​obtained from its elements) are also indicated as channel levels and related information of the original signal 212. x (or its elements, or from C y or values ​​obtained from its elements), is also indicated as covariance information associated with the downmix signal 212.

[0455] For a given frame (and frequency band), only one or two covariance matrices C y and / or C x , which may be output by the estimator block 218, for example. The process is slot-based rather than frame-based, and different implementations may be used with respect to the relationship between the matrix for a given slot and for the entire frame. As an example, the covariance matrix may be calculated for each slot within a frame and summed to output the matrix for a frame. Note that the definitions used to calculate the covariance matrix are mathematical definitions, but it is also feasible to calculate or at least modify those matrices in advance if it is desired to obtain an output signal with specific characteristics.

[0456] As mentioned above, the matrix C y and / or C x Not all elements of C need actually be encoded in the side information 228 of the bitstream 248. x , it is feasible to simply estimate it from the downmix signal 246 encoded by applying formula (1), and thus the encoder 200 can easily and simply avoid the need for C x (or more generally, the covariance information associated with the downmix signal). y(or for channel levels and related information associated with the original signal), C is estimated at the decoder side using the techniques discussed below y At least one of the elements is feasible.

[0457] Aspect 2a: Transmitting covariance matrices and / or energies to describe and reconstruct multichannel audio signals

[0458] As mentioned before, the covariance matrix is ​​used for synthesis. It is feasible to transfer those covariance matrices (or a subset of them) directly from the encoder to the decoder.

[0459] In some examples, the matrix C x It does not necessarily have to be transmitted, since the matrix can be calculated again on the decoder side using the downmix signal 246, but depending on the application scenario, this matrix may be required as a parameter to be sent.

[0460] From an implementation point of view, those matrices C x , C y Not all values ​​in M ​​...

[0461] Aspect 2b: Transmitting inter-channel coherence and inter-channel level differences to describe and reconstruct multi-channel signals

[0462] From the covariance matrix C x , C y A set of alternate parameters is defined and used at the decoder side to reconstruct the multi-channel signal 212. These parameters may be, for example, inter-channel coherence (ICC) and / or inter-channel level difference (ICLD).

[0463] Inter-channel coherence describes the coherence between each channel in a multi-channel stream. The parameters can be obtained from the covariance matrix C y is derived and is calculated as follows (for a given parameter band and for two given channels i and j):

[0464]

[0465] in

[0466] -ξ i,j The ICC between channels i and j of the input signal 212

[0467] - The values ​​in the covariance matrix of the multi-channel signal between channels i and j of the input signal 212, previously defined in equation (1), are

[0468] ICC values ​​can be calculated between each channel of a multi-channel signal, which can result in a large amount of data as the size of the multi-channel signal grows. In practice, a reduced set of ICCs can be encoded and / or sent. In some examples, the values ​​that are encoded and / or transmitted must be defined based on performance requirements.

[0469] As an example, when processing a signal generated by a 5.1 (or 5.0) speaker setup as defined by the ITU Recommendation "ITU-R BS.2159-4", it is feasible to choose to send only four ICCs. These four ICCs can be one of the following:

[0470] - Center and right channels

[0471] - Center and left channels

[0472] -Left and left bypass

[0473] -Right and right roundabout

[0474] Typically, the indices of the ICCs selected from the ICC matrix are described by an ICC map.

[0475] Typically, for each speaker setup, a fixed set of ICCs averaging the best quality may be selected to be encoded and / or transmitted to a decoder. The number of ICCs and which ICCs to be transmitted may depend on the speaker setup and / or the total bit rate available, and may be available at both the encoder and the decoder without transmitting the ICC map in the bitstream 248. In other words, a fixed set of ICCs and / or corresponding fixed ICC maps may be used, for example depending on the speaker setup and / or the total bit rate.

[0476] This fixed set may not be suitable for a particular material, and in some cases, using a fixed set of ICCs produces a quality that is significantly worse than the average quality of all material. To overcome this in another example, for each frame (or time slot), an optimal set of ICCs and a corresponding ICC map can be estimated based on the importance feature of a certain ICC. The ICC map for the current frame is then explicitly encoded and / or transmitted in the bitstream 248 together with the quantized ICC.

[0477] For example, similar to the decoder using equations (4) and (6) from 4.3.2, the downmix covariance C can be obtained by using the downmix covariance C from equation (1) x Generate covariance An estimate or ICC matrix The importance features of the ICC are determined by using an estimate of . Depending on the selected features, the features are calculated for each ICC or the corresponding entry in the covariance matrix, for each band, for which parameters will be sent in the current frame and combined for all bands. The combined feature matrix is ​​then used to determine the most important ICCs, thereby determining the ICC set to be used and the ICC map to be sent.

[0478] For example, the important feature of ICC is the estimated covariance The covariance C with the actual y The absolute error between the items of , and the combined feature matrix is ​​the sum of the absolute errors of each ICC to be transmitted on all frequency bands in the current frame. From the combined feature matrix, n items are selected where the summed absolute error is the highest, n being the number of ICCs to be transmitted for the speaker / bit rate combination, and an ICC map is constructed from these items.

[0479] In addition, in Figure 6b In another example shown, to avoid the ICC map changing too much between frames, the feature matrix can be emphasized for each entry in the selected ICC map of the previous parameter frame, for example, in the case of absolute error of covariance, by applying a coefficient > 1 (220k) to the entries of the ICC map of the previous frame.

[0480] Furthermore, in another example, a flag transmitted in the side information 228 of the bitstream 248 may indicate whether a fixed ICC map or an optimal ICC map is used in the current frame, and if the flag indicates a fixed group, the ICC map is not transmitted in the bitstream 248 .

[0481] The optimal ICC map is, for example, encoded and / or sent as a bitmap (eg, the ICC map may implement Figure 6a information 254').

[0482] Another example for transmitting an ICC map is to transmit an index into a table of all possible ICC maps, where the index itself is, for example, additionally entropy encoded. For example, the table of all possible ICC maps is not stored in memory, but the ICC map indicated by the index is directly calculated from the index.

[0483] A second parameter that may be transmitted together with ICC (or separately) is ICLD. "ICLD" stands for Inter-Channel Level Difference, and it describes the energy relationship between each channel of the input multi-channel signal 212. There is no unique definition of ICLD; the important aspect of this value is that it describes the energy ratio within a multi-channel stream.

[0484] As an example, from C y The conversion to ICLD can be obtained as follows:

[0485]

[0486] in:

[0487] -X i ICLD for channel i.

[0488] -P i The power of the current channel i can be obtained from C y The diagonal of: Extraction.

[0489] -P dmx,i depends on channel i, but will always be at C x is a linear combination of the values ​​in , which also depends on the original speaker setup.

[0490] In the example, P dmx,i is not the same for each channel, but depends on the mapping associated with the downmix matrix (also the prototype matrix for the decoder), which is usually mentioned in one of the points under equation (3). It depends on whether channel i is downmixed to only one of the downmix channels or to more than one of them. In other words, in the presence of non-zero elements in the downmix matrix, P dmx,i May be or include C x The sum of all diagonal elements of , so formula (3) can be rewritten as:

[0491]

[0492] Among them, α i is a weighting factor related to the expected energy contribution of the channel to the downmix, which is fixed for a specific input speaker configuration and is known at both the encoder and the decoder. The concept of the matrix Q will be provided below. α is also provided at the end of the document i and some values ​​of the matrix Q.

[0493] In the case of an implementation where a mapping is defined for each input channel i, where the mapping index is the downmixed channel j, input channel i is only mixed into it, or if the mapping index is greater than the number of downmixed channels. Thus, we have a mapping index m ICLD,i , which is used to determine P as follows dmx,i :

[0494]

[0495] 4.2.3 Parameter Quantization

[0496] To obtain the quantization parameter 224, an example of quantization of the parameter 220 may be obtained, for example, by Figure 2band 4 are performed by the parameter quantization module 222.

[0497] Once the parameter set 220 is calculated, it means that the covariance matrix {C x ,C y} or ICC and ICLD {ξ,χ}, which are quantized. The choice of quantizer may be a trade-off between quality and the amount of data to be transmitted, but there are no restrictions regarding the quantizer used.

[0498] As an example, in the case of using ICC and ICLD; a nonlinear quantizer including 10 quantization steps in the interval [-1, 1] for ICC and another nonlinear quantizer including 20 quantization steps in the interval [-30, 30] for ICLD may be provided.

[0499] Likewise, as an implementation optimization, it is possible to choose to downsample the parameters to be transmitted, meaning that the quantized parameters 224 are used by two or more frames in a row.

[0500] In one aspect, the subset of parameters transmitted in the current frame is signaled by a parameter frame index in the bitstream.

[0501] 4.2.4 Transient processing, downsampling parameters

[0502] Some of the examples discussed below can be understood, for example, Figure 5 It is shown in Figure 1 and 2d An example of block 214 of FIG.

[0503] In the case of downsampled parameter sets (e.g. Figure 5 265 in ), i.e., a parameter set 220 for a subset of parameter bands may be used for more than one processed frame, and transients that appear in more than one subset cannot be preserved in terms of localization and coherence. Therefore, it may be advantageous to send parameters for all bands in such a frame. This special type of parameter frame may be signaled, for example, by a flag in the bitstream.

[0504] In one aspect, transient detection at 258 is used to detect such transients in signal 212. The position of the transient in the current frame can also be detected. The temporal granularity can advantageously be linked to the temporal granularity of the filter bank 214 used, so that each transient position can correspond to a time slot or a group of time slots of the filter bank 214. The covariance matrix C is then selected based on the transient position. y and C x time slots, for example, only using the time slots from including the transient to the end of the current frame.

[0505] The transient detector (or transient analysis block 258) may be a transient detector also used for encoding of the downmix signal 212, for example, the time domain transient detector of the IVAS core encoder. Figure 5 The examples of may also be applied upstream of the downmix computation block 244 .

[0506] In one example, one bit is used to encode the occurrence of a transient (such as: "1", meaning "a transient is present in the frame", and "0", meaning "there is no transient in the frame"), and if a transient is detected, additionally the location of the transient is encoded and / or sent as an encoded field 261 (information about the transient) in the bitstream 248 to allow similar processing in the decoder 300.

[0507] If a transient is detected and transmission of all bands is performed (e.g., signaled), sending the parameters 220 using normal partition grouping may result in a spike in the data rate required to transmit the parameters 220 as side information 228 in the bitstream 248. In addition, time resolution is more important than frequency resolution. Therefore, at block 265, it may be advantageous to change the partition grouping for such a frame to have fewer bands to send (e.g., from many bands in the signal version 264 to fewer bands in the signal version 266). An example uses this different partition grouping, for example, by combining two adjacent bands on all bands into a normal downsampling factor of 2 for the parameters. In general, the occurrence of a transient implies that the covariance matrix itself can be expected to be very different before and after the transient. In order to avoid artifacts in the time slot before the transient, only the transient time slot itself and all subsequent time slots until the end of the frame can be considered. This is also based on the assumption that the signal is sufficiently stable beforehand and it is possible to use information and mixing rules that are derived for the previous frame and are also applicable to the time slot before the transient.

[0508] In summary, an encoder may be configured to determine in which time slot of a frame a transient has occurred, and encode channel levels and related information (220) of an original signal (212, y) associated with the time slot in which the transient has occurred and / or a subsequent time slot in the frame, without encoding channel levels and related information (220) of the original signal (212, y) associated with a time slot prior to the transient.

[0509] Similarly, when the presence and location of a transient in a frame is signaled (261), the decoder may (e.g., at block 380):

[0510] associating current channel levels and related information (220) with the time slot in which the transient has occurred and / or with subsequent time slots in the frame; and

[0511] The channel level and related information (220) of the time slot of the frame preceding the time slot where the transient has occurred are compared with the previous time slot.

[0512] Associated.

[0513] Another important aspect of transients is that if a transient is determined to exist in the current frame, no smoothing is performed on the current frame. y and C x Smoothing is performed, but C from the current frame yR and C x Used in the calculation of the mixing matrix.

[0514] 4.2.5 Entropy Coding

[0515] The entropy coding module (bitstream writer) 226 may be the last module of the encoder; its purpose is to convert the previously obtained quantized values ​​into a binary bit stream, which will also be called "side information".

[0516] The method used to encode the values ​​can be, for example, Huffman coding [6] or delta coding. The encoding method is not critical and will only affect the final bit rate. One should adapt the encoding method depending on the bit rate one wants to achieve.

[0517] Several implementation optimizations may be performed to reduce the size of the bitstream 248. As an example, a switching mechanism may be implemented that switches from one encoding scheme to another depending on which is more efficient from a bitstream size perspective.

[0518] For example, the parameters may be differentially encoded along the frequency axis of a frame, and the resulting sequence of incremental index entropies encoded by a range encoder.

[0519] Likewise, in the case of parameter downsampling, also as an example, a mechanism may be implemented to send only a subset of the parameter bands per frame in order to send data continuously.

[0520] Both examples require signaling bits to signal specific processing aspects to the decoder on the encoder side.

[0521] 4.2.6 Downmix Calculation

[0522] The downmix portion 244 of the processing may be simple, but is critical in some examples. The downmix used in the present invention may be a passive downmix, meaning that the way it is calculated remains the same during processing and is independent of the signal or its characteristics at a given time. However, it is understood that the downmix calculation at 244 may be extended to an active downmix calculation (such as described in [7]).

[0523] The downmix signal 246 can be calculated at two different locations:

[0524] - First, parameter estimation is performed on the encoder side (see 4.2.2 ), as it may be necessary (in some examples)

[0525] Calculate the covariance matrix C x .

[0526] A second time on the encoder side, between the encoder 200 and the decoder 300 (in the time domain), the downmix signal 246 is encoded and / or transmitted to the decoder 300 and is used as the basis for the synthesis at the module 334 .

[0527] As an example, for a stereo downmix of a 5.1 input, the downmix signal may be calculated as follows:

[0528] - The downmixed left channel is the sum of the left channel, left surround channel and center channel.

[0529] The right channel of the downmix is ​​the sum of the right channel, the surround channels and the center channel. Alternatively, in the case where the 5.1 input is a mono downmix, the downmix signal is calculated as the sum of each channel in the multi-channel stream.

[0530] In an example, each channel of the downmix signal 246 may be obtained as a linear combination of the channels of the original signal 212 , eg, with constant parameters, thereby achieving a passive downmix.

[0531] Depending on the processing needs, the calculation of the downmix signal can be extended and adapted to other loudspeaker setups.

[0532] Aspect 3: Low-latency processing using passive downmixing and low-latency filter banks

[0533] The present invention can provide low delay processing by using a passive downmix such as previously described for 5.1 input and a low delay filter bank. Using these two elements, it is possible to achieve a delay of less than 5 milliseconds between the encoder 200 and the decoder 300.

[0534] 4.3 Decoder

[0535] The purpose of the decoder is to synthesize an audio output signal (336, 340, y) on a given loudspeaker setup by using the encoded (eg transmitted) downmix signal (246, 324) and the encoded side information 228. R The decoder 300 may render the output audio signal (334, 240, y) on the same speaker setup as the speaker setup used for the input (212, y) or on a different speaker setup.R ). Without loss of generality, it will be assumed that the input and output speaker setups are the same (but in the example, they may be different). In this section, the different modules that may make up the decoder 300 will be described.

[0536] Figure 3a and 3b A detailed overview of possible decoder processing is depicted. It is important to note that, depending on the needs and requirements of a given application, Figure 3b At least some of the modules in (particularly the modules with dashed borders, such as 320, 330, 338) may be discarded. The decoder 300 may input (eg, receive) two sets of data from the encoder 200:

[0537] - Side information 228 with encoded parameters (as described in 4.2.2)

[0538] The downmix signal (246, y) may be in the time domain (eg 4.2.6 ).

[0539] The encoded parameters 228 may need to be decoded first (e.g., via input unit 312), for example, using the inverse encoding method previously used. Once this step is completed, the relevant parameters for synthesis, such as the covariance matrix, can be reconstructed. In parallel, the downmix signal (246, x) can be processed by several modules: first, the analysis filter bank 320 (see 4.2.1 ) to obtain a frequency domain version 324 of the downmix signal 246. Then, the prototype signal 328 can be calculated (see 4.3.3 ), and an additional decorrelation step (at 330) may be performed (see 4.3.4 The key point of the synthesis is the synthesis engine 334, which uses the covariance matrix (e.g., reconstructed at block 316) and the prototype signal (328 or 332) as input and produces the final signal 336 as output (see 4.3.5 Finally, the final step at the synthesis filter bank 338 may be completed (eg if the analysis filter bank 320 was previously used), resulting in an output signal 340 in the time domain.

[0540] 4.3.1 Entropy Decoding (e.g. Block 312)

[0541] The entropy decoding at block 312 (input interface) may allow obtaining the 4 The quantization parameter 314 obtained in the bitstream 248 can be understood as a straightforward operation; 4.2.5 The encoding method used in reads the bit stream 248 and then decodes it.

[0542] From an implementation point of view, the bitstream 248 may include signaling bits that are not data, but are to indicate certain peculiarities of processing at the encoder side.

[0543] For example, in case the encoder 200 has the possibility to switch between several encoding methods, the two first bits used may indicate which encoding method has been used.The following bits may also be used to describe which parameter bands are currently being transmitted.

[0544] Other information that may be encoded in the side information of the bitstream 248 may include a flag indicating a transient and a field 261 indicating in which time slot of the frame the transient has occurred.

[0545] 4.3.2 Parameter Reconstruction

[0546] Parameter reconstruction may be performed, for example, by block 316 and / or mixing rule calculator 402 .

[0547] The purpose of this parameter reconstruction is to reconstruct the covariance matrix C from the downmix signal 246 and / or from the side information 228 (or in its version represented by the quantization parameters 314). x and C y (or more generally, the covariance information associated with the downmix signal 246 and the level and related information of the original signal). These covariance matrices C x and C y They may be necessary for synthesis since they are matrices that efficiently describe the multi-channel signal 246 .

[0548] Parameter reconstruction at module 316 can be a two-step process:

[0549] First, the matrix C x (or more generally, the covariance information associated with the downmix signal 246) is recomputed from the downmix signal 246 (this step may be avoided in case the covariance information associated with the downmix signal 246 is actually encoded in the side information 228 of the bitstream 248); and

[0550] Then, the matrix C y (or more generally, the level and related information of the original signal 212) can be restored, for example, at least in part using the transmitted parameters and C x Or more generally covariance information associated with the downmix signal 246 (this step may be avoided in case the levels and related information of the original signal 212 are actually encoded in the side information 228 of the bitstream 248).

[0551] Note that in some examples, for each frame, it is feasible to use a linear combination of the reconstructed covariance matrices of previous frames with the current frame, such as by addition, averaging, etc., to smooth the covariance matrix C of the current frame. x For example, at the tth frame, the final covariance to be used in formula (4) can be considered as the target covariance reconstructed from the previous frame, e.g.

[0552] C x,t =C x,t +C x,t-1 .

[0553] However, in the case where it is determined that a transient exists in the current frame, no smoothing operation is performed on the current frame. In the case of a transient, no smoothing is performed using the current frame. x .

[0554] An overview of the process can be found below.

[0555] Note: As for the encoder, the processing here can be done independently for each band on a parameter band basis, for the sake of clarity, the processing will be described only for one specific band and the representation will be adapted accordingly.

[0556] Aspect 4a: Reconstructing parameters when the covariance matrix is ​​transmitted

[0557] For this aspect, it is assumed that the parameters encoded (e.g., transmitted) in the side information 228 (covariance matrix associated with the downmix signal 246 and channel levels and related information of the original signal 212) are the covariance matrix (or a subset thereof), as defined in aspect 2a. However, in some examples, the covariance matrix associated with the downmix signal 246 and / or the channel levels and related information of the original signal 212 may be implemented by other information.

[0558] If the complete covariance matrix C x and C y If only a subset of at least one of those matrices is encoded (e.g. transmitted), then no further processing is done at block 318 (so block 318 can be avoided in such an example). If only a subset of at least one of those matrices is encoded (e.g. transmitted), then the missing values ​​must be estimated. The final covariance matrix as used in the synthesis engine 334 (or more specifically in the synthesis processor 404) will consist of the encoded (e.g. transmitted) values ​​228 and the estimated values ​​at the decoder side. For example, if only the matrix C y Some elements of are encoded in the side information 228 of the bitstream 248, then C y The remaining elements of are estimated here.

[0559] The covariance matrix C of the downmix signal 246 isx , this is possible by using the downmix signal 246 at the decoder side to calculate the missing values ​​and applying equation (1).

[0560] In one aspect, where the occurrence and location of the transient is transmitted or encoded, the same time slot is used at the encoder side for calculating the covariance matrix C of the downmix signal 246 x .

[0561] For the covariance matrix C y , the missing values ​​can be calculated with a first estimate as follows:

[0562]

[0563] in:

[0564] - An estimate of the covariance matrix of the original signal 212 (this is an example of an estimated version of the original channel levels and related information)

[0565] -Q The so-called prototype matrix (prototype rule, estimation rule), which describes the relationship between the downmixed signal and the original signal (see 4.3.3 )(This is an example of a prototype rule)

[0566] -C x Covariance matrix of the downmix signal (this is an example of covariance information of the downmix signal 212)

[0567] -* indicates conjugate transpose

[0568] Once these steps are completed, the covariance matrix is ​​again obtained and can be used for the final synthesis.

[0569] Aspect 4b: Reconstruction parameters when ICC and ICLD are transmitted

[0570] For this aspect, it may be assumed that the encoded (eg transmitted) parameters in the side information 228 are ICC and ICLD (or a subset thereof) defined in aspect 2b.

[0571] In this case, it may be necessary to first recalculate the covariance matrix C x This can be done using the downmix signal 212 at the decoder side and applying equation (1).

[0572] In one aspect, the occurrence and location of the transients are transmitted as in the encoder using the same time slots for calculating the covariance matrix C of the downmix signal x Then, the covariance matrix C y can be recalculated from ICC and ICLD; this operation can be performed as follows:

[0573] The energy (also called level) of each channel of a multi-channel input can be obtained. These energies are derived using the transmitted inter-channel level differences and the following formula

[0574]

[0575] in

[0576]

[0577] where is a weighting factor related to the expected energy contribution of the channel to the downmix, which is fixed for some input speaker configuration and known at both the encoder and the decoder. In the case where one implementation defines a mapping for each input channel i, where the mapping index is the downmixed channel j, only input channel i is mixed into it, or if the mapping index is greater than the number of downmixed channels. Thus, we have a mapping index m ICLD,i It is used to determine P in the following way dmx,i :

[0578]

[0579] These symbols and 4.2.3 The same symbols are used in the parameter estimates in .

[0580] These energies can be used to normalize the estimated C y In case not all ICCs are transmitted from the encoder side, C can be calculated for the values ​​that are not transmitted. y The estimated covariance matrix is Formula (4) can be used to calculate the prototype matrix Q and the covariance matrix C x get.

[0581] This estimate of the covariance matrix leads to an estimate of the ICC matrix, for which the entry with index (i,j) can be given by:

[0582]

[0583] Therefore, the "reconstruction" matrix can be defined as follows:

[0584]

[0585] in:

[0586] - The subscript R indicates a reconstruction matrix (which is an example of a reconstructed version of the original levels and related information)

[0587] The ensemble {transmitted indices} corresponds to all (i, j) pairs that have been decoded in the side information 228 (eg transmitted from the encoder to the decoder).

[0588] In the example, since is not as good as the encoded value ξ i,j Exactly, so i,j Maybe Preferable.

[0589] Finally, from this reconstructed ICC matrix, the reconstructed covariance matrix can be inferred This matrix can be obtained by applying the energy obtained in formula (5) to the reconstructed ICC matrix, so for the index (i, j):

[0590]

[0591] In case the complete ICC matrix is ​​transmitted, only equations (5) and (8) are needed. The previous paragraphs describe one method to reconstruct the missing parameters, other methods may be used and the proposed method is not exclusive.

[0592] From the example of aspect 1b using a 5.1 signal, it can be noted that the values ​​that are not transmitted are the values ​​that need to be estimated at the decoder side.

[0593] Now we can get the covariance matrix C x and It is important to interpret the reconstruction matrix The covariance matrix C of the input signal 212 may be y The trade-off of the present invention may be to make the estimate of the covariance matrix at the decoder side close enough to the original, but also to transmit as few parameters as possible. These matrices may be necessary for the final synthesis described in 4.3.5.

[0594] Note that in some examples, for each frame, a linear combination of the reconstructed covariance matrix of the previous frame can be used to smooth the reconstructed covariance matrix of the current frame, such as by addition, averaging, etc. For example, at the tth frame, the final covariance to be used for synthesis can be considered to be the target covariance reconstructed for the previous frame, such as

[0595]

[0596] However, in the transient case, no smoothing is done and the C for the current frame yR Used in the calculation of the mixing matrix.

[0597] It should also be noted that in some examples, for each frame, the downmix channel C x The unsmoothed covariance matrix of is used for parameter reconstruction, while the smoothed covariance matrix C as in Section 4.2.3 x,t was used in the synthesis.

[0598] Figure 8a At the decoder 300, the covariance matrix C is restored x and Operation (e.g., performed at block 386 or 316 . . . ). Figure 8a In the block, the formula used by the specific block is also indicated between brackets. It can be seen that by formula (1), the covariance estimator 384 allows to reach the covariance C of the downmix signal 324 (or its downconverted version 385) x By using formula (4) and a rule Q of the appropriate type, the first covariance estimator block 384' allows arriving at the covariance C y First estimate Then, by applying equation (6), the covariance pair coherence block 390 obtains the coherence Then, the ICC replacement block 392 uses equation (7) to calculate the estimated and the ICC signaled in the side information 228 of the bitstream 348. The selected coherent ξ R Input to the energy application block 394, the energy application block 394 according to ICLD (χ i ) applies energy. Then, the target covariance matrix Provided to Figure 3a The mixer rule calculator 402 or the covariance synthesis block 388, or Figure 3c Mixer Rule Calculator or Figure 3b Synthesis Engine 344.

[0599] 4.3.3 Prototype Signal Calculation (Block 326)

[0600] The purpose of the prototype signal module 326 is to shape the downmix signal 212 (or its frequency domain version 324) in a way that it can be used by the synthesis engine 334 (see 4.3.5 ). The prototype signal module 326 may perform an upmix of the downmix signal. The prototype signal module 326 may perform the calculation of the prototype signal 328 by multiplying the downmix signal 212 (or 324) by the so-called prototype matrix Q:

[0601] Y p =XQ (9)

[0602] in

[0603] - Q is the prototype matrix (which is an example of a prototype rule)

[0604] -X is the downmix signal (212 or 324)

[0605] -Y p is the prototype signal (328).

[0606] The manner in which the prototype matrix is ​​built may be process dependent and may be defined to meet the requirements of the application. The only restriction may be that the number of channels of the prototype signal 328 must be the same as the number of channels of the desired output; this directly limits the size of the prototype matrix. For example, Q may be a matrix having the number of rows being the number of channels of the downmix signal (212, 324) and the number of columns being the number of channels of the final synthesized output signal (332, 340).

[0607] As an example, in the case of a 5.1 or 5.0 signal, the prototype matrix may be established as follows:

[0608]

[0609] Note that the prototype matrix may be predetermined and fixed. For example, Q may be the same for all frames, but may be different for different frequency bands. Furthermore, there are different Qs for different relationships between the number of channels of the downmix signal and the number of channels of the synthesized signal. For example, Q may be selected from a plurality of pre-stored Qs based on a particular number of downmix channels and a particular number of synthesized channels.

[0610] Aspect 5: Re-setting parameters when the output speaker settings are different from the input speaker settings:

[0611] One application of the proposed invention is to generate an output signal 336 or 340 on a loudspeaker setup that is different from the original signal 212 (for example meaning having a greater or lesser number of loudspeakers).

[0612] To do this, the prototype matrix must be modified accordingly. In this case, the prototype signal obtained by formula (9) will include multiple channels as the output speaker settings. For example, if we have a 5-channel signal as input (on the signal 212 side) and want to obtain a 7-channel signal as output (on the signal 336 side), the prototype signal will already include 7 channels.

[0613] In this way, the estimation of the covariance matrix in equation (4) still holds and will still be used to estimate the covariance parameters of the channels that do not exist in the input signal 212.

[0614] The parameters 228 transmitted between the encoder and the decoder are still relevant and formula (7) can still be used. More precisely, the encoded (e.g. transmitted) parameters must be assigned to channel pairs that are geometrically as close as possible to the original setting. Basically, an adaptation operation needs to be performed.

[0615] For example, if the ICC value between a speaker on the right and a speaker on the left is estimated on the encoder side, this value can be assigned to a channel pair of an output setup with the same left and right positions; in case of different geometries, this value can be assigned to a speaker pair whose position is as close as possible to the original speakers.

[0616] Then, once the target covariance matrix C for the new output setting is obtained y , the rest of the processing remains unchanged.

[0617] Therefore, in order to make the target covariance matrix Depending on the number of channels, the following is possible:

[0618] Using the prototype matrix Q, it converts from the number of downmix channels to the number of synthesis channels; this can be done by

[0619] Adapt formula (9) so that the prototype signal has the number of synthesized channels;

[0620] Adapt formula (4) to estimate the number of synthesized channels

[0621] Keeping formulas (5) to (8), it is possible to obtain the number of original channels;

[0622] However, groups of original channels (eg pairs of original channels) are assigned to a single synthesized channel (eg the assignment is selected based on the geometry) and vice versa.

[0623] exist Figure 8b An example is provided in Figure 8a A version of where the number of channels of some matrices and vectors are indicated. When the ICC (obtained from the side information 228 of the bitstream 348) is applied to the ICC matrix at 392, groups of original channels (e.g., pairs of original channels) are moved to a single synthesized channel (the allocation is chosen in terms of geometry) and vice versa.

[0624] Another possibility to generate a target covariance matrix for a number of output channels that is different from the number of input channels is to first generate a target covariance matrix for the number of input channels (e.g. the number of original channels of the input signal 212) and then adapt this first target covariance matrix to the number of synthesized channels, obtaining a second target covariance matrix corresponding to the number of output channels. This can be done by applying an upmix rule or a downmix rule, e.g. applying a matrix comprising factors for combinations of certain input (original) channels to the output channels to the first target covariance matrix Then in the second step this matrix is applied to the transmitted input channel power (ICLD) and obtains a channel power vector for the number of output (synthesized) channels and adjusts the first target covariance matrix according to the vector to obtain a second target covariance matrix with the desired number of synthesized channels. The adjusted second target covariance matrix can now be used in the synthesis. Figure 8c An example of this is provided in Figure 8c yes Figure 8a The operations of blocks 390 to 394 are performed to reconstruct the target covariance matrix to have a version of the original signal 212 with the number of original channels. After that, at block 395, the prototype signal QN (converted to the number of synthesized channels) and the vector ICLD can be applied. It is worth noting that Figure 8c The block 386 and Figure 8a The block 386 is the same as Figure 8c In , the number of channels for reconstructing the target covariance is exactly the same as the number of original channels of the input signal 212 (and in Figure 8a In , for generality, the reconstructed target covariance has the number of synthesized channels).

[0625] 4.3.4 Decorrelation

[0626] The purpose of the decorrelation module 330 is to reduce the number of correlations between each channel of the prototype signal. Highly correlated speaker signals may cause phantom sources and degrade the quality and spatial characteristics of the output multi-channel signal. This step is optional and may or may not be performed depending on the application requirements. In the present invention, decorrelation is used before the synthesis engine. As an example, an all-pass frequency decorrelator may be used.

[0627] Note about MPEG Surround:

[0628] In MPEG Surround according to the prior art, so-called "mixing matrices" are used (denoted M1 and M2 in the standard). The matrix M1 controls how the available downmix signal is input to the decorrelator. The M2 matrix describes how the direct signal and the decorrelated signal should be combined to produce the output signal.

[0629] Although it may be related to 4.3.3 The usage of the prototype matrix defined in and the decorrelator described in this section are similar, but it is important to note that:

[0630] - The function of the prototype matrix Q is completely different from the one used in MPEG Surround, the point of this matrix is ​​to generate the prototype signal. The purpose of the prototype signal is to be input to the synthesis engine.

[0631] - The prototype matrix is ​​not intended to prepare a downmix signal for a decorrelator and may be adapted depending on the requirements and the intended application. For example the prototype matrix may generate a prototype signal for an output loudspeaker setup that is larger than the input loudspeaker setup.

[0632] - In the proposed invention, the use of a decorrelator is not mandatory; the process relies on the use of the covariance matrix within the synthesis engine (see 5.1).

[0633] - The proposed invention does not generate an output signal by combining the direct signal and the decorrelated signal.

[0634] - The calculation of M1 and M2 is highly dependent on the tree structure, from a structural point of view, different coefficients of these matrices are used as appropriate. This is not the case in the proposed invention, the processing is independent of the downmix calculation (see 5.2 ), and conceptually, the proposed processing aims to consider the relationship between each channel, rather than just channel pairs, as can be done using a tree structure.

[0635] Therefore, the present invention is different from MPEG Surround according to the prior art.

[0636] 4.3.5 Synthesis Engine, Matrix Calculation

[0637] The final step of the decoder includes a synthesis engine 334 or synthesis processor 402 (and a synthesis filter bank 338 if necessary). The purpose of the synthesis engine 334 is to produce a final output signal 336 with respect to certain constraints. The synthesis engine 334 can calculate the output signal 336, and the characteristics of the output signal 336 are constrained by the input parameters. In the present invention, in addition to the prototype signal 328 (or 332), the input parameter 318 of the synthesis engine 338 is the covariance matrix C x and C y Since the characteristics of the output signal should be as close as possible to that of the yThe target covariance matrix is ​​defined as It is particularly known as the target covariance matrix (it will be shown that both estimated and pre-built versions of the target covariance matrix are discussed).

[0638] The synthesis engine 334 that can be used is not exclusive, as an example, prior art covariance synthesis can be used [8], which is incorporated herein by reference. Another synthesis engine 333 that can be used would be the synthesis engine described in the DirAC process of [2].

[0639] The output signal of synthesis engine 334 may require further processing through synthesis filter bank 338.

[0640] As a final result, an output multi-channel signal 340 is obtained in the time domain.

[0641] Aspect 6: High-quality output signals using “covariance synthesis”

[0642] As mentioned above, the synthesis engine 334 used is not unique and any engine using the transmitted parameters or a subset thereof may be used. However, an aspect of the invention may provide a high quality output signal 336, for example by using covariance synthesis [8].

[0643] This synthesis method aims to calculate the output signal 336, the characteristics of which are given by the covariance matrix To this end, the so-called optimal mixing matrices are calculated, which will mix the prototype signal 328 into the final output signal 336, from a mathematical point of view, given the target covariance matrix Provides the best results in the case of

[0644] The mixing matrix M is the relationship between y R =Mx P The prototype signal x P Transformed into output signal y R (336) matrix.

[0645] The mixing matrix can also be obtained by converting the relationship y R = Mx is the matrix that transforms the downmix signal x into the output signal. From this relationship, we can also infer

[0646] In the processing presented and C x and in some examples may be known (because they are the target covariance matrices of the downmix signal 246, respectively). and the covariance matrix C x ).

[0647] From a mathematical point of view, one solution is to Given, where K y and It is through C x and All matrices obtained by performing singular value decomposition. For P, it is an open parameter here, but the best solution (from the listener's perception point of view) can be found with respect to the constraints governed by the prototype matrix Q. The mathematical proof of this description can be found in [8].

[0648] The synthesis engine 334 provides a high quality output 336 because the method is designed to provide an optimal mathematical solution to the reconstruction of the output signal problem.

[0649] In less mathematical terms, it is important to understand that the covariance matrix represents the energy relationship between the different channels of the multi-channel audio signal. The matrix C for the original multi-channel signal 212 is y and the matrix C for downmixing the multi-channel signal 246 x Each value of these matrices reflects the energy relationship between two channels of a multi-channel stream.

[0650] Therefore, the philosophy behind covariance synthesis is to generate signals whose characteristics are determined by the target covariance matrix Drive. This matrix The way π is calculated is to describe the original input signal 212 (or the output signal we want to obtain in case it is different from the input signal). Then, with these elements, covariance synthesis will optimally mix the prototype signals to produce the final output signal.

[0651] On the other hand, the mixing matrix for the synthesis of the time slot is the mixing matrix M of the current frame and the mixing matrix M of the previous frame. p to ensure smooth synthesis, such as linear interpolation based on the time slot index within the current frame.

[0652] In another aspect, where the occurrence and location of a transient is transmitted, the previous mixing matrix M is converted to p for all time slots, and the mixing matrix M is used for the time slot including the transient position and all subsequent time slots in the current frame. Note that in some examples, for each frame or time slot, a linear combination with the mixing matrix for the previous frame or time slot can be used to smooth the mixing matrix of the current frame or time slot, such as by addition, averaging, etc. Let us assume that for the current frame t, the time slot s band i of the output signal is passed through Y s,i =M s,i X s,i Get, where M s,i is the mixing matrix M for the previous framet-1,i The combination of M t,i is the mixing matrix calculated for the current frame, e.g. linearly interpolating between them:

[0653]

[0654] Among them, n s is the number of time slots in a frame (e.g., 16), and t-1 and t indicate the previous frame and the current frame. More generally, the mixing matrix M as calculated for the current frame is scaled by increasing the coefficients along the subsequent time slots of the current frame t. t,i , and by adding the mixing matrix M scaled by the reduced coefficients along the subsequent time slots of the current frame t t-1,i The mixing matrix M associated with each time slot can be obtained s,i The coefficients may be linear.

[0655] It can be provided that in the case of a transient (e.g. signaled in information 261) the current mixing matrix and the past mixing matrix are not combined, but rather the previous ones up to the time slot including the transient and the current ones for the time slot including the transient and all subsequent time slots until the end of the frame.

[0656]

[0657] where s is the time slot index, i is the frequency band index, t and t-1 indicate the current frame and the previous frame, and s t is the time slot including transients.

[0658] Differences from prior art documents[8]

[0659] It is also important to note that the proposed invention goes beyond the scope of the approach proposed in [8]. Notable differences are in particular:

[0660] - Target covariance matrix is calculated on the encoder side of the proposed process.

[0661] - Target covariance matrix It can also be calculated in a different way (in the proposed invention, the covariance matrix is ​​not a sum of diffuse parts).

[0662] - Processing is not done for each frequency band individually, but for parameter bands grouped together (as in 0 ).

[0663] - From a more global perspective: Covariance synthesis is here only one block of the whole process and has to be used together with all other elements on the decoder side.

[0664] 4.3 As a preferred aspect of the list

[0665] The present invention may be characterized by at least one of the following aspects:

[0666] 1. On the encoder side

[0667] a. Input multi-channel audio signal 246.

[0668] b. Convert signal 212 from the time domain to the frequency domain using filter bank 214 (216)

[0669] c. Calculate the downmix signal 246 at block 244

[0670] d. From the original signal 212 and / or the downmix signal 246, estimate a first set of parameters to describe the multi-channel stream (signal)

[0671] 246: Covariance matrix C x and / or C y

[0672] e. Transmission and / or coding covariance matrix C x and / or C y Directly or compute the ICC and / or ICLD and transmit them

[0673] f. Encoding the transmitted parameters 228 in the bitstream 248 using an appropriate encoding scheme

[0674] g. Calculating the downmix signal in the time domain 246

[0675] h. Transmitting side information (ie, parameters) and downmix signal in the time domain 246

[0676] 2. On the decoder side

[0677] a. Decoding the bitstream 248 including the side information 228 and the downmix signal 246

[0678] b. (optional) applying the filter bank 320 to the downmix signal 246 to obtain a version 324 of the downmix signal 246 in the frequency domain

[0679] c. Reconstruct the covariance matrix C from the previously decoded parameters 228 and the downmix signal 246 x and

[0680] d. Calculating the prototype signal 328 (324) from the downmix signal 246

[0681] e. (Optional) Decorrelate the prototype signal (at block 330)

[0682] f. Use as the refactored C x and Applying Synthesis Engine 334 to the Prototype Signal

[0683] g. (Optional) Applying the synthesis filter bank 338 to the output 336 of the covariance synthesis 334

[0684] h. Obtain output multi-channel signal 340

[0685] 4.5 Covariance Synthesis

[0686] In this section, the discussion can be Figures 1 to 3c However, these techniques can also be implemented independently: for example, in some examples, it is not necessary to Figures 8a to 8c and the covariance calculations implemented in equations (1) to (8). Therefore, in some examples, when referring to (Reconstructing the target covariance) can also be done by C y (which can also be provided directly, without refactoring.) Nevertheless, the techniques of this section can be used advantageously with the techniques described above.

[0687] Reference now Figures 4a to 4d Here, examples of covariance synthesis blocks 388a to 388d are discussed. Blocks 388 to 388d may be implemented as, for example Figure 3c Block 388a to 388d may be, for example, Figure 3a The synthesis processor 404 and the mixing rule calculator 402 of the synthesis engine 334 and / or a portion of the synthesis processor 404 and the mixing rule calculator 402 of the parameter reconstruction block 316. Figures 4a to 4d In , the downmix signal 324 is in the frequency domain FD (i.e., downstream of the filter bank 320) and is indicated by X, while the synthesized signal 336 is also in FD and is indicated by Y, however, it is possible to summarize these results in the time domain. Note that Figures 4a to 4d Each of the covariance synthesis blocks 388a to 388d can be referred to as a single frequency band (eg, decomposed once in 380), and the covariance matrix C x and (or other reconstructed information) can thus be associated with a specific frequency band. For example, the covariance synthesis can be performed in a frame-by-frame manner, and in that case the covariance matrix C x and (or other reconstructed information) is associated with a single frame (or with multiple consecutive frames): thus, covariance synthesis can be performed in a frame-by-frame manner or in a multiple-frame-by-multiple-frame manner.

[0688] exist Figure 4a In the example, the covariance synthesis block 388a may be composed of an energy compensated optimal mixing block 600a and a missing correlator block. Basically, a single mixing matrix M is found and the only significant operation additionally performed is the calculation of the energy compensated mixing matrix M'.

[0689] Figure 4b A covariance synthesis block 388b inspired by [8] is shown. The covariance synthesis block 388b may allow obtaining the synthesized signal 336 as a synthesized signal having a first principal component 336M and a second residual component 336R. Although the principal component 336M may be obtained at the optimal principal component mixing matrix 600b, for example by x and Find the mixing matrix M in M , and no decorrelator is used, but the residual component 336R can be obtained in another way. R In principle, the relationship Usually, the obtained mixing matrix does not fully meet the requirements and can be used Find the residual target covariance. It can be seen that the downmix signal 324 can be derived on the path 610b (path 610b can be referred to as the second path, the second path is parallel to the first path 610b', the first path 610b' includes block 600b). The prototype version 613b of the downmix signal 324 (with Y pR (represented) can be obtained at the prototype signal block (upmix block) 612b. For example, a formula such as formula (9) can be used, that is,

[0690] Y pR =XQ

[0691] Examples of Q (prototype matrix or upmix matrix) are provided in this document. Downstream of block 612b, a decorrelator 614b is provided so that the prototype signal 613b is decorrelated to obtain a decorrelated signal 615b (also referred to as Indicated). At block 616b, from the decorrelated signal 615b, the decorrelated signal is estimated. The covariance matrix of By using C as the principal component mixture x The equivalent value of is the decorrelated signal The covariance matrix of and C as the target covariance in another optimal mixing block r , the residual component 336R of the synthesized signal 336 can be obtained at the optimal residual component mixing matrix block 618b. The optimal residual component mixing matrix block 618b can be implemented in such a way that the mixing matrix M is generated. R, to mix the decorrelated signal 615b and obtain a residual component 336R (for a particular frequency band) of the synthesized signal 336. At adder block 620b, the residual component 336R is added to the main component 336M (so paths 610b and 610b' are connected together at adder block 620b).

[0692] Figure 4c Show alternative Figure 4b 388c of the covariance synthesis 388b. The covariance synthesis block 388c allows obtaining the synthesized signal 336 as a signal Y having a first principal component 336M' and a second residual component 336R'. Although the principal component 336M' can be obtained at the optimal principal component mixing matrix 600c, for example by x and (or C y Other information 220) find the mixing matrix M M , and no correlator is used, but the residual component 336R' can be obtained in another way. The downmix signal 324 can be derived to the path 610c (path 610c can be called the second path, the second path is parallel to the first path 610c', and the first path 610c' includes block 600c). By applying the prototype matrix Q (for example, a matrix that upmixes the downmix signal 234 to the version 613c of the downmix signal 234 with the number of channels, i.e., the number of synthesized channels), the prototype version 613c of the downmix signal 324 can be obtained at the downmix block (upmix block) 612c. For example, a formula such as formula (9) can be used. This document provides an example of Q. Downstream of block 612c, a decorrelator 614c can be provided. In some examples, the first path does not have a decorrelator, while the second path has a decorrelator.

[0693] Decorrelator 614c can provide a decorrelated signal 615c (also referred to as instructions). However, unlike Figure 4b The covariance synthesis block 388b is the opposite of the technique used in Figure 4c The covariance synthesis block 388c does not decorrelate the signal 615c from Estimate the covariance matrix of the decorrelated signal 615c In contrast, the covariance matrix of the decorrelated signal 615c is is obtained (at block 616c) from:

[0694] The covariance matrix C of the downmix signal 324 is x (For example, in Figure 3c at block 384 and / or estimated using equation (1); and

[0695] Prototype matrix Q.

[0696] By using the covariance matrix C from the downmix signal 324 x The estimated covariance matrix C as the principal component mixing matrix x and C r As an equivalent of the target covariance matrix, the residual component 336R' of the synthesized signal 336 is obtained at the optimal residual component mixing matrix block 618c. The optimal residual component mixing matrix block 618c can be used to generate the residual component mixing matrix M R is implemented by mixing the matrix M according to the residual components R The decorrelated signal 615c is mixed to obtain the residual component 336R'. At adder block 620c, the residual component 336R' is added to the principal component 336M' to obtain the composite signal 336 (paths 610c and 610c' are thus coupled together at adder block 620c).

[0697] In some examples, the residual component 336R or 336R' is not always or need not be calculated (and the path 610b or 610c is not always used). In some examples, although covariance synthesis is performed for some frequency bands without calculating the residual signal 336R or 336R', for other frequency bands of the same frame, the residual signal 336R or 336R' is also considered to process the covariance synthesis. Figure 4d An example of a covariance synthesis block 388d is shown, which may be a specific case of a covariance synthesis block 388b or 388c: Here, a band selector 630 may select or deselect (in a manner represented by a switch 631) the calculation of a residual signal 336R or 336R'. For example, a path 610b or 610c may be selectively enabled by the selector 630 for certain frequency bands and disabled for other frequency bands. In particular, the path 610b or 610c may be disabled for frequency bands exceeding a predetermined threshold (e.g., a fixed threshold), and the predetermined threshold (e.g., a maximum value) may be a frequency band to which the human ear is insensitive to phase (frequency bands above the threshold) and a frequency band to which the human ear is sensitive to phase (frequency bands below the threshold), so that the residual component 336R or 336R' is not calculated for the frequency bands below the threshold, and the residual component 336R or 336R' is calculated for the frequency bands above the threshold.

[0698] Figure 4d The example can also be used Figure 4a The block 600a replaces the block 600b or 600c, and uses Figure 4b The covariance synthesis block 388b or Figure 4c The covariance synthesis block 388c is obtained by replacing block 610b or 610c.

[0699] Here some indication is provided on how to obtain the mixing rule (matrix) at blocks 338, 402 (or 404), 600a, 600b, 600c, etc. As mentioned above, there are many ways to obtain the mixing matrix, but some of them will be discussed in more detail here.

[0700] In particular, first, refer to Figure 4b At the optimal principal component mixing matrix block 600c, for example, the mixing matrix M of the principal component 336M of the synthesized signal 336 can be obtained from the following formula:

[0701] The covariance matrix C of the original signal 212 is y (C y can be estimated using at least some of the equations (6) to (8) discussed above, see for example FIG8 ; it can be of the form of a so-called “target version” For example, the value estimated according to formula (8)); and

[0702] The covariance matrix C of the downmixed signal 246, 324 x (C y It can be estimated using, for example, formula (1).

[0703] For example, as proposed in [8], it is admitted that the covariance matrix C is decomposed according to the following factorization x and C y , which are Hermitian and positive semidefinite matrices:

[0704]

[0705] K x and K y For example, by x and C y Obtained by applying two singular value decompositions (SVD). For example:

[0706] C x The SVD of can provide a matrix U of singular vectors (e.g., left singular vectors) Cx ;and

[0707] The diagonal matrix S of singular values Cx ;

[0708] Therefore, K x By using U Cx It is obtained by multiplying by a diagonal matrix, which has S in its entries Cx The square root of the value in the corresponding term of .

[0709] In addition, regarding C y The SVD can provide:

[0710] The matrix V of singular vectors (e.g. right singular vectors) Cy ;and

[0711] The diagonal matrix S of singular values Cy

[0712] Therefore, K y By using U C y is obtained by multiplying the diagonal matrix, which has S in its entries C The square root of the value in the corresponding entry of y.

[0713] It is then possible to obtain a principal component mixing matrix M_M which, when applied to the downmix signal 324, will allow obtaining the principal components 336M of the synthesized signal 336. M can be obtained as follows:

[0714]

[0715] If K x is a non-invertible matrix, the regularized inverse matrix can be obtained using known techniques and replaced instead of

[0716] The parameter P is usually free, but it can be optimized. To find P, SVD can be applied to:

[0717] C x (covariance matrix of the downmix signal 324); and

[0718] (Covariance matrix of prototype signal 613b).

[0719] Once SVD is performed, it is possible to obtain P as

[0720] P=VΛU*

[0721] Λ is a matrix with the same number of rows as the number of synth channels and the same number of columns as the number of downmix channels. Λ is identified in its first square block and is completed with zeros in the remaining entries. Now explain how V and U are derived from C x and V and U are matrices of singular vectors obtained from SVD:

[0722]

[0723] S is a diagonal matrix of singular values ​​obtained typically by SVD. is a diagonal matrix that transforms the prototype signal The energy of each channel is normalized to the energy of the synthetic signal y. First, we need to calculate Prototype Signal Then, in order to get the covariance matrix of Conclusion Will The diagonal values ​​of C are normalized to y The corresponding diagonal value of An example is The diagonal terms of are calculated as in It is C y The values ​​of the diagonal terms of yes The values ​​of the diagonal items.

[0724] Once obtained The covariance matrix C of the residual components r Can be obtained from

[0725]

[0726] Once you get C r , it is possible to obtain a mixing matrix for mixing the decorrelated signal 615b to obtain the residual signal 336R, where in the same optimal mixing C r With the best mix with the main Same effect, decorrelate prototypes The covariance of is the input signal covariance C x With master optimal mix.

[0727] However, it is understood that Figure 4b Compared with the technology Figure 4c The technique has some advantages. In some examples, Figure 4c Technology and Figure 4c The technique is the same as that used to compute the principal matrix and to generate the principal components of the synthesized signal. Figure 4c Technology and Figure 4b The difference between the techniques lies in the computation of the residual mixing matrix and, more generally, the residual component used to generate the synthesized signal. Fig.11 Combination Figure 4c Used to calculate the residual mixing matrix. Figure 4c In the example of , a decorrelator 614c in the frequency domain is used, which ensures decorrelation of the prototype signal 613c but preserves the energy of the prototype signal 613b itself.

[0728] In addition, Figure 4cIn the example of , we can assume (at least by approximation) that the decorrelated channels of the decorrelated signal 615c are mutually inharmonic, so all non-diagonal elements of the covariance matrix of the decorrelated signal are zero. With these two assumptions, we can simply pass in C x Q is applied to estimate the covariance of the decorrelated prototype, and only the main diagonal of the covariance (i.e., the energy of the prototype signal) is used. Starting with the decorrelated signal 615b, Figure 4c Technology is better than Figure 4b It is more efficient to estimate the example where we need to make the same estimate as we have already done for C x The same frequency band / time slot aggregation is performed. Figure 4c In the example above, we can simply apply the already aggregated C x Therefore, the same mixing matrix is ​​calculated for all bands of the same aggregated band group.

[0729] Therefore, the covariance 711 of the decorrelated signal can be estimated at 710 using

[0730] P decorr =diag(QC x Q * )

[0731] As the main diagonal of a matrix with all off-diagonal elements set to zero, it is used as the input signal covariance In the example C x Smoothed to perform synthesis of the principal component 336M' of the synthesized signal, the technique may be used according to C x Used to calculate P decorr C is a non-smooth x .

[0732] Now, the prototype matrix QR should be used. However, it has to be noted that for the residual signal, QR is the identity matrix. The properties of (diagonal matrix) and QR (identity matrix) can further simplify the calculation of the mixing matrix (at least one SVD can be omitted), see the following techniques and Matlab list.

[0733] First, something like Figure 4b For example, the residual target covariance matrix C of the input signal 212 r (Hermitian, positive semidefinite) can be decomposed into The matrix K can be obtained by SVD (702) r : SVD 702 is used for C_r generation:

[0734] The matrix U of singular vectors (e.g. left singular vectors) Cr ;

[0735] The diagonal matrix S of singular values Cr ;

[0736] Therefore K r By replacing U in the diagonal matrix Cr Multiplication by a diagonal matrix is ​​obtained (at 706) which has in its entries the values ​​of Cr The square root of the value in the corresponding element of (the latter has been obtained at 704).

[0737] At this point, in theory, one could apply another SVD, this time to the covariance of the decorrelated prototypes

[0738] However, in this example ( Figure 4c ), in order to reduce the amount of calculation, different paths have been selected. decort =diag(QC x Q * ) Estimated is a diagonal matrix, so no SVD is needed (the SVD of a diagonal matrix gives the singular values ​​as the ordered vectors of the diagonal elements, while the left and right singular vectors only indicate the ordered indices). By calculating (at 712) Take the square root of each value on the diagonal of , and get the diagonal matrix Diagonal Matrix is to make The advantage is to obtain No SVD is needed. From the decorrelated signal The estimated covariance matrix of the decorrelated signal 615c is calculated by But since the prototype matrix is ​​Q r (i.e., isomorphic matrices), so we can directly use To Formulated as in It is C r The values ​​of the diagonal terms of yes The values ​​of the diagonal items. is a diagonal matrix (obtained at 722) that decorrelates the signal The per-channel energy of (615b) is normalized to the expected energy of the synthesized signal y.

[0739] At this point, it is possible (at 734) to Multiply (also called the result 735 of multiplication 734). Then (736), K rMultiply Get K′ y (Right now ). From K′ y SVD (738) can be performed to obtain a left singular vector matrix U and a right singular vector matrix V. By multiplying V and U* (740), the matrix P (P = VU H ). Finally (742), the mixing matrix M of the residual signal can be obtained by applying the following R :

[0740]

[0741] in (obtained at 745) can be replaced by the inverse. R It can therefore be used at block 618c to perform residual mixing.

[0742] Here is the Matlab code for performing the covariance synthesis as described above. Note that the asterisks (*) in the code represent multiplications, and the vertices (') represent Hermitian matrices.

[0743] % Calculate the residual mixing matrix

[0744] function[M]=

[0745] ComputeMixingMatrixResidual(C_hat_y,Cr,reg_sx,reg_ghat)

[0746] EPS_=single(1e-15); % Epsilon to avoid zero division

[0747] num_outputs = size(Cr, 1);

[0748] Decomposition of %Cy

[0749] [U_Cr, S_Cr] = svd(Cr);

[0750] Kr = U_Cr*sqrt(S_Cr);

[0751] % The singular value decomposition of a diagonal matrix is ​​the sorted diagonal elements,

[0752] % We can skip the sorting and get Kx directly from Cx

[0753] K_hat_y=sqrt(diag(C_haty));

[0754] limit=max(K_hat_y)*reg_sx+EPS_;

[0755] S_hat_y_reg_diag=max(K_hat_y,limit);

[0756] %Formulate the regularized Kx

[0757] K_hat_y_reg_inverse=1. / S_hat_y_reg_diag;

[0758] %Formulate the standardized matrix G_hat

[0759] %Q is the identity matrix in case of the residual / diffuse part so

[0760] %Q*Cx*Q′=Cx

[0761] Cy_hat_diag=diag(C_hat_y);

[0762] limit=max(Cy_hat_diag)*reg_ghat+EPS_;

[0763] Cy_hat_diag=max(Cy_hat_diag,limit);

[0764] G_hat=sqrt(diag(Cr). / Cy_hat_diag);

[0765] %Formulation Optimum P

[0766] %Kx, G_hat are diagonal matrices, Q is I...

[0767] K_hat_y=K_hat_y.*G_hat;

[0768] for k=1:num_outputs

[0769] Ky_dash(k,:)=Kr(k,:)*K_hat_y(k);

[0770] end

[0771] [U,~,V]=svd(Ky_dash);

[0772] P = V * U';

[0773] %Formulation M

[0774] M = Kr*P;

[0775] for k=1:num_outputs

[0776] M(:,k)=M(:,k)*K_hat_y_reg_inverse(k);

[0777] end

[0778] end

[0779] Here we provide information about Figure 4b and 4c In some examples, two synthesis methods can be considered for each frequency band. For some frequency bands, a frequency band higher than a certain frequency to which the human ear is insensitive to phase is usually used, including from Figure 4b The remaining paths are fully synthesized to achieve the required energy for applying energy compensation in the vocal channel.

[0780] Therefore, also in Figure 4b In the example of , for bands below a certain (fixed, decoder-known) band boundary (threshold), the following can be performed Figure 4b Complete synthesis (e.g. Figure 4d In the case of Figure 4b In the example of is derived from the decorrelated signal 615b itself. Figure 4c In the example of , a decorrelator 614c in the frequency domain is used, which ensures decorrelation of the prototype signal 613c but preserves the energy of the prototype signal 613b itself.

[0781] Further considerations:

[0782] ·exist Figure 4b and 4c In both examples: at the first path (610b', 610c'), by relying on the covariance C of the original signal 212 y and the covariance C of the downmix signal 324 x To generate the mixing matrix M M (at blocks 600b, 600c);

[0783] ·exist Figure 4b and 4c In both examples: at the second path (610b, 610c), there are decorrelators (614b, 614c), and a mixing matrix M is generated R (at blocks 618b, 618c), which should take into account the covariance of the decorrelated signals (616b, 616c) but

[0784] In Figure 4bIn the example of FIG. 1 , the covariance of the decorrelated signal (616b, 616c) is intuitively calculated using the decorrelated signal (616b, 616c) as and is weighted by the energy of the original channel y.

[0785] In Figure 4c In the example, by x The covariance of the decorrelated signal (616b, 616c) is estimated and back-calculated in an intuitive manner and weighted in the energy of the original channel y.

[0786] Note that the covariance matrix may be the reconstruction target matrix discussed above (e.g., obtained from the channel levels and related information 220 written in the side information 228 of the bitstream 248), and may therefore be considered to be associated with the covariance of the original signal 212. In any case, as it will be used for the synthesized signal 336, the covariance matrix It can also be considered as the covariance associated with the synthesized signal. The same applies to the residual covariance matrix C r , which can also be understood as the residual covariance matrix (C r ), and the principal covariance matrix can also be understood as the principal covariance matrix associated with the synthetic signal.

[0787] 5. Advantages

[0788] 5.1 Reduce the use of decorrelation and optimize the use of the synthesis engine

[0789] Given the proposed techniques, and the parameters used for processing and the way these parameters are combined with the synthesis engine 334, it is shown that the need for strong decorrelation of the audio signal (for example in its version 328) is reduced, and even in the absence of a decorrelation module 330, the effects of decorrelation (such as artifacts or degradation of spatial characteristics or degradation of signal quality) can be reduced if not removed.

[0790] More precisely, as previously mentioned, the decorrelation portion 330 of the process is optional. In practice, the synthesis engine 334 performs the de-correlation by using the target covariance matrix C y (or a subset thereof) to decorrelate signal 328 and ensure that the channels making up output signal 336 are properly decorrelated among themselves. y The values ​​in the covariance matrix represent the energy relations between the different channels of our multi-channel audio signal, which is why it is used as the target for synthesis.

[0791] Furthermore, the encoded (eg, transmitted) parameters 228 (eg, in their versions 314 or 318) combined with the synthesis engine 334 can ensure a high quality output 336, given the fact that the synthesis engine 334 uses the target covariance matrix C y , so that the reproduced output multi-channel signal 336, the spatial characteristics and sound quality of the output multi-channel signal 336 are as close to the input signal 212 as possible.

[0792] 5.2 Downmix Agnostic Processing

[0793] Given the proposed technique, and the way the prototype signals 328 are calculated and how they are used with the synthesis engine 334, it is explained here that the proposed decoder is independent of the way the downmix signal 212 is calculated at the encoder.

[0794] This means that the proposed invention can be performed at the decoder 300 independently of the way the downmix signal 246 is calculated at the encoder and the output quality of the signal 336 (or 340) is not dependent on a specific downmix method.

[0795] 5.3 Scalability of parameters

[0796] Given the proposed technique, and the way the parameters (28, 314, 318) are calculated and the way they are used with the synthesis engine 334, and the way they are estimated on the decoder side, it is demonstrated that the parameters used to describe the multi-channel audio signal are scalable in both number and usage.

[0797] Typically, only a subset of parameters (e.g., C y and / or C x A subset of, for example, its elements) is coded (for example transmitted): this allows reducing the bit rate used by the process. Thus, given the fact that the non-transmitted parameters are reconstructed at the decoder side, the coded (for example transmitted) parameters (for example C y and / or C x The number of elements of may be scalable. This gives the opportunity to scale the entire process in terms of output quality and bit rate, the more parameters transmitted, the better the output quality and vice versa.

[0798] Moreover, those parameters (such as C y and / or C x The parameters (or elements thereof) are scalable in purpose, which means that they can be controlled by user input to modify the characteristics of the output multi-channel signal. Furthermore, those parameters can be calculated for each frequency band and thus allow a scalable frequency resolution.

[0799] For example, it may be decided to cancel a loudspeaker in the output signal (336, 340), so that parameters can be manipulated directly on the decoder side to achieve such a change.

[0800] 5.4 Flexibility of output settings

[0801] Given the proposed technique, and the synthesis engine 334 and parameters used (e.g., C y and / or C x or elements thereof), it is explained here that the proposed invention allows a wide range of rendering possibilities involving output settings.

[0802] More precisely, the output setup does not have to be identical to the input setup. It is feasible to manipulate the reconstructed target covariance matrix fed into the synthesis engine to produce an output signal 340 on a speaker setup that is larger or smaller or has only a geometry different from the original speaker setup. This is possible because the parameters to be transmitted and the proposed system are independent of the downmix signal (see 5.2).

[0803] For these reasons it is flexible to interpret the proposed invention from the point of view of the output loudspeaker setup.

[0804] 5. Some examples of prototype matrices

[0805] Here, the following table has been for 5.1, but LFE is excluded, henceforth we also include LFE in the processing (only one ICC for the relation LFE / C and ICLD for LFE are sent only in the lowest parameter band and all other bands are set to 1 and 0 respectively for the synthesis at the decoder side). The channel naming and order follows the CICP in ISO / IEC 23091-3 "Information technology – Coding independent code points – Part 3: Audio", Q is always used as the prototype matrix in the decoder and the downmix matrix in the encoder. 5.1 (CICP6). α i To be used to calculate ICLD.

[0806]

[0807] α i =[0.4444 0.4444 0.2 0.2 0.4444 0.4444]

[0808] 7.1(CICP12)

[0809]

[0810] α i=[0.2857 0.2857 0.5714 0.5714 0.2857 0.2857 0.2857 0.2857]

[0811] 5.1+4(CICP16)

[0812]

[0813] α i = [0.1818 0.1818 0.3636 0.3636 0.1818 0.1818 0.1818 0.1818 0.1818 0.1818 0.1818]

[0815] 7.1+4(CICP19)

[0816]

[0817] α i = [0.1538 0.1538 0.3077 0.3077 0.1538 0.1538 0.1538 0.1538 0.1538 0.1538

[0819] 6. Methods

[0820] Although the above technology is mainly discussed as components or functional devices, the present invention can also be implemented as a method. The blocks and elements discussed above can also be understood as steps and / or stages of the method.

[0821] For example, a decoding method for generating a synthetic signal from a downmix signal is provided, wherein the synthetic signal has a plurality of synthetic channels, the method comprising:

[0822] A downmix signal (246, x) is received, the downmix signal (246, x) having a plurality of downmix channels, and side information

[0823] (228), the side information (228) includes:

[0824] Channel levels and related information (220) of an original signal (212, y), the original signal (212, y)

[0825] With multiple original channels;

[0826] Using the channel levels and related information (220) of the original signal (212, y) and the signal

[0827] (246, x) associated with the covariance information (C x) to generate the synthetic signal.

[0828] The decoding method may include at least one of the following steps:

[0829] calculating a prototype signal from the downmix signal (246, x), the prototype signal having the number of synthesized channels;

[0830] Using the channel levels and related information of the original signal (212, y) and the downmix signal (246,

[0831] x) associated covariance information to calculate the mixing rule; and

[0832] The synthesized signal is generated using the prototype signal and the mixing rule.

[0833] A decoding method is also provided for generating a synthesized signal (336) from a downmix signal (324, x) having a plurality of downmix channels, the downmix signal (336) having a plurality of synthesized channels, the downmix signal (324, x) being a downmixed version of an original signal (212) having a plurality of original channels, the method comprising the following stages:

[0834] Phase 1 (610c') includes:

[0835] According to the first mixing matrix (M M ) synthesizes a first component (336M') of the synthesized signal:

[0836] The covariance matrix associated with the composite signal (e.g. the reconstructed target version of the covariance of the original signal); and

[0837] The covariance matrix (C x ).

[0838] The second stage (610c) is used to synthesize a second component (336R') of the synthesized signal, wherein the second component

[0839] The quantity (336R') is a residual component, and the second stage (610c) comprises:

[0840] a prototype signal step (612c) of upmixing the downmix signal (324) from the number of downmix channels to the number of synthesis channels;

[0841] a decorrelator step (614c) of decorrelating the upmixed prototype signal (613c);

[0842] A second mixing matrix step (618c) is performed based on a second mixing matrix (M) of the decorrelated version (615c) of the downmix signal (324). R ) synthesizes the second component (336R') of the synthesized signal, the second mixing matrix (M R ) is the residual mixing matrix,

[0843] The method calculates the second mixing matrix (M) from R ):

[0844] The residual covariance matrix (C r );and

[0845] From the covariance matrix (C x ) The decorrelated prototype signal obtained An estimate of the covariance matrix of

[0846] The method further comprises an adder step (620c) of adding the first component (336M') of the composite signal

[0847] The composite signal (336) is added to the second component (336R') of the composite signal to obtain the composite signal (336).

[0848] Furthermore, a coding method is provided for generating a downmix signal (246, x) from an original signal (212, y), the original signal (212, y) having a plurality of original channels, the downmix signal (246, x) having a plurality of downmix channels, the method comprising:

[0849] estimating (218) channel levels and related information (220) of the original signal (212, y),

[0850] The downmix signal (246, x) is encoded (226) into a bit stream (248) such that the downmix signal (246,

[0851] x) is encoded in the bitstream (248) so as to have side information (228) comprising channel levels and related information (220) of the original signal (12, y).

[0852] These methods can be implemented in any of the encoders and decoders discussed above.

[0853] 7. Storage Unit

[0854] Furthermore, the present invention may be implemented in a non-transitory storage unit storing instructions which, when executed by a processor, cause the processor to perform the method as described above.

[0855] Furthermore, the present invention may be implemented in a non-transitory storage unit storing instructions which, when executed by the processor, cause the processor to control at least one of the functions of the encoder or the decoder.

[0856] The storage unit may be, for example, a part of the encoder 200 or the decoder 300 .

[0857] 8. Other aspects

[0858] Although some aspects have been described in the context of devices, it is obvious that these aspects also represent the description of the corresponding method, wherein the block or device corresponds to the feature of the method step or the method step. Similarly, the aspects described in the context of the method step also represent the description of the corresponding block or project or feature of the corresponding device. Some or all of the method steps can be performed by (or using) hardware devices such as microprocessors, programmable computers or electronic circuits. In some respects, such a device can perform one or more of the most important method steps.

[0859] Depending on certain implementation requirements, aspects of the present invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system so that the corresponding method is performed. Therefore, the digital storage medium may be computer readable.

[0860] Some aspects according to the invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0861] Generally speaking, aspects of the present invention can be implemented as a computer program product with a program code, the program code being operative to perform one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0862] Other aspects comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0863] In other words, one aspect of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0864] Therefore, another aspect of the method of the invention is a data carrier (or a digital storage medium or a computer-readable medium) comprising the computer program recorded thereon, the computer program being configured to perform one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[0865] Therefore, another aspect of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured via a data communication connection, for example via the Internet.

[0866] Another aspect comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0867] Another aspect comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0868] According to another aspect of the invention, an apparatus or a system is included, which is configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device or the like. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0869] In some aspects, a programmable logic device (e.g., a programmable logic array) can be used to perform some or all of the functions of the methods described herein. In some aspects, a programmable logic array can collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.

[0870] The apparatus described herein may be implemented using a hardware device or a computer, or a combination of a hardware device and a computer.

[0871] The methods described herein may be performed using a hardware device or using a computer, or using a combination of a hardware device and a computer.

[0872] The aspects described above are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to those of ordinary skill in the art. Therefore, the intent of the present invention is limited only by the scope of the appended patent claims, and is not limited by the specific details presented by the description and explanation of the various aspects of this article.

[0873] 9. References

[0874] [1] J. Herre, K. J.Breebart,C.Faller,S.Disch,H.Purnhagen,J.Koppens,J.Hilpert,J.

[0875] W.Oomen, K.Linzmeier and KSChong, "MPEG Surround—The ISO / MPEGStandard for Efficient and Compatible Multichannel Audio Coding," AudioEnglish Society, vol.

[0876] 56, no.11, pp.932-955, 2008.

[0877] [2]V.Pulkki, "Spatial Sound Reproduction with Directional AudioCoding," Audio English Society,

[0878] vol.55,no.6,pp.503-516,2007.

[0879] [3] C.Faller and F.Baumgarte, "Binaural Cue Coding-Part II:Schemes andApplications," IEEE Transactions on Speech and Audio Processing, vol.11, no.6, pp.520-531, 2003.

[0880] [4]O.Hellmuth,H.Purnhagen,J.Koppens,J.Herre,J. J.Hilpert,L.Villemoes,L.Terentiv,C.Falch,A. M.L.Valero,B.Resch,H.Mundt and H.-O.Oh,“MPEG Spatial Audio Object Coding–The ISO / MPEG Standard for EfficientCoding of Interactive Audio Scenes,”in AES,San Fransisco,2010.

[0881] [5]L.Mikko-Ville and V.Pulkki,“Converting 5.1.Audio Recordings to B-Format for Directional Audio Coding Reproduction,”in ICASSP,Prague,2011.

[0882] [6]D.A.Huffman,“A Method for the Construction of Minimum-RedundancyCodes,”Proceedings of the IRE,vol.40,no.9,pp.1098-1101,1952.

[0883] [7]A.Karapetyan,F.Fleischmann and J.Plogsties,“Active MultichannelAudio Downmix,”in 145th Audio Engineering Society,New York,2018.

[0884] [8]J.Vilkamo,T. and A.Kuntz,“Optimized Covariance DomainFramework for Time-Frequency Processing of Spatial Audio,”Journal of theAudio Engineering Society,vol.61,no.6,pp.403-411,2013.

Claims

1. An audio encoder (200) for generating a downmix signal (246, x) from an original signal (212, y), the original signal (212, y) having a plurality of original channels, the downmix signal (246, x) having a plurality of downmix channels, the audio encoder (200) comprising: a parameter estimator (218) configured to estimate channel levels and related information (220) of the original signal (212, y), and A bitstream writer (226) is configured to encode the downmix signal (246, x) into a bitstream (248), such that the downmix signal (246, x) is encoded in the bitstream (248) to have side information (228), wherein the side information (228) includes channel levels and related information (220) of the original signal (212, y).

2. An audio encoder according to claim 1, configured to provide the channel levels and related information (220) of the original signal (212, y) as normalized values.

3. The audio encoder according to claim 1, wherein the channel levels and related information (220) of the original signal (212, y) encoded in the side information (228) at least comprises or represents channel level information associated with a total number of the original channels.

4. The audio encoder according to claim 1, wherein the channel levels and related information (220) of the original signal (212, y) encoded in the side information (228) at least comprises or represents related information (220, 908), the related information (220, 908) describing an energy relationship between at least one pair of different original channels, but less than the total number of the original channels.

5. The audio encoder according to claim 1, wherein the channel levels and related information (220) of the original signal (212, y) include at least one coherence value (ξ i,j ), the coherence value (ξ i,j ) describes the coherence between two channels in a pair of original channels. The audio encoder of claim 5 , wherein the coherence value is normalized.

7. An audio encoder according to claim 5, wherein the coherence value is in is the covariance between channels i and j, and are the levels associated with channels i and j respectively.

8. The audio encoder according to claim 1, wherein the channel levels and related information (220) of the original signal (212, y) comprise at least one inter-channel level difference ICLD.

9. The audio encoder of claim 8, wherein the at least one ICLD is provided as a logarithmic value.

10. The audio encoder of claim 8, wherein the at least one ICLD is standardized.

11. The audio encoder according to claim 10, wherein the ICLD is in -x i is the ICLD for channel i, -P i is the power of the current channel i, -P dmx,i is a linear combination of the values ​​of the covariance information of the downmix signal.

12. An audio encoder according to claim 1, configured to select (250) whether to encode or not encode at least a part of the channel levels and related information (220) of the original signal (212, y) on the basis of state information (252), so as to include an increased amount of channel levels and related information (220) in the side information (228) in case of a relatively low payload.

13. An audio encoder according to claim 1, configured to select (250) which part of the channel levels and related information (220) of the original signal (212, y) is to be encoded in the side information (228) on the basis of a measure (252) about the channel, so as to include the channel levels and related information (220) associated with the more sensitive measure in the side information (228).

14. The audio encoder according to claim 1, wherein the channel levels and related information (220) of the original signal (212, y) are in the form of a matrix (C y ) in the form of an item.

15. The audio encoder according to claim 14, wherein the matrix is ​​a symmetric matrix or a Hermitian matrix, wherein the entries of the channel level and related information (220) are for the matrix (C y ) or less than the total number of entries in the diagonal and / or for less than the matrix (C y ) is provided by half of the off-diagonal elements of .

16. The audio encoder of claim 1, wherein the bitstream writer (226) is configured to encode an identification of at least one channel.

17. The audio encoder according to claim 1, wherein the original signal (212, y) or a processed version thereof (216) is divided into a plurality of subsequent frames of equal time length.

18. The audio encoder according to claim 17, configured to encode channel levels and related information (220) of the original signal (212, y) specific to each frame in the side information (228).

19. An audio encoder according to claim 18, configured to encode in the side information (228) the same channel levels and related information (220) of the original signal (212, y) commonly associated with a plurality of consecutive frames.

20. The audio encoder of claim 18, configured to select the number of consecutive frames for which the same channel level and related information (220) of the original signal (212, y) are selected such that: A relatively higher bit rate or a higher payload implies an increase in the number of consecutive frames to which the same channel level and related information (220) of the original signal (212, y) are associated, and vice versa.

21. An audio encoder according to claim 19, configured to reduce the number of consecutive frames to which identical channel levels and related information (220) of the original signal (212, y) are associated when a transient is detected.

22. The audio encoder of claim 17, wherein each frame is subdivided into an integer number of consecutive time slots.

23. The audio encoder according to claim 22, configured to estimate the channel level and the related information (220) for each time slot and to encode in the side information (228) a sum or an average or another predetermined linear combination of the channel levels and the related information (220) estimated for different time slots.

24. The audio encoder of claim 23, configured to perform a transient analysis (258) on the time domain version of the frame to determine occurrences of transients within the frame.

25. An audio decoder according to claim 24, configured to determine in which time slot of the frame the transient has occurred, and: encoding the channel levels and related information (220) of the original signal (212, y) associated with the time slot in which the transient has occurred and / or a subsequent time slot in the frame, The channel levels and related information (220) of the original signal (212, y) associated with a time slot preceding the transient are not encoded.

26. An audio encoder according to claim 24, configured to signal (261) in the side information (228) that the occurrence of the transient occurs in a time slot of the frame.

27. An audio encoder according to claim 26, configured to signal (261) in the side information (228) in which time slot of the frame the transient has occurred.

28. An audio encoder according to claim 24, configured to estimate the channel levels and related information (220) of the original signal (212, y) associated with multiple time slots of the frame, and sum them or average them or linearly combine them to obtain the channel levels and related information (220) associated with the frame.

29. An audio encoder according to claim 1, wherein the original signal (212, y) is converted (263) into a frequency domain signal (264, 266), wherein the audio encoder is configured to encode the channel levels and related information (220) of the original signal (212, y) in the side information (228) in a band-by-band manner.

30. An audio encoder according to claim 29, configured to aggregate (265) a plurality of frequency bands of the original signal (212, y) into a further reduced number of frequency bands (266) so as to encode the channel levels and related information (220) of the original signal (212, y) in the side information (228) in an aggregated band-by-aggregated band manner.

31. The audio encoder according to claim 29, configured to further aggregate (265) the frequency bands in case a transient in the frame is detected, such that: The number of frequency bands (266) is reduced; and / or The width of at least one frequency band is increased by aggregating with another frequency band.

32. The audio encoder of claim 29, further configured to encode (226) at least one channel level and related information (220) of a frequency band in a bitstream (248) as a delta relative to previously encoded channel level and related information.

33. An audio encoder according to claim 1, configured to encode in the side information (228) of the bitstream (248) an incomplete version of the channel levels and related information (220) relative to the channel levels and related information (220) estimated by the estimator (218).

34. An audio encoder according to claim 33, configured to adaptively select selected information to be encoded in the side information (228) of the bitstream (248) among the overall channel levels and related information (220) estimated by the estimator (218), such that remaining non-selected information of the channel levels and / or related information (220) estimated by the estimator (218) is not encoded.

35. An audio encoder according to claim 33, configured to reconstruct the channel levels and related information (220) from the selected channel levels and related information (220), thereby simulating estimates of non-selected channel levels and related information (220) at the decoder (300), and calculating error information between: the non-selected channel levels and related information estimated by the encoder (220); and the non-selected channel levels and related information reconstructed by simulating estimates of unencoded channel levels and related information (220) at the decoder (300); as well as So that on the basis of the calculated error information: Correctly reconstructed channel levels and related information; and Channel levels and related information cannot be correctly reconstructed, In order to determine: selecting the incorrectly reconstructible channel levels and related information to be encoded in the side information (228) of the bitstream (248); as well as The correctly reconstructible channel levels and related information are not selected, thereby avoiding encoding the correctly reconstructible channel levels and related information in the side information (228) of the bitstream (248).

36. An audio encoder according to claim 34, wherein the channel levels and related information (220) are indexed according to a predetermined ordering, wherein the encoder is configured to signal an index associated with the predetermined ordering in the side information (228) of the bitstream (248), the index indicating which of the channel levels and related information (220) is encoded.

37. An audio encoder according to claim 36, wherein the index is provided by a bitmap.

38. An audio encoder according to claim 36, wherein the index is defined according to a combined numbering system that associates a one-dimensional index with an entry of a matrix.

39. An audio encoder according to claim 36, configured to select between: adaptive provision of said channel levels and related information (220), wherein an index associated with said predetermined ordering is encoded in said side information of said bitstream; and The fixed provision of the channel levels and related information (220) enables the encoded channel levels and related information (220) to be predetermined and ordered according to a predetermined fixed order without providing an index.

40. An audio encoder according to claim 39, configured to signal in the side information (228) of the bitstream (248) whether the channel levels and related information (220) are provided according to an adaptive provision or according to a fixed provision.

41. The audio encoder of claim 1, further configured to encode (226) current channel levels and related information (220t) in the bitstream (248) as a delta (220k) relative to previous channel levels and related information (220(t-1)).

42. The audio encoder of claim 1, further configured to generate the downmix signal (246) based on a static downmix (244).

43. An audio encoder according to claim 1, wherein the audio encoder is independent of the audio synthesizer.

44. A system comprising an audio synthesizer (300) for generating a synthesized signal (336, 340, y) from a downmix signal (246, x) R ), the synthesized signal (336, 340, y R ) has a plurality of synthesized channels, the audio synthesizer (300) comprising: an input interface (312) configured to receive the downmix signal (246, x) having a plurality of downmix channels and side information (228), the side information (228) comprising channel levels and related information (314, ξ, χ) of an original signal (212, y) having a plurality of original channels; and A synthesis processor (404) is configured to generate the synthesized signal (336, 340, y according to at least one mixing rule using the following R ): Channel levels and related information (220, 314, ξ, χ) of the original signal (212, y); and Covariance information (C x ),as well as The audio encoder according to claim 1.

45. The system of claim 44, wherein the audio encoder is independent of the audio synthesizer.

46. ​​The system of claim 44, wherein the audio synthesizer is independent of the encoder.

47. A method for generating a downmix signal (246, x) from an original signal (212, y), the original signal (212, y) having a plurality of original channels, the downmix signal (246, x) having a plurality of downmix channels, the method comprising: estimating (218) channel levels and related information (220) of the original signal (212, y), The downmix signal (246, x) is encoded (226) into a bitstream (248) such that the downmix signal (246, x) is encoded in the bitstream (248) so as to have side information (228) including channel levels and related information (220) of the original signal (12, y).

48. A non-transitory storage unit storing instructions which, when executed by a processor, cause the processor to perform the method of claim 47.

Images