Multi-signal audio coding using signal whitening as preprocessing

Abstract

A multi-signal encoder for encoding at least three audio signals comprises: a signal preprocessor (100) for preprocessing each audio signal individually to obtain at least three preprocessed audio signals, wherein the preprocessing is performed so that the preprocessed audio signal is whitened relative to the signal before the preprocessing; an adaptive joint signal processor (200) for processing the at least three preprocessed audio signals to obtain at least three jointly processed signals or an unprocessed signal and at least two jointly processed signals; a signal encoder (300) for encoding each signal to obtain one or more encoded signals; and an output interface (400) for transmitting or storing the encoded multi-signal audio signal, the encoded multi-signal audio signal comprising the one or more encoded signals, side information related to the preprocessing and side information related to the processing.

Description

Technical Field

[0001] The present application relates to multi-signal audio coding using signal whitening as pre-processing. Background Art

[0002] Embodiments relate to a MDCT-based multi-signal encoding and decoding system with signal-adaptive joint channel processing, wherein the signal may be a channel and the multi-signal is a multi-channel signal, or alternatively, the audio signal is a component of a sound field description, such as an Ambisonics component, i.e., W, X, Y, Z in a first-order ambisonic or any other component in a higher-order ambisonic description. The signal may also be a signal of an A-format or B-format or any other format description of a sound field.

[0003] In MPEG USAC [1], joint stereo coding of two channels is performed using complex prediction, MPS 2-1-2, or unified stereo with a band-limited or full-band residual signal.

[0004] MPEG Surround [2] hierarchically combines OTT and TTT boxes for joint coding of multi-channel audio with or without transmission of a residual signal.

[0005] The MPEG-H Quadraphonic element [3] hierarchically applies the MPS2-1-2 stereo box followed by the Complex Prediction / MS stereo box which builds a “fixed” 4x4 remixing tree.

[0006] AC4 [4] introduced new 3-channel, 4-channel and 5-channel elements that allow the transmitted channels to be remixed via a transmitted mixing matrix and subsequent joint stereo coding information.

[0007] • Previous publications have suggested using orthogonal transforms such as the Karhunen-Loeve transform (KLT) for enhanced multi-channel audio coding [5].

[0008] Multi-channel Coding Tool (MCT) [6], which supports joint coding of more than two channels, enabling flexible and signal-adaptive joint channel coding in the MDCT domain. This is achieved by iterative combination and concatenation of stereo coding techniques such as real-valued complex stereo prediction and rotational stereo coding (KLT) of two specified channels.

[0009] In the context of 3D audio, the loudspeaker channels are distributed in several height layers, resulting in horizontal and vertical channel pairs. The joint coding of only two channels defined in USAC is not sufficient to take into account the spatial and perceptual relations between the channels. MPEG Surround is applied in an additional pre-processing / post-processing step, and the residual signal is sent separately without the possibility of joint stereo coding to exploit, for example, the dependency between the left and right vertical residual signals. The dedicated N-channel element in AC-4 was introduced, which allows efficient coding of joint coding parameters, but without a generic loudspeaker setup with more channels, which is proposed for new immersive playback scenarios (7.1+4, 22.2). The MPEG-H Quad Channel element is also limited to 4 channels and cannot be applied dynamically to any channel, but only to a preconfigured and fixed number of channels. MCT introduces the flexibility of signal-adaptive joint channel coding of any channel, but the stereo processing is performed on a windowed and transformed non-normalized (non-whitened) signal. Furthermore, the coding of the prediction coefficients or angles for each stereo box in each frequency band requires a large number of bits. Summary of the invention

[0010] It is an object of the present invention to provide an improved and more flexible concept for multiple signal encoding or decoding.

[0011] The present invention is based on the following discovery: the multi-signal coding efficiency is significantly enhanced by performing adaptive joint signal processing not on the original signal but on a pre-processed audio signal, wherein the pre-processing is performed so that the pre-processed audio signal is whitened relative to the signal before the pre-processing. On the decoder side, this means that post-processing is performed after the joint signal processing to obtain at least three processed decoded signals. The at least three processed decoded signals are post-processed according to the side information included in the encoded signal, wherein the post-processing is performed in such a way that the post-processed signal is less whitened than the signal before the post-processing. The post-processed signal ultimately represents the decoded audio signal, i.e. the decoded multi-signal, directly or after further signal processing operations.

[0012] Especially for immersive 3D audio formats, efficient multi-channel coding is obtained that exploits the properties of multiple signals to reduce the amount of transmitted data while preserving the overall perceived audio quality. In a preferred embodiment, adaptive joint coding of signals within a multi-channel system is performed using a perceptually whitened and additionally inter-channel level difference (ILD) compensated spectrum. The joint coding is preferably performed using a simple per-band M / S transform decision driven based on an estimated number of bits for the entropy encoder.

[0013] A multi-signal encoder for encoding at least three audio signals comprises a signal preprocessor for preprocessing each audio signal individually to obtain at least three preprocessed audio signals, wherein the preprocessing is performed so that the preprocessed audio signals are whitened relative to the signals before preprocessing. Adaptive joint signal processing of the at least three preprocessed audio signals is performed to obtain at least three jointly processed signals. The processing operates on the whitened signals. The preprocessing results in the extraction of certain signal characteristics (e.g., spectral envelopes), or such that if they are not extracted, the efficiency of the joint signal processing (e.g., joint stereo or joint multi-channel processing) will be reduced. In addition, in order to improve the efficiency of the joint signal processing, broadband energy normalization is performed on the at least three preprocessed audio signals so that each preprocessed audio signal has a normalized energy. The broadband energy normalization is signaled into the encoded audio signal as side information so that after inverse joint stereo or joint multi-channel signal processing, the broadband energy normalization can be reversed at the decoder side. By means of this preferred additional wideband energy normalization process, the efficiency of the adaptive joint signal processing can be increased, so that the number of frequency bands or even the number of complete frames that can be subjected to mid / side processing is substantially increased compared to left / right processing (dual mono processing). The efficiency of the entire stereo encoding process is increasingly higher, the greater the number of frequency bands or even complete frames that are subjected to conventional stereo or multichannel processing (e.g. mid / side processing).

[0014] From the perspective of stereo processing, the lowest efficiency is obtained when the adaptive joint signal processor has to adaptively decide for a band or a frame whether the band or the frame is to be processed by "dual mono" or left / right processing. Here, the left and right channels are processed as they are, but naturally in a whitened and energy-normalized domain. However, when the adaptive joint signal processor adaptively determines to perform mid / side processing for a certain band or frame, the mid signal is calculated by adding the first channel and the second channel, and the side signal is calculated by calculating the difference between the first channel and the second channel in the channel pair. Usually, with respect to its value range, the mid signal is comparable to one of the first channel and the second channel, but the side signal will generally be a signal with less energy, which can be efficiently encoded, or even in the most preferred case, the side signal is zero or close to zero, so that the spectral region of the side signal can even be quantized to zero and thus entropy encoded in an efficient manner. The entropy encoding is performed by a signal encoder to encode each signal to obtain one or more encoded signals, and the output interface of the multi-signal encoder sends or stores the encoded multi-signal audio signal, which includes the one or more encoded signals, the side information related to the preprocessing, and the side information related to the adaptive joint signal processing.

[0015] At the decoder side, a signal decoder, which typically comprises an entropy decoder, typically relies on preferably included bit allocation information to decode the at least three encoded signals. The bit allocation information is included as side information in the encoded multi-signal audio signal and can be derived, for example, at the encoder side by looking at the energy of the signal at the input of the signal (entropy) encoder. The output of the signal decoder within the multi-signal decoder is input into a joint signal processor for performing joint signal processing according to the side information included in the encoded signal to obtain at least three processed decoded signals. The joint signal processor preferably undoes the joint signal processing performed at the encoder side and typically performs inverse stereo or inverse multi-channel processing. In a preferred embodiment, the joint signal processor applies processing operations to calculate left / right signals from the mid / side signals. However, when the joint signal processor determines from the side information that dual mono processing already exists for a channel pair, this situation will be noted and used in the decoder for further processing.

[0016] Like the adaptive joint signal processor on the encoder side, the joint signal processor on the decoder side can be a processor operating in a cascaded channel pair tree or a simplified tree mode. The simplified tree also represents some kind of cascade processing, but the simplified tree differs from the cascaded channel pair tree in that the output of a processed pair cannot be used as the input of another pair to be processed.

[0017] It may be the case that, with respect to a first channel pair used by a joint signal processor at the multi-signal decoder side to start joint signal processing, this first channel pair, being the last channel pair processed at the encoder side, has side information for a certain frequency band indicating dual mono channels, but these dual mono channel signals may be used later in the channel pair processing as mid or side signals. This is signaled by corresponding side information associated with a paired processing that is performed to obtain the at least three individually encoded channels to be decoded at the decoder side.

[0018] Embodiments relate to a MDCT-based multi-signal encoding and decoding system with signal-adaptive joint channel processing, wherein the signal may be a channel and the multi-signal is a multi-channel signal, or alternatively, the audio signal is a component of a sound field description, such as an Ambisonics component, i.e., W, X, Y, Z in a first-order ambisonic or any other component in a higher-order ambisonic description. The signal may also be a signal of an A-format or B-format or any other format description of a sound field.

[0019] Other advantages of the preferred embodiment are subsequently pointed out. The codec incorporates the flexibility of adaptive joint coding of signals of arbitrary channels described in [6] using new concepts by introducing the concepts described in [7] for joint stereo coding. These new concepts are:

[0020] a) Use a perceptually whitened signal for further encoding (similar to how they are used in speech encoders). This has several advantages:

[0021] • Simplified codec architecture

[0022] • Compact representation of noise shaping properties / masking thresholds (e.g. as LPC coefficients)

[0023] • Unified transform and speech codec architecture, thus enabling combined audio / speech coding

[0024] b) Use ILD parameters of any channel to efficiently encode panned sources

[0025] c) Flexible bit allocation among processed channels based on energy.

[0026] In addition, the codec uses frequency domain noise shaping (FDNS) to perceptually whiten the signal via a rate loop as described in [8] in combination with a spectral envelope warping as described in [9]. The codec further normalizes the FDNS-whitened spectrum towards the mean energy level using the ILD parameter. The channel pairs for joint encoding are selected in an adaptive manner as described in [6], where the stereo encoding consists of a comparison of bandwise M / S and L / R decisions. When encoding in L / R and M / S modes as described in [7], the bandwise M / S decision is based on the estimated bitrate in each band. The bitrate distribution between the channels processed by bandwise M / S is based on energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0028] Figure 1 A block diagram showing single channel preprocessing in a preferred embodiment is shown;

[0029] Figure 2 A preferred embodiment of a block diagram of a multi-signal encoder is shown;

[0030] Figure 3 Shows Figure 2 A preferred implementation of the cross-correlation vector and channel pair selection process;

[0031] Figure 4 shows the indexing scheme of channel pairs in a preferred embodiment;

[0032] Figure 5a A preferred embodiment of the multi-signal encoder according to the present invention is shown;

[0033] Figure 5bshows a schematic representation of an encoded multi-channel audio signal frame;

[0034] Figure 6 Shown by Figure 5a The process performed by the adaptive joint signal processor;

[0035] Figure 7 Shown by Figure 5a A preferred embodiment of the adaptive joint signal processor;

[0036] Figure 8 Shown by Figure 5a Another preferred embodiment of the adaptive joint signal processor;

[0037] Fig. 9 shows the method for executing Figure 5a Another process of bit allocation used by the quantization encoding processor;

[0038] Fig.10 A block diagram showing a preferred embodiment of a multi-signal decoder is shown;

[0039] Fig.11 Shown by Fig.10 A preferred embodiment of the combined signal processor execution;

[0040] Fig.12 Shows Fig.10 A preferred embodiment of a signal decoder;

[0041] Fig.13 Another preferred embodiment of the joint signal processor in the context of bandwidth extension or intelligent gap filling (IGF) is shown;

[0042] Fig.14 Shows Fig.10 Another preferred embodiment of the combined signal processor;

[0043] Fig.15a Shown by Fig.10 A signal decoder and a preferred processing block performed by a joint signal processor; and

[0044] Fig.15b An embodiment of a post-processor for performing a de-whitening operation and optional other processes is shown. DETAILED DESCRIPTION

[0045] Figure 5aA preferred embodiment of a multi-signal encoder for encoding at least three audio signals is shown. The at least three audio signals are input into a signal processor 100 to preprocess each audio signal individually to obtain at least three preprocessed audio signals 180, wherein the preprocessing is performed so that the preprocessed audio signals are whitened relative to the corresponding signals before the preprocessing. The at least three preprocessed audio signals 180 are input into an adaptive joint signal processor 200, which is configured to perform processing on the at least three preprocessed audio signals to obtain at least three jointly processed signals, or in an embodiment, to obtain an unprocessed signal and at least two jointly processed signals, as will be described later. The multi-signal encoder includes a signal encoder 300, which is connected to the output of the adaptive joint signal processor 200 and is configured to encode each signal output by the adaptive joint signal processor 200 to obtain one or more encoded signals. These encoded signals at the output of the signal encoder 300 are forwarded to an output interface 400. The output interface 400 is configured for sending or storing an encoded multi-signal audio signal 500, wherein the encoded multi-signal audio signal 500 at the output of the output interface 400 comprises the one or more encoded signals generated by the signal encoder 300, side information 520 (i.e. whitening information) related to the pre-processing performed by the signal pre-processor 200, and additionally, the encoded multi-signal audio signal also comprises side information 530 related to the processing performed by the adaptive joint signal processor 200 (i.e. side information related to adaptive joint signal processing).

[0046] In a preferred embodiment, the signal encoder 300 comprises a rate loop processor which is controlled by the bit allocation information 536 which is generated by the adaptive joint signal processor 200 and forwarded not only from the block 200 to the block 300 but also in the side information 530 to the output interface 400 and thus to the encoded multi-signal audio signal. The encoded multi-signal audio signal 500 is usually generated in a frame-by-frame manner, wherein the framing and usually the corresponding windowing and time-frequency conversion are performed in the signal preprocessor 100.

[0047] exist Figure 5b An exemplary diagram of a frame of an encoded multi-signal audio signal 500 is shown in FIG. Figure 5b A bitstream portion 510 is shown for the individually encoded signal generated by block 300. Block 520 is for the pre-processed side information generated by block 100 and forwarded to output interface 400. In addition, the jointly processed side information 530 is generated by Figure 5a The adaptive joint signal processor 200 generates and is introduced into Figure 5bIn the encoded multi-signal audio signal frame shown. Figure 5b To the right of the diagram, the next frame of the encoded multi-signal audio signal is written into the serial bit stream, while Figure 5b To the left of the diagram, early frames of the encoded multi-signal audio signal will be written.

[0048] As will be explained later, the preprocessing includes a temporal noise shaping process and / or a frequency domain noise shaping process or an LTP (long term prediction) process or a windowing process operation. The corresponding preprocessing side information 550 may include at least one of temporal noise shaping (TNS) information, frequency domain noise shaping (FDNS) information, long term prediction (LTP) information or windowing or window information.

[0049] Temporal noise shaping consists of a prediction of a spectral frame in frequency. Spectral values ​​with higher frequencies are predicted using a weighted combination of spectral values ​​with lower frequencies. TNS side information includes the weights of the weighted combination, also known as LPC coefficients derived by the prediction in frequency. The whitened spectral values ​​are prediction residual values, i.e., the difference between the original spectral value and the predicted spectral value for each spectral value. On the decoder side, an inverse prediction of the LPC synthesis filter is performed in order to undo the TNS processing on the encoder side.

[0050] The FDNS processing comprises weighting the spectral values ​​of the frame using weighting factors for the corresponding spectral values, wherein the weighting values ​​are derived from LPC coefficients calculated from the block / frame of the windowed time domain signal.The FDNS side information comprises a representation of the LPC coefficients derived from the time domain signal.

[0051] Another whitening process also useful for the present invention is spectral equalization using a scaling factor so that the equalized spectrum represents a more whitened version than the unequalized version. The side information would be the scaling factor used for weighting, and the inverse process would consist of undoing the equalization at the decoder side using the sent scaling factor.

[0052] Another whitening process consists in performing an inverse filtering of the spectrum using an inverse filter controlled by LPC coefficients derived from the time domain frame as known in the art of speech coding. The side information is the inverse filter information and the inverse filtering is undone in the decoder using the transmitted side information.

[0053] Another whitening process consists in performing LPC analysis in the time domain and calculating time domain residual values ​​which are then converted to the spectral range. Typically, the spectral values ​​thus obtained are similar to those obtained by FDNS. On the decoder side, post-processing consists in performing LPC synthesis using the transmitted LPC coefficient representation.

[0054] In a preferred embodiment, the joint processing side information 530 includes paired processing side information 532, energy scaling information 534, and bit allocation information 536. The paired processing side information may include at least one of the following: channel pair side information bits, full mid / side or dual mono or band-wise mid / side information, and a mid / side mask in the case of band-wise mid / side indication, wherein the mid / side mask indicates for each bandwidth in the frame whether the band is processed by mid / side processing or L / R processing. The paired processing side information may additionally include intelligent gap filling (IGF) or other bandwidth extension information such as SBR (spectral band replication) information, etc.

[0055] For each whitened (i.e., preprocessed) signal 180, the energy scaling information 534 may include an energy scaling value and a flag indicating whether the energy scaling is an upscaling or a downscaling. For example, in the case of eight channels, box 534 will include eight scaling values ​​(e.g., eight quantized ILD values) and eight flags indicating, for each of the eight channels, whether upscaling or downscaling is to be done within the encoder, or whether upscaling or downscaling must be done within the decoder. When the actual energy of a preprocessed channel in a frame is lower than the average energy of all channels for the frame, upscaling in the encoder is necessary, and when the actual energy of a channel in a frame is higher than the average energy of all channels in the frame, downscaling is necessary. The joint processing side information may include bit allocation information for each of the jointly processed signals or for the unprocessed signal (if available) and each jointly processed signal, and the bit allocation information is used by the signal encoder 300 (e.g., Figure 5a shown) and accordingly Fig.10 The illustrated signal decoder uses a signal decoder which receives the bitstream information from the encoded signal via an input interface.

[0056] Figure 6 The preferred embodiment of the adaptive joint signal processor is shown. The adaptive joint signal processor 200 is configured to perform broadband energy normalization on the at least three preprocessed audio signals so that each preprocessed audio signal has normalized energy. The output interface 400 is configured to include a broadband energy normalization value for each preprocessed audio signal as further side information, wherein the value corresponds to Figure 5b Energy scaling information 534. Figure 6 A preferred embodiment of broadband energy normalization is shown. In step 211, the broadband energy of each channel is calculated. The input to box 211 consists of the pre-processed (whitened) channels. The result is C totalThe broadband energy value for each of the channels is calculated. In block 212, the average broadband energy is typically calculated by adding the individual values ​​and dividing the individual values ​​by the number of channels. However, other average calculation processes may be performed, such as a geometric mean, etc.

[0057] In step 213, each channel is normalized. To this end, a scaling factor or value and amplification or reduction information are determined. Therefore, block 213 is configured to output a scaling flag for each channel indicated at 534a. In block 214, the actual quantization of the scaling ratio determined in block 212 is performed and the quantized scaling ratio is output for each channel at 534b. The quantized scaling ratio is also indicated as the inter-channel level difference , i.e., for a certain channel k, relative to a reference channel with average energy. In block 215, the spectrum of each channel is scaled using the quantized scaling factor. The scaling operation in block 215 is controlled by the output of block 213, i.e., by the information about whether upscaling or downscaling is to be performed. The output of block 215 represents the scaled spectrum for each channel.

[0058] Figure 7 A preferred embodiment of the adaptive joint signal processor 200 with respect to cascaded pair processing is shown. The adaptive joint signal processor 200 is configured to calculate the cross-correlation value for each possible channel pair as indicated by block 221. Block 229 shows that the pair with the highest cross-correlation value is selected, and in block 232a, a joint stereo processing mode is determined for the pair. The joint stereo processing mode may include mid / side coding for the complete frame, mid / side coding in a band-wise manner, i.e., determining for each of a plurality of bands whether the band is to be processed in mid / side mode or in L / R mode, or for the actual frame, whether full-band dual mono processing is to be performed for the particular pair being considered. In block 232b, the joint stereo processing for the selected pair is actually performed using the mode determined in block 232a.

[0059] In blocks 235, 238, cascade processing using the full tree or simplified tree processing or non-cascade processing is continued until a certain termination criterion. At the certain termination criterion, a pair indication such as output by block 229 and stereo mode processing information output by block 232a is generated and the pair indication is generated in relation to the stereo mode processing information. Figure 5b The illustrated paired processing side information 532 inputs the pair indication and the stereo mode processing information into the bitstream.

[0060] Figure 8 shows the preparation for Figure 5aThe preferred embodiment of the adaptive joint signal processor 200 performs signal encoding of the signal encoder 300. To this end, in block 282, the adaptive joint signal processor 200 calculates the signal energy of each stereo processed signal. Block 282 receives as input the joint stereo processed signals, and in the case where a channel has not been subjected to stereo processing because the channel has not been found to have sufficient cross-correlation with any other channel to form a useful channel pair, the channel is input to block 282 with an inverted or modified or non-normalized energy. This is generally indicated as an "energy restored signal", but in Figure 6 The energy normalization performed in block 215 of does not have to be completely restored. There are some alternatives for processing channel signals that have not yet been found to be used with another channel for channel pair processing. One process is to reverse the energy normalization that was originally performed in block 215 of Figure 6 The scaling performed in block 215 of . Another process is to only partially reverse the scaling, or another process is to weight the scaled channels in some different way, as the case may be.

[0061] In block 284, the total energy in all signals output by the adaptive joint signal processor 200 is calculated. In block 286, bit allocation information is calculated for each signal based on the signal energy of each stereo processed signal or (if applicable) the energy restored or energy weighted signal, and based on the total energy output by block 284. In one aspect, this side information 536 generated by block 286 is forwarded to Figure 5a signal encoder 300 and additionally forwarded to output interface 400 via logical connection 530 so that the bit allocation information is included in Figure 5a or Figure 5b In the encoded multi-signal audio signal 500.

[0062] In a preferred embodiment, based on Fig. 9 The process shown performs the actual bit allocation. In the first pass, the minimum number of bits for non-LFE (low frequency enhancement) channels is assigned, and, if available, low frequency enhancement channel bits are assigned. The signal encoder 300 requires these minimum number of bits regardless of a certain signal content. Figure 8 The remaining bits are assigned using bit allocation information 536 generated by block 286 and input to block 291. The assignment is done based on the quantized energy ratios, and preferably the quantized energy ratios are used instead of the non-quantized energies.

[0063] In step 292, refinement is performed. When quantization results in remaining bits being assigned and the result is higher than the number of available bits, a subtraction must be performed on the bits assigned in box 291. However, when the quantization of the energy ratio results in the assignment process in box 291 resulting in bits still being further assigned, these bits can be additionally given or allocated in the refinement step 292. If after the refinement step there are still any bits to be used by the signal encoder, a final donation step 293 is performed and a final donation is made to the channel with the maximum energy. At the output of step 293, the bit budget assigned for each signal is available.

[0064] In step 300, quantization and entropy coding are performed on each signal using the assigned bit budget generated by the process of steps 290, 291, 292, 293. Basically, bit allocation is performed in such a way that higher energy channels / signals are quantized more accurately than lower energy channels / signals. Importantly, the bit allocation is not performed using the original signal or the whitened signal, but rather the signal at the output of the adaptive joint signal processor 200, whose energy is different from the energy of the signal input into the adaptive joint signal processing due to the joint channel processing. In this context, it should also be noted that although channel pair processing is a preferred embodiment, other channel groups can be selected and processed by means of cross-correlation. For example, groups of three or even four channels can be formed by means of the adaptive joint signal processor and processed accordingly in a cascaded full process or a cascaded process with a simplified tree or in a non-cascaded process.

[0065] With the help of Fig.10 The signal decoder 700 performs the bit allocation shown in blocks 290, 291, 292, 293 on the decoder side in the same manner using the allocation information 536 extracted from the encoded multi-signal audio signal 500.

[0066] Preferred Embodiments

[0067] In this embodiment, the codec incorporates the flexibility of signal-adaptive joint coding of arbitrary channels described in [6] using new concepts by introducing the concepts described in [7] for joint stereo coding. These new concepts are:

[0068] a) Use a perceptually whitened signal for further encoding (similar to how they are used in speech encoders). This has several advantages:

[0069] • Simplified codec architecture

[0070] • Compact representation of noise shaping properties / masking thresholds (e.g. as LPC coefficients)

[0071] • Unified transform and speech codec architecture, thus enabling combined audio / speech coding

[0072] b) Use ILD parameters of any channel to efficiently encode panned sources

[0073] c) Flexible bit allocation among processed channels based on energy.

[0074] The codec uses frequency domain noise shaping (FDNS) to perceptually whiten the signal via a rate loop as described in [8] in combination with a spectral envelope warping as described in [9]. The codec further normalizes the FDNS-whitened spectrum towards the mean energy level using the ILD parameter. The channel pairs for joint encoding are selected in an adaptive manner as described in [6], where the stereo encoding consists of a comparison of bandwise M / S and L / R decisions. When encoding in L / R and M / S modes as described in [7], the bandwise M / S decision is based on the estimated bitrate in each band. The bitrate distribution between the channels processed by bandwise M / S is based on energy.

[0075] Embodiments relate to a MDCT-based multi-signal encoding and decoding system with signal-adaptive joint channel processing, wherein the signal may be a channel and the multi-signal is a multi-channel signal, or alternatively, the audio signal is a component of a sound field description, such as an Ambisonics component, i.e., W, X, Y, Z in a first-order ambisonics or any other component in a higher-order ambisonics description. The signal may also be a signal described in an A format or B format or any other format of a sound field. Therefore, the same disclosure given for a "channel" is also valid for a "component" or other "signal" of a multi-signal audio signal.

[0076] Encoder single channel processing dependent on whitened spectrum

[0077] According to Figure 1 According to the processing steps shown in the block diagram, each single channel is analyzed and transformed into a whitened MDCT domain spectrum.

[0078] The processing blocks of the time domain transient detector, windowing, MDCT, MDST and OLA are described in [8]. The MDCT and MDST form a modulated complex lapped transform (MCLT); performing the MDCT and MDST separately is equivalent to performing the MCLT; "MCLT to MDCT" means that only the MDCT part of the MCLT is used and the MDST is discarded.

[0079] Temporal noise shaping (TNS) is performed similarly to the description in [8], and the order of TNS and frequency domain noise shaping (FDNS) is adaptive. The presence of two TNS boxes in the figure will be understood as the possibility of changing the order of FDNS and TNS. The order decision of TNS and FDNS can be, for example, the decision described in [9].

[0080] Frequency Domain Noise Shaping (FDNS) and the calculation of FDNS parameters are similar to the process described in [9]. One difference is that the FDNS parameters for frames where TNS is not activated are calculated based on the MCLT spectrum. In frames where TNS is activated, the MDST spectrum is estimated based on the MDCT spectrum.

[0081] Figure 1 A preferred embodiment of a signal processor 100 is shown, which performs whitening on at least three audio signals to obtain individually preprocessed whitened signals 180. The signal preprocessor 100 includes an input of a time domain input signal for channel k. The signal is input to a window adder 102, a transient detector 104 and an LTP parameter calculator 106. The transient detector 104 detects whether the current portion of the input signal is transient, and in the case of confirming that the current portion of the input signal is transient, the transient detector 104 controls the window adder 102 to set a smaller window length. The window indication (i.e., which window length has been selected) is also included in the side information, and is specifically included in Figure 5b In the pre-processing side information 520 of frame 106. In addition, the LTP parameters calculated by frame 106 are also introduced into the side information frame, and these LTP parameters can be used, for example, to perform a certain type of post-processing of the decoded signal or other processes known in the art. Window adder 140 generates a windowed time domain frame, which is introduced into the time to spectrum converter 108. Time to spectrum converter 108 preferably performs a complex overlapped transform. As indicated by frame 112, from this complex overlapped transform, the real part can be derived to obtain the result of the MDCT transform. The result of frame 112 (that is, the MDCT spectrum) is input into the TNS frame 114a and the FDNS frame 116 connected subsequently. Alternatively, only FDNS is performed without TNS frame 114a, or vice versa, or TNS is performed after FDNS processing, as indicated by frame 114b. Usually, there is frame 114a or frame 114b. At the output of block 114b (when block 114a is not present) or at the output of block 116 (when block 114b is not present), a whitened, individually processed signal, i.e. a preprocessed signal, is obtained for each channel k. The TNS block 114a or 114b and the FDNS block 116 generate the preprocessing information and forward it to the side information 520.

[0082] In any case, it is not necessary to perform a complex transform in block 108. Furthermore, for some applications, it is sufficient that the time-to-spectral converter performs only the MDCT, and if the imaginary part of the transform is required, it can optionally be estimated from the real part. A feature of the TNS / FDNS process is that, in the case where TNS is not activated, the FDNS parameters are calculated from the complex spectrum (i.e., from the MCLT spectrum), while in frames where TNS is activated, the MDST spectrum is estimated from the MDCT spectrum, so that for the frequency domain noise shaping operation, a complete complex spectrum is always available.

[0083] Joint Channel Coding System Description

[0084] In the described system, after transforming each channel into the whitened MDCT domain, a signal adaptive exploitation of similarities in variations between arbitrary channels for joint coding is applied based on the algorithm described in [6]. Through this process, individual channel pairs to be jointly encoded using a band-wise M / S transform can be detected and selected.

[0085] Figure 2 An overview of the coding system is given. For simplicity, the box arrows represent single channel processing (i.e., a processing box is applied to each channel), and Figure 1 This is shown in detail in the box “MDCT domain analysis”.

[0086] In the following paragraphs, the individual steps of the algorithm applied for each frame will be described in detail. Figure 3 A data flow diagram of the described algorithm is given.

[0087] It should be noted that in the initial configuration of the system, there is a channel mask that indicates which channels the multi-channel joint encoding tool is active for. Therefore, for inputs where LFE (Low Frequency Effects / Enhancement) channels are present, these LFE channels will not be considered in the processing step of the tool.

[0088] Energy normalization towards the mean energy of all channels

[0089] If there is ILD, that is, if the channels are panned, the M / S transform is not efficient. We normalize the amplitude of the perceptually whitened spectra of all channels to the average energy level , to avoid this problem.

[0090] oEnergy calculation for each channel ,

[0091]

[0092] in is the total number of spectral coefficients.

[0093] oCalculation of average energy

[0094]

[0095] oNormalize the spectrum of each channel to the average energy

[0096] if (zoom out)

[0097]

[0098] in is the scaling factor. The scaling factor is uniformly quantized and sent to the decoder as side information bits.

[0099]

[0100] in

[0101] The quantized scaling factor is then given by the following formula, and the spectrum is finally scaled with the quantized scaling factor

[0102]

[0103] if (enlarge)

[0104]

[0105] as well as

[0106]

[0107] in The calculation is performed as before.

[0108] To distinguish whether we have performed downscaling or upscaling at the decoder and to restore the normalization, in addition to the In addition to the value, a 1-bit flag is sent (0 = reduce / 1 = increase). Indicates the quantized scaling value used for transmission The number of bits required is known to the encoder and decoder and does not have to be sent in the encoded audio signal.

[0109] Calculate the normalized inter-channel cross-correlation values ​​for all possible channel pairs

[0110] In this step, in order to decide and select which channel pair has the highest degree of similarity and is therefore suitable to be selected as a pair for stereo joint coding, the normalized cross-correlation value between channels is calculated for each possible channel pair. The normalized cross-correlation value for each channel pair is given by the cross-spectrum as follows:

[0111]

[0112] in

[0113]

[0114] is the total number of spectral coefficients per frame, and is the corresponding spectrum of the channel pair under consideration.

[0115] The normalized cross-correlation values ​​for each channel pair are stored in the following cross-correlation vector

[0116]

[0117] in is the maximum number of possible pairs.

[0118] like Figure 1 As shown, depending on the transient detector, we can have different box sizes (e.g., 10 or 20 ms window box size). Therefore, the inter-channel cross-correlation is calculated assuming that the spectral resolution of the two channels is the same. Otherwise, the value is set to 0, ensuring that such a channel pair is not selected for joint coding.

[0119] Use an indexing scheme that uniquely identifies each channel pair. Figure 4 An example of such a scheme for indexing six input channels is shown in .

[0120] The same indexing scheme is maintained throughout the algorithm, which is also used to signal the channel pairs to the decoder. The number of bits required to signal the number of channel pairs is

[0121]

[0122] Channel pair selection and joint coded stereo processing

[0123] After calculating the cross-correlation vector, the first channel pair to be considered for joint encoding is the corresponding channel pair having the highest cross-correlation value and being above a minimum value threshold of preferably 0.3.

[0124] The selected channel pairs are used as input to the stereo encoding process (i.e., band-wise M / S transform). For each spectral band, the decision whether the channel will be encoded using M / S coding or discrete L / R coding depends on the bit rate estimated for each case. The encoding method that is less demanding in terms of bits is chosen. The process is described in detail in [7].

[0125] The output of this process results in an updated spectrum for each channel of the selected channel pair. In addition, information about the channel pair that needs to be shared with the decoder (side information) is created, namely which stereo mode is selected (full M / S, dual mono or band-wise M / S) and if band-wise M / S is the selected mode, a corresponding mask indicating whether M / S encoding (1) or L / R (0) is selected.

[0126] For the next steps, there are two variants of the algorithm:

[0127] o Cascade channel pairs

[0128] For this variant, the cross-correlation vector is updated for the channel pairs that are affected by the changed spectrum of the selected channel pair (if we use M / S transform). For example, in the case of 6 channels, if the channel pair selected and processed is Figure 4 If the channel pair with index 0 in , it means encoding of channel 0 and channel 1, then after stereo processing we will need to recalculate the cross-correlation for the affected channel pairs (i.e. index 0, 1, 2, 3, 4, 5, 6, 7, 8).

[0129] The process then continues as before: the channel pair with the largest cross-correlation is selected, confirmed to be above a minimum threshold, and the stereo operation is applied. This means that channels that were part of a previous channel pair can be reselected to be used as inputs for a new channel pair, known as "concatenation". This can happen when there can still be residual correlation between the output of a channel pair and another arbitrary channel representing a different direction in the spatial domain. Of course, the same channel pair should not be selected twice.

[0130] When the maximum allowed number of iterations is reached (the absolute maximum is ), or after updating the cross-correlation vector, no channel pair value exceeds the threshold of 0.3 (no correlation exists between any channels), the process continues.

[0131] oSimplify the tree

[0132] The cascaded channel pair tree process is theoretically optimal because it attempts to remove all arbitrary channel correlations and provides maximum energy compression. On the other hand, it is quite complex because the number of selected channel pairs may be greater than More, resulting in additional computational complexity (from the M / S decision process for stereo operation) and also in additional metadata that needs to be sent to the receiver for each channel pair.

[0133] For the simplified tree variant, "cascading" is not allowed. This ensures that, when starting from the above process, when updating the cross-correlation vector, the values ​​of the affected channel pairs of the previous channel pair stereo operation are not recalculated, but set to 0. Therefore, it is not possible to select a channel pair where one of its channels is already part of an existing channel pair.

[0134] This is the description Figure 2 A variation of the "Adaptive Joint Channel Processing" box in .

[0135] This case results in similar complexity to a system with predefined channel pairs (e.g., L and R, rear L and rear R), since the maximum channel pair that can be selected is .

[0136] It should be noted that there may be cases where stereo operation of the selected channel pair does not change the frequency spectrum of the channels. This occurs when the M / S decision algorithm decides that the coding mode should be "dual mono". In this case, any of the channels involved are no longer considered as channel pairs, as they will be coded separately. Furthermore, updating the cross-correlation vector will have no effect. To continue the process, the channel pair with the next highest value is considered. The steps in this case continue as described above.

[0137] Preserve channel pair selection (stereo tree) from previous frame

[0138] In many cases, the normalized cross-correlation values ​​for any channel pair may be close from frame to frame, and therefore the selection may frequently switch between such close values. This can result in frequent channel pair tree switching, which may lead to audible instability in the output system. Therefore, the selection uses a stabilization mechanism where a new set of channel pairs is only selected when the signal changes significantly and the similarity between any channels changes. To detect this, the cross-correlation vectors of the current frame are compared to the previous frame, and when the difference is greater than a certain threshold, a new channel pair is allowed to be selected.

[0139] The variation of the cross-correlation vector over time is calculated as follows:

[0140]

[0141] if , then a new channel pair to be jointly encoded is selected (as described in the previous step). The selected threshold is given by

[0142]

[0143] On the other hand, if the difference is small, the same channel pair tree as in the previous frame is used. For each given channel pair, a band-wise M / S operation is applied as previously described. However, if the normalized cross-correlation value for a given channel pair does not exceed a threshold of 0.3, the selection of a new channel pair for creating a new tree is initiated.

[0144] Restore the energy of a single channel

[0145] After the iterative process for channel pair selection is terminated, there may be channels that are not part of any channel / pair and are therefore coded separately. For those channels, the initial normalization of the energy levels towards the average energy level is restored back to their original energy levels. Depending on the flag signaling upscaling or downscaling, the inverse of the quantized scaling factor is used to restore the energy of these channels to their original state.

[0146] IGF for multi-channel processing

[0147] Regarding the IGF analysis, in the case of stereo channel pairs, an additional joint stereo processing is applied (which is fully described in

[10] ). This is necessary because for a certain destination range in the IGF spectrum, the signals can be highly correlated panned sources. In case the source regions selected for this particular region are not well correlated, the spatial image can be affected by the uncorrelated source regions, although the energy is matched for the destination region.

[0148] Therefore, if the stereo mode of the core area is different from the stereo mode of the IGF area, or if the stereo mode of the core is marked as band-wise M / S, stereo IGF is applied for each channel pair. If these conditions do not apply, a single channel IGF analysis is performed. If there are single channels that are not jointly coded in a channel pair, these single channels will also undergo single channel IGF analysis.

[0149] Allocation of available bits for encoding the spectrum of each channel

[0150] After the process of joint channel pair stereo processing, each channel is quantized and encoded separately by an entropy encoder. Therefore, for each channel, the number of available bits should be given. In this step, the energy of the processed channels is used to allocate the total available bits to each channel.

[0151] Due to the joint processing, the spectrum of each channel may have changed, so the energy of each channel is recalculated (the calculation of which is described in the normalization step above). The new energy is denoted as As a first step, the energy-based ratio (which will be used to allocate bits) is calculated as:

[0152]

[0153] It should be noted here that in case the input component also comes from the LFE channel, the ratio calculation does not take this input into account. For the LFE channel, the minimum number of bits is assigned only if this channel has non-zero content. . The ratio is uniformly quantized:

[0154]

[0155]

[0156] The quantified proportion Stored in the bitstream, it will be used by the decoder to assign the same amount of bits to each channel in order to read the transmitted channel spectral coefficients.

[0157] The bit allocation scheme is described below:

[0158] o Assign the minimum number of bits required for the entropy encoder to each channel

[0159] oThe remaining bits, i.e., Quantified proportion of used To divide:

[0160]

[0161] Because of the quantized ratio, the bits are roughly allocated and thus it can Therefore, in the second refining step, the difference is proportional to the channel bit Subtract:

[0162]

[0163] oAfter the refining step, if Compared to still exist If they do not match, the difference (usually a small number of bits) is donated to the channel with the most energy.

[0164] The decoder follows exactly the same process to determine the number of bits to read to decode the spectral coefficients for each channel. Indicates bit allocation information The number of bits required is known to the encoder and decoder and does not have to be sent in the encoded audio signal.

[0165] Quantization and encoding of each channel

[0166] Quantization, noise filling and entropy coding, including rate loop, as described in [8]. The estimated to optimize the rate loop. The power spectrum P (magnitude of the MCLT) is used for quantization and pitch / noise measurement in intelligent gap filling (IGF) as described in [8]. Since the whitened and band-wise M / S processed MDCT spectrum is used for the power spectrum, the same FDNS and M / S processing must be done on the MDST spectrum. The same ILD-based normalization scaling as done for the MDCT must be done for the MDST spectrum. For frames where TNS is activated, the MDST spectrum used for the power spectrum calculation is estimated from the whitened and M / S processed MDCT spectrum.

[0167] Figure 2 shows the encoder and specifically Figure 2 The block diagram of a preferred embodiment of an adaptive joint signal processor 200 of FIG. The at least three preprocessed audio signals 180 are all input to an energy normalization block 210, which generates at its output channel energy ratio edge bits 534, which on the one hand include quantized ratios and on the other hand include flags indicating amplification or reduction for each channel. However, other processes without explicit flags for amplification or reduction may also be performed.

[0168] The normalized channels are input to block 220 to perform cross-correlation vector calculation and channel pair selection. Based on the process in block 220, which is preferably an iterative process using cascaded full-tree or cascaded reduced-tree processing, or alternatively a non-iterative non-cascaded process, a corresponding stereo operation is performed in block 240, which may perform full-band or band-wise mid / edge processing or any other corresponding stereo processing operation, such as rotation, scaling, any weighted or non-weighted linear or non-linear combination, etc.

[0169] At the output of block 240, a stereo intelligent gap filling (IGF) process or any other bandwidth extension process, such as a spectral band replication process or a harmonic bandwidth process, may be performed. The processing of the individual channel pairs is signaled via the channel pair side information bits, and although Figure 2 Not shown, the IGF or general bandwidth extension parameters generated by block 260 are also written to the Figure 5b The side information 530 is processed jointly and specifically the side information 532 is processed in pairs in the bit stream.

[0170] Figure 2 The final stage is the channel bit allocation processor 280, which calculates the bit allocation ratio, such as for example Fig. 9 As described. Figure 2A schematic representation of a signal encoder 300 as quantizer and encoder is shown, which is controlled by the channel bitrate side information 530, and additionally a schematic representation of an output interface 400 or a bitstream writer 400 which communicates the result of the signal encoder 300 with the Figure 5b All required side information bits 520, 530 are combined.

[0171] Figure 3 FIG. 2 shows a preferred embodiment of the substantial process performed by blocks 210, 220, and 240. After the process is started, as shown in FIG. Figure 2 or Figure 3 ILD normalization is performed as indicated at 210 in FIG. In step 221, a cross-correlation vector is calculated. The cross-correlation vector consists of the normalized cross-correlation values ​​for each possible channel pair from 0 to N channels output by block 210. Figure 4 In the example of , where there are six channels, 15 different possibilities can be checked from 0 to 14. The first element of the cross-correlation vector has the cross-correlation value between channel 0 and channel 1, and the element with index 11 of the cross-correlation vector has the cross-correlation between channel 2 and channel 5, for example.

[0172] In step 222, a calculation is performed to determine whether the tree determined for the previous frame is to be maintained. To this end, the changes in the cross-correlation vectors over time are calculated and, preferably, the sum of the individual differences of the cross-correlation vectors and, in particular, the magnitude of the differences are calculated. In step 223, it is determined whether the sum of the differences is greater than a threshold value. If this is the case, in step 224, the flag keepTree is set to 0, which means that the tree is not kept, but a new tree is calculated. However, when it is determined that the sum is less than the threshold value, box 225 sets the flag keepTree to 1, so that the tree determined from the previous frame is also applied to the current frame.

[0173] In step 226, the iteration termination criteria are checked. In case it is determined that the maximum number of channel pairs (CPs) has not been reached (of course, this is the case when block 226 is accessed for the first time), and when the flag keepTree is set to 0 as determined in block 228, the process continues with block 229 to select the channel pair with the maximum cross-correlation in the cross-correlation vector. However, when the tree of the earlier frame is maintained, i.e., when keepTree is equal to 1 as checked in block 225, block 230 determines whether the cross-correlation of the "forced" channel pair is greater than a threshold. If this is not the case, the process continues with step 227, which means that a new tree is still to be determined despite the fact that the process in block 223 determined the opposite conclusion. The evaluation in block 230 and the corresponding result in block 227 can overturn the determination in blocks 223 and 225.

[0174] In block 231, it is determined whether the channel pair with the largest cross-correlation is above 0.3. If this is the case, the stereo operation in block 232 is performed. Figure 2 234. When it is determined in block 233 that the stereo operation is dual mono, the value of keepTree is set equal to 0 in block 234. However, when it is determined that the stereo mode is different from dual mono, the cross-correlation vector 235 must be recalculated because mid / side operation has been performed and the output of the stereo operation block 240 (or 232) is different due to this processing. The CC vector 235 must only be updated when there is actually a mid / side stereo operation or a stereo operation that is generally different from dual mono.

[0175] However, when the check in block 226 or the check in block 231 results in a "no" answer, control passes to block 236 to check whether a single channel is present. If this is the case, i.e., if a single channel is found that is not processed with another channel in the channel pair processing, the ILD normalization is reversed in block 237. Alternatively, the reversal in block 237 may only be a partial reversal, or may be some kind of weighting.

[0176] In the case where the iteration is completed and in the case where blocks 236 and 237 are also completed, the process ends and all channel pairs have been processed and at the output of the adaptive joint signal processor there are at least three jointly processed signals in the case where block 236 resulted in a "no" answer, or at least two jointly processed signals and an unprocessed signal corresponding to a "single channel" when block 236 has resulted in a "yes" answer.

[0177] Decoding system description

[0178] The decoding process starts with decoding and inverse quantization of the spectra of the jointly coded channels, followed by noise filling, as described in 6.2.2 "MDCT based TCX" in

[11] or

[12] . The number of bits allocated to each channel is based on the window length, stereo mode and bitrate ratio encoded in the bitstream. Before you can fully decode the bitstream, you must know the number of bits allocated to each channel.

[0179] In the intelligent gap filling (IGF) block, lines quantized to zero in a certain range of the spectrum (called the target block) are filled with processed content from a different range of the spectrum (called the source block). Due to the band-wise stereo processing, the stereo representation (i.e., L / R or M / S) may be different for the source block and the target block. To ensure good quality, if the representation of the source block differs from that of the target block, the source block is processed to transform it to the representation of the target block before gap filling in the decoder. This process has been described in

[10] . In contrast to

[11] and

[12] , the IGF itself is applied in the whitened spectral domain instead of the original spectral domain. In contrast to known stereo codecs (e.g.,

[10] ), the IGF is applied in the whitened, ILD-compensated spectral domain.

[0180] From the bitstream signaling, it is also known whether there are jointly coded channel pairs. The inverse processing should start from the last channel pair formed in the encoder, in particular for the cascaded channel pair tree, in order to convert back to the original whitened spectrum of each channel. For each channel pair, the inverse stereo processing is applied based on the stereo mode and the band-wise M / S decision.

[0181] For all channels involved in a channel pair and jointly coded, based on the values, the spectra are denormalized to the original energy levels.

[0182] Fig.10 A preferred embodiment of a multi-signal decoder for decoding an encoded signal 500 is shown. The multi-signal decoder comprises an input interface 600, a signal decoder 700 for decoding at least three encoded signals output by the input interface 600. The multi-signal decoder comprises a joint signal processor 800 for performing joint signal processing according to side information included in the encoded signals to obtain at least three processed decoded signals. The multi-signal decoder comprises a post-processor 900 for post-processing the at least three processed decoded signals according to the side information included in the encoded signals. In particular, the post-processing is performed in such a way that the post-processed signal is less whitened than the signal before the post-processing. The post-processed signal directly or indirectly represents the decoded audio signal 1000.

[0183] The side information extracted by the input interface 600 and forwarded to the joint signal processor 800 is Figure 5b The side information 530 shown and extracted from the encoded multi-signal audio signal by the input interface 600 (which is forwarded to the post-processor 900 for performing the de-whitening operation) is about Figure 5b Side information 520 is shown and described.

[0184] The joint signal processor 800 is configured to extract or receive an energy normalization value for each joint stereo decoded signal from the input interface 600. The energy normalization value for each joint stereo decoded signal corresponds to Figure 5b The adaptive joint signal processor 200 is configured to perform pairwise processing 820 on the decoded signal using the joint stereo side information or the joint stereo mode as indicated by the joint stereo side information 532 included in the encoded audio signal 500 to obtain a joint stereo decoded signal at the output of block 820. In block 830, a rescaling operation and in particular an energy rescaling of the joint stereo decoded signal is performed using the energy normalization value to obtain a joint stereo decoded signal at the output of block 820. Fig.10 A processed decoded signal is obtained at the output of block 800 .

[0185] To ensure that the channel has been received as described in block 237 Figure 3 In the illustrated inverse ILD normalization, the joint signal processor 800 is configured to check whether the energy normalization value for a specific signal extracted from the encoded signal has a predefined value. If this is the case, if the energy normalization value has the predefined value, no energy rescaling is performed, or only a reduced energy rescaling of the specific signal is performed, or any other weighting operation is performed on the individual channel.

[0186] In an embodiment, the signal decoder 700 is configured to receive a bit allocation value for each encoded signal from the input interface 600 as indicated by block 620. Fig.12 The bit allocation value shown at 536 in is forwarded to block 720 so that the signal decoder 700 determines the bit allocation used. Fig.12 The bit allocation used is determined in block 720 of the signal decoder 700, and the bit allocation is performed by the signal decoder 700. Figure 6 and Fig. 9 The same steps as those described on the encoder side in FIG. 7 (i.e., steps 290, 291, 292, 293). In block 710 / 730, a separate decoding is performed to obtain Fig.10 The input of the joint signal processor 800.

[0187] Using certain side information included in the side information box 532, the joint signal processor 800 has a frequency band replication, bandwidth extension or smart gap filling processing function. The side information is forwarded to box 810, and box 820 uses the result of the bandwidth extension process applied by box 810 to perform joint stereo (decoder) processing. In box 810, the smart gap filling process is configured to: when the destination range of the bandwidth extension or IGF processing is indicated as having another stereo representation, the source range is transformed from one stereo representation to the other stereo representation. When the destination range is indicated as having a mid / side stereo mode, and when the source range is indicated as having an L / R stereo mode, the L / R source range stereo mode will be transformed into a mid / side source range stereo mode, and then the IGF processing is performed with the mid / side stereo mode representation of the source range.

[0188] Fig.14 A preferred embodiment of a joint signal processor 800 is shown. The joint signal processor is configured to extract the ordered signal pair information, as shown in block 630. This extraction may be performed by the input interface 600, or the joint signal processor may extract the information from the output of the input interface, or may directly extract the information without a specific input interface, which is also the case with other extraction processes described with respect to the joint signal processor or signal decoder.

[0189] In block 820, the joint signal processor performs the inverse processing of the preferred cascade starting from the last signal pair, where the term "last" refers to the processing order determined and performed by the encoder. In the decoder, the "last" signal pair is the first signal pair processed. Block 820 receives side information 532 for the signal pair information indicated in block 630 and, for example, as described with respect to Figure 4 For each signal pair implemented in the described manner, the side information indicates whether the particular pair is a dual mono, full MS or band-wise MS process with an associated MS mask.

[0190] After the inverse processing in block 820, denormalization of the signals involved in the channel pair is performed in block 830, again relying on the side information 534 indicating the normalization information for each channel. Fig.14 The denormalization shown in block 830 in is preferably a rescaling using the energy normalization value, a reduction when the marker 534a has a first value, and a rescaling performed as an enlargement when the marker 534a has a second value (different from the first value).

[0191] Fig.15a Shown as Fig.10 A preferred embodiment of a block diagram of a signal decoder and a joint signal processor, and Fig.15b Shows Fig.10A block diagram representation of a preferred embodiment of a post-processor 900.

[0192] The signal decoder 700 comprises a decoder and a dequantizer stage 710 for the spectrum comprised in the encoded signal 500. The signal decoder 700 comprises a bit allocator 720 which preferably receives a window length, a certain stereo mode and bit allocation information for each encoded signal as side information. In a preferred embodiment, the bit allocator 720 specifically uses steps 290, 291, 292, 293 to perform bit allocation, wherein the bit allocation information for each encoded signal is used in step 291, and wherein the information about the window length and the stereo mode is used in blocks 290 or 291.

[0193] In block 730, noise filling, also preferably using noise filling side information, is also performed on the ranges in the spectrum that are quantized to zero and are not within the IGF range. The noise filling is preferably limited to the low-band portion of the signal output by block 710. In block 810, and using some of the side information, a smart gap filling or generally bandwidth extension process is performed, which importantly operates on the whitened spectrum.

[0194] In block 820, and using the side information, the inverse stereo processor performs a function to undo the Figure 2 The process of processing performed in item 240 of . The final descaling is performed using the sent quantized ILD parameters for each channel included in the side information. The output of box 830 is input to box 910 of the post-processor, which performs inverse TNS processing and / or inverse frequency domain noise shaping processing or any other de-whitening operation. The output of box 910 is a simple spectrum, which is converted to the time domain by the frequency to time converter 920. According to a certain encoding or decoding rule, the output of box 920 for adjacent frames is overlap-added in the overlap-add processor 930 to finally obtain a plurality of (multiplicity of) decoded audio signals from the overlap operation, or generally, a decoded audio signal 1000. The signal 1000 may be composed of individual channels, or may be composed of components of a sound field description such as an ambient stereo component, or may be composed of any other component of a high-order ambient stereo description. The signal may also be a signal described in A format or B format or any other format of the sound field. All of these alternatives are in Fig.15b hereinafter collectively referred to as decoded audio signal 1000 .

[0195] Further advantages and specific features of the preferred embodiments are subsequently pointed out.

[0196] The scope of the present invention is to provide a solution to the principles in [6] when processing signals that are perceptually whitened and ILD compensated.

[0197] o FDNS with a rate loop as described in [8] combined with spectral envelope warping as described in [9] provides a simple but very effective way to separate the perceptual shaping of quantization noise and the rate loop.

[0198] o Using the average energy level over all channels of the FDNS whitened spectrum allows a simple and efficient way to decide whether there is an advantage of M / S processing as described in [7] for each channel pair selected for joint coding.

[0199] o For the described system it is sufficient to encode a single wideband ILD per channel and thus a bit saving is achieved compared to known methods.

[0200] o By selecting channel pairs with highly cross-correlated signals for joint coding typically resulting in a full spectrum M / S transform, additional average bits are saved since signaling M / S or L / R for each frequency band is mostly replaced by a single bit signaling the full M / S transform.

[0201] o Flexible and simple bit allocation based on the energy of the processed channels.

[0202] Features of the preferred embodiments

[0203] As described in the previous paragraphs, in this embodiment, the codec uses a new approach to incorporate the flexibility of adaptive joint coding of signals for arbitrary channels described in [6] by introducing the concepts described in [7] for joint stereo coding. The novelty of the proposed invention can be summarized in the following differences:

[0204] o Regarding global ILD compensation, the joint processing for each channel pair is different from the multi-channel processing described in [6]. Global ILD equalizes the levels of the channels before selecting the channel pair and making the M / S decision and processing, and thus enables more efficient stereo coding, especially of panned sources.

[0205] o Regarding global ILD compensation, the joint processing for each channel pair is different from the stereo processing described in [7]. In the proposed system, there is no global ILD compensation for each channel pair. In order to be able to use the M / S decision mechanism described in [7] for any channel, all channels are normalized to a single energy level, namely the mean energy level. This normalization occurs before selecting the channel pair for joint processing.

[0206] o After the adaptive channel pair selection process, if there is a channel that is not part of the channel pair for joint processing, the energy level of that channel will be inverted to the initial energy level.

[0207] o The bit allocation for entropy coding is not performed on each channel pair as described in [7]. Instead, the energies of all channels are considered and the bits are allocated as described in the corresponding paragraphs of this document.

[0208] o There is an explicit "low complexity" mode of adaptive channel pair selection described in [6], in which a single channel that is part of a channel pair during the iterative channel pair selection process is not allowed to be part of another channel pair during that channel pair selection process.

[0209] o The advantage of using simple band-wise M / S for each channel and thus reducing the amount of information that needs to be sent in the bitstream is enhanced by the fact that we use the signal-adaptive channel pair selection of [6]. By selecting highly correlated channels for joint coding, for most cases a wideband M / S conversion is optimal, i.e. M / S coding is used for all bands. This can be signaled with a single bit and thus requires much less signaling information than the band-wise M / S decision. It significantly reduces the total amount of information bits that need to be sent for all channel pairs.

[0210] Embodiments of the invention relate to signal adaptive joint coding of a multi-channel system with a perceptually whitened and ILD compensated spectrum, where the joint coding consists of a simple per-band M / S transform decision based on an estimated number of bits for the entropy encoder.

[0211] Although some aspects have been described in the context of an apparatus, it will be clear that these aspects also represent descriptions of corresponding methods, wherein a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent descriptions of features of a corresponding block or item or a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0212] The novel encoded audio signal may be stored on a digital storage medium or may be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium (eg, the Internet).

[0213] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or in software. Implementation may be performed using a digital storage medium (e.g., a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory) having stored thereon electronically readable control signals that cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.

[0214] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0215] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0216] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0217] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0218] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory.

[0219] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (eg, via the Internet).

[0220] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0221] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0222] Another embodiment according to the invention comprises an apparatus or system configured to transmit (e.g. electronically or optically) a computer program to a receiver, the computer program being configured to perform one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0223] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0224] The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

[0225] The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0226] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended not to be limited by the specific details given by way of the description and explanation of the embodiments herein.

[0227] References (all incorporated herein by reference in their entirety):

[0228] [1] "Information technology - MPEG audio technologies Part 3: Unifiedspeech and audio coding," ISO / IEC 23003-3, 2012.

[0229] [2] "Information technology - MPEG audio technologies Part 1: MPEGSurround," ISO / IEC 23003-1, 2007.

[0230] [3] J. Herre, J. Hilpert, K. Achim and J. Plogsties, "MPEG-H 3DAudio—The New Standard for Coding of Immersive Spatial Audio," Journal ofSelected Topics in Signal Processing, vol. 5, no. 9, pp. 770-779, August2015.

[0231] [4] "Digital Audio Compression (AC-4) Standard," ETSI TS 103 190V1.1.1, 2014-04.

[0232] [5] D. Yang, H. Ai, C. Kyriakakis and C. Kuo, "High-fidelitymultichannel audio coding with Karhunen-Loeve transform," Transactions onSpeech and Audio Processing, vol. 11, no. 4, pp. 365-380, July 2003.

[0233] [6] F. Schuh, S. Dick, R. Füg, C. R. Helmrich, N. Rettelbach and T.Schwegler, "Efficient Multichannel Audio Transform Coding with Low Delay andComplexity," in AES Convention, Los Angeles, September 20, 2016.

[0234] [7] G. Markovic, E. Fotopoulou, M. Multrus, S. Bayer, G. Fuchs, J.Herre, E. Ravelli, M. Schnell, S. Doehla, W. Jaegers, M. Dietz and C.Helmrich, "Apparatus and method for mdct m / s stereo with global ild withimproved mid / side decision". International Patent WO2017125544A1, 27 July2017.

[0235] [8] 3GPP TS 26.445, Codec for Enhanced Voice Services (EVS); Detailedalgorithmic description.

[0236] [9] G. Markovic, F. Guillaume, N. Rettelbach, C. Helmrich and B.Schubert, "Linear prediction based coding scheme using spectral domain noiseshaping". EU Patent 2676266 B1, 14 February 2011.

[0237]

[10] S. Disch, F. Nagel, R. Geiger, B. N. Thoshkahna, K. Schmidt, S.Bayer, C. Neukam, B. Edler and C. Helmrich, "Audio Encoder, Audio Decoder andRelated Methods Using Two-Channel Processing Within an Intelligent GapFilling Framework". International Patent PCT / EP2014 / 065106, 15 07 2014.

[0238]

[11] "Codec for Encanced Voice Services (EVS); Detailed algorithmicdescription," 3GPP TS 26.445 V 12.5.0, December 2015.

[0239]

[12] "Codec for Encanced Voice Services (EVS); Detailed algorithmicdescription," 3GPP TS 26.445 V 13.3.0, September 2016.

[0240]

[13] Sascha Dick, F. Schuh, N. Rettelbach, T. Schwegler, R. Fueg, J. Hilpert and M. Neusinger, "Apparatus and Method for Encoding or Decoding a Multi-Channel Signal". International Patent PCT / EP2016 / 054900, 8 March 2016.

Claims

1. A multi-signal encoder for encoding at least three audio signals, comprising: A signal preprocessor (100) for preprocessing each audio signal individually to obtain at least three preprocessed audio signals, wherein the preprocessing is performed such that the preprocessed audio signals are whitened relative to the signals before preprocessing; An adaptive joint signal processor (200) is used to process the at least three pre-processed audio signals after energy level normalization to obtain at least three jointly processed signals or an unprocessed signal and at least two jointly processed signals, wherein in order to perform the energy level normalization, the adaptive joint signal processor (200) is configured to: calculating (212) information about the average energy of the preprocessed audio signal; calculating (211) information about the energy of each preprocessed audio signal; Calculating a broadband energy normalization value of the specific preprocessed audio signal based on the information about the average energy and the information about the energy of the specific preprocessed audio signal (534); and performing wideband energy normalization (210) of the at least three preprocessed audio signals using the wideband energy normalization value (534) of the particular preprocessed audio signal, wherein each preprocessed audio signal has a normalized energy; a signal encoder (300) for encoding the at least three jointly processed signals or the unprocessed signal and the at least two jointly processed signals to obtain one or more encoded signals (510); and An output interface (400) for transmitting or storing an encoded multi-signal audio signal, the encoded multi-signal audio signal comprising the one or more encoded signals (510), side information (520) related to the pre-processing, and side information (530) related to the processing, the side information (530) related to the processing comprising wideband energy normalized values ​​(534) of the pre-processed audio signals.

2. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to calculate (213, 214) a scaling factor (534b) as a broadband energy normalized value (534) of the specific preprocessed audio signal based on the average energy and the energy of the specific preprocessed audio signal, and wherein the adaptive joint signal processor (200) is configured to determine a flag (534a) indicating whether the scaling (534b) is for upscaling or downscaling, and wherein the flag (534b) for each preprocessed audio signal is included in the side information (530) associated with the processing.

3. The multi-signal encoder according to claim 2, in, The adaptive joint signal processor (200) is configured to quantize (214) the scaling factor (534b) to the same quantization range regardless of whether the scaling is up- or down-scaling.

4. The multi-signal encoder according to claim 1, wherein: The adaptive joint signal processor (200) is configured to: calculating (220) a cross-correlation value for each possible signal pair of preprocessed signals having normalized energies of the at least three preprocessed signals; selecting (229) the signal pair having the highest cross-correlation value; determining (232a) a joint stereo processing mode for the selected signal pair; as well as The selected signal pair is subjected to joint stereo processing according to the determined joint stereo processing mode (232b) to obtain a processed signal pair.

5. The multi-signal encoder according to claim 4, wherein: The adaptive joint signal processor (200) is configured to apply cascaded signal pair processing, or wherein the adaptive joint signal processor (200) is configured to apply non-cascaded signal pair processing, wherein in the cascaded signal pair processing, signals of the processed signal pair are selectable in a further iterative step, the further iterative step comprising the following operations: calculating updated cross-correlation values, selecting the signal pair with the highest cross-correlation value, determining a joint stereo processing mode for the selected signal pair, and performing joint stereo processing on the selected signal pair according to the determined joint stereo processing mode, or Wherein, in the non-cascaded signal pair processing, the signals in the processed signal pair are not selectable in the following operations: additionally selecting the signal pair with the highest cross-correlation value, determining the joint stereo processing mode of the selected signal pair, and performing joint stereo processing on the selected signal pair according to the determined joint stereo processing mode.

6. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to determine the signal to be encoded separately as the signal remaining after the pairwise processing process, and Therein, the adaptive joint signal processor (200) is configured to modify an energy normalization applied to the signal before performing the pairwise processing process.

7. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to determine bit allocation information (536) for each signal to be processed by the signal encoder (300), wherein the output interface (400) is configured to introduce the bit allocation information (536) for each signal into the side information (530) related to the processing.

8. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to calculate (282) signal energy information for each signal to be processed by the signal encoder (300), calculating (284) a total energy of a plurality of signals to be encoded by the signal encoder (300); calculating (286) bit allocation information (536) for each signal based on the signal energy information and the total energy information, and The output interface (400) is configured to introduce the bit allocation information into the side information (530) related to the processing for each signal.

9. The multi-signal encoder according to claim 8, in, The adaptive joint signal processor (200) is configured to: assign (290) an initial number of bits to each signal, assign (291) a number of bits based on the bit allocation information (536), perform (292) a further refinement step, or perform (292) a final donation step, and Therein, the signal encoder (300) is configured to perform signal encoding using the assigned bits of each signal.

10. The multi-signal encoder according to claim 1, wherein: The signal preprocessor (100) is configured to perform, for each audio signal: A time-to-spectral conversion operation (108, 110, 112) to obtain a signal spectrum of each audio signal; a temporal noise shaping operation (114a, 114b) and / or a frequency domain noise shaping operation (116) for each signal spectrum, and wherein the signal preprocessor (100) is configured to feed the signal spectrum to the adaptive joint signal processor (200) after the temporal noise shaping operation and / or the frequency domain noise shaping operation, and The adaptive joint signal processor (200) is configured to perform joint signal processing on a received signal spectrum.

11. The multi-signal encoder according to claim 1, wherein: The adaptive joint signal processor (200) is configured to for each signal of the selected signal pair, determining: the necessary bit rate for full band separate coding mode or the necessary bit rate for full band joint coding mode or the bit rate for band-wise joint coding mode plus the necessary bits for band-wise signalling, for all frequency bands of a signal pair, determining a separate coding mode or a joint coding mode as a specific coding mode, when most of the frequency bands have been determined for the specific coding mode, and a small number of frequency bands less than 10% of all the frequency bands have been determined as a minority coding mode in the separate coding mode or the joint coding mode, the minority coding mode being different from the specific coding mode; or determining a coding mode requiring a minimum number of bits, and Wherein the output interface (400) is configured to include an indication (532) in the side information (530) associated with the processing, the indication (532) indicating that the specific coding mode is to be used for all frequency bands of the frame instead of applying a coding mode mask to the frame.

12. The multi-signal encoder according to claim 1, in, The signal encoder (300) includes a rate loop processor for each individual signal or across two or more signals, the rate loop processor being configured to receive and use bit allocation information (536) for a particular signal or for two or more signals.

13. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to adaptively select a signal pair for joint encoding, or wherein the adaptive joint signal processor (200) is configured to determine a band-wise mid / side coding mode, a full-band mid / side coding mode or a full-band left / right coding mode for each selected signal pair, and wherein the output interface (400) is configured to indicate the selected coding mode as side information (532) in the encoded multi-signal audio signal.

14. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to form a comparison of the mid / side decision and the left / right decision in a band-wise manner based on the estimated bit rate in each band when encoding in the mid / side mode or in the left / right mode, and wherein the final joint coding mode is determined based on the result of the comparison of the mid / side decision and the left / right decision in a band-wise manner.

15. The multi-signal encoder according to claim 1, wherein: The adaptive joint signal processor (200) is configured to perform (260) a spectrum band replication processing or a smart gap filling processing for determining parameterized side information for the spectrum band replication processing or the smart gap filling processing, and wherein the output interface (400) is configured to include the spectrum band replication or smart gap filling side information (532) as additional side information in the side information (530) associated with the processing.

16. The multi-signal encoder according to claim 15, in, The adaptive joint signal processor (200) is configured to perform a stereo smart gap filling process on a coded signal pair and, additionally, to perform a single signal smart gap filling process on at least one signal to be individually coded.

17. The multi-signal encoder according to claim 1, in, The at least three audio signals include a low frequency enhancement signal, and wherein the adaptive joint signal processor (200) is configured to apply a signal mask indicating for which signals the adaptive joint signal processor (200) is to be active, and wherein the signal mask indicates that the low frequency enhancement signal is not to be used in paired processing of the at least three pre-processed audio signals.

18. The multi-signal encoder according to claim 1, wherein: The adaptive joint signal processor (200) is configured to calculate the energy of the MDCT spectrum of each preprocessed signal as information on the energy of each preprocessed signal, or Average energies of MDCT spectra of the at least three preprocessed audio signals are calculated as information on average energies of the at least three preprocessed audio signals.

19. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to calculate (213) a scaling factor for each signal based on information about the energy of the specific signal and information about the average energy of the at least three pre-processed audio signals, wherein the adaptive joint signal processor (200) is configured to quantize (214) the scaling factor to obtain a quantized scaling factor value, the quantized scaling factor value being used to derive side information of the scaling factor for each signal included in the encoded multi-signal audio signal, and The adaptive joint signal processor (200) is configured to derive a quantized scaling ratio from the quantized scaling ratio value, wherein the pre-processed audio signal is scaled using the quantized scaling ratio before being used for paired processing of the scaled signal with another corresponding scaled signal.

20. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to calculate (221) normalized inter-signal cross-correlation values ​​of possible signal pairs in order to decide and select which signal pair has the highest degree of similarity and is therefore suitable for selection as one of the at least three pre-processed audio signals for pairwise processing, where the normalized cross-correlation values ​​for each signal pair are stored in the cross-correlation vector, and The adaptive joint signal processor (200) is configured to determine whether to retain the signal pair selection of one or more previous frames by comparing the cross-correlation vector of the previous frame with the cross-correlation vector of the current frame (222, 223), and wherein the signal pair selection of the previous frame is retained (225) when the difference between the cross-correlation vector of the current frame and the cross-correlation vector of the previous frame is less than a predefined threshold.

21. The multi-signal encoder according to claim 1, in, The signal preprocessor (100) is configured to perform a time-frequency conversion using a window length selected from a plurality of different window lengths. wherein the adaptive joint signal processor (200) is configured to determine whether the signal pairs have the same correlation window length when comparing the pre-processed audio signals to determine the signal pairs to be processed in pairs, and Therein, the adaptive joint signal processor (200) is configured to allow paired processing of the two signals only if the two signals have been associated with the same window length applied by the signal preprocessor (100).

22. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to apply non-cascaded signal pair processing, in which signals in a processed signal pair are not selectable in further signal pair processing, wherein the adaptive joint signal processor (200) is configured to select signal pairs based on cross-correlation between signal pairs for paired processing, and wherein paired processing of several selected signal pairs is performed in parallel.

23. The multi-signal encoder according to claim 22, in, The adaptive joint signal processor (200) is configured to determine a stereo coding mode for the selected signal pair, and wherein, when the stereo coding mode is determined to be a dual mono mode, the signals involved in the signal pair are at least partially rescaled and indicated as signals to be separately encoded.

24. The multi-signal encoder according to claim 15, in, The adaptive joint signal processor (200) is configured to: if the stereo mode of the core area is different from the stereo mode of the intelligent gap filling (IGF) area, or if the stereo mode of the core is marked as band-wise mid / side coding, perform a stereo IGF operation on the pair-wise processed signal pair, or The adaptive joint signal processor (200) is configured to apply a single signal IGF analysis to the signal in the paired processed signal pair if the stereo mode of the core area is not different from the stereo mode of the IGF area, or the stereo mode of the core is not marked as a band-wise mid / side coding mode.

25. The multi-signal encoder according to claim 1, in, The adaptive joint signal processor (200) is configured to: perform an intelligent gap filling (IGF) operation before the result of the intelligent gap filling (IGF) operation is separately encoded by the signal encoder (300), wherein the power spectrum is used for quantization and pitch / noise determination in the IGF, and wherein the signal preprocessor (100) is configured to perform the same frequency domain noise shaping on the MDST spectrum as has been applied to the MDCT spectrum, and wherein the adaptive joint signal processor (200) is configured to perform the same mid / side processing on the pre-processed MDST spectrum so that the result of the processed MDST spectrum is used within the quantization performed by the signal encoder (300) or within the smart gap filling process performed by the adaptive joint signal processor (200), or The adaptive joint signal processor (200) is configured to apply the same normalized scaling based on the full-band scaling vector for the MDST spectrum as the scaling of the MDCT spectrum using the same quantized scaling vector.

26. The multi-signal encoder according to claim 1, wherein: The adaptive joint signal processor (200) is configured to perform pairwise processing on the at least three preprocessed audio signals to obtain the at least three jointly processed signals or a signal to be individually encoded and at least two jointly processed signals.

27. The multi-signal encoder according to claim 1, wherein: The audio signals of the at least three audio signals are audio channels, or The audio signal among the at least three audio signals is an audio component signal describing a sound field.

28. The multi-signal encoder according to claim 1, in, The signal encoder (300) is configured for encoding each signal individually to obtain at least three individually encoded signals, or for performing (entropy) encoding on more than one signal.

29. A multi-signal decoder for decoding an encoded signal, the encoded signal comprising side information (530), the multi-signal decoder comprising: A signal decoder (700) for decoding at least three encoded signals; The joint signal processor (800) is configured to: extracting (610) energy normalized values ​​(534, 534a, 534b) of the joint stereo decoded signal from the encoded signal; performing joint signal processing including pairwise processing (820) of the decoded signals using a joint stereo mode indicated by side information (530) included in the encoded signals; as well as energy rescaling (830) the joint stereo decoded signals using the energy normalization values ​​(534, 534a, 534b) to obtain at least three processed decoded signals; as well as A post-processor (900) for post-processing the at least three processed decoded signals based on the side information (530) included in the encoded signal, wherein the post-processing is performed such that the post-processed signals are less whitened than the signals before post-processing, and wherein the post-processed signals represent decoded audio signals.

30. The multi-signal decoder according to claim 29, in, The joint signal processor (800) is configured to check whether an energy normalization value (534, 534a, 534b) for a specific signal extracted from the encoded signal has a predefined value, and The joint signal processor (800) is configured to: when the energy normalization value (534, 534a, 534b) has the predefined value, perform no energy rescaling on the specific signal or perform only reduced energy rescaling on the specific signal.

31. The multi-signal decoder according to claim 29, wherein: The signal decoder (700) is configured to extracting (620) from the coded signals a bit allocation value for each coded signal, determining (720) a used bit allocation for the signal using the bit allocation value for the signal, the number of remaining bits for all signals, and a further refinement step or a final donation step; and Separate decoding is performed (710, 730) based on the used bit allocation for each signal.

32. The multi-signal decoder according to claim 29, wherein: The joint signal processor (800) is configured to performing (820) a spectral band replication or smart gap filling process on the separately decoded signals using side information in the encoded signals to obtain spectrally enhanced separate signals; as well as Joint processing is performed according to a joint processing mode using the spectrally enhanced separate signals (820).

33. The multi-signal decoder according to claim 32, in, The joint signal processor (800) is configured to transform the source range from one stereo representation to another stereo representation when the destination range is indicated as having the other stereo representation.

34. The multi-signal decoder according to claim 29, wherein: The joint signal processor (800) is configured to extracting a flag (534, 534a) from the encoded signal, the flag (534, 534a) indicating whether the energy normalized value (534, 534b) is an enlarged value or a reduced value, and The energy rescaling is performed (830) using the energy normalization value (534, 534b), the rescaling being a reduction when the flag (534, 534a) has a first value, and the rescaling being an increase when the flag (534, 534a) has a second value different from the first value.

35. The multi-signal decoder according to claim 29, wherein: The joint signal processor (800) is configured to extracting (630) from the encoded signal side information (532) indicative of a pair of signals resulting from the joint encoding operation, Starting from the last signal pair, inverse stereo or multi-channel processing is performed (820) to obtain the encoded signals in order to convert back to the original pre-processed spectrum of each signal, and the inverse stereo processing is performed based on the stereo mode and / or band-wise mid / side decision indicated in the side information (532) of the encoded signals.

36. The multi-signal decoder according to claim 29, in, The joint signal processor (800) is configured to energy rescale (830) all jointly stereo decoded signals involved in a signal pair to corresponding original energy levels based on the quantized energy normalization value (534, 534a, 534b) of each jointly stereo decoded signal, and wherein other signals not involved in the signal pair processing are not energy rescaled like the signals involved in the signal pair processing.

37. The multi-signal decoder according to claim 29, in, The post-processor (900) is configured to perform a temporal noise shaping operation (910) or a frequency domain noise shaping operation (910) for each individually processed decoded signal as well as a conversion from the spectral domain to the time domain (920) and a subsequent overlap / addition operation (930) between subsequent time frames of the post-processed signal.

38. The multi-signal decoder according to claim 29, in, The joint signal processor (800) is configured to extract from the coded signal a flag indicating whether to use mid / side or left / right coding to inversely process several frequency bands of a time frame of a signal pair, and wherein the joint signal processor (800) is configured to use the flag to cause corresponding frequency bands of the signal pair to all be subjected to mid / side processing or left / right processing depending on the value of the flag, and wherein, for different time frames of the same signal pair or different signal pairs at the same time frame, a coding mode mask indicating a separate coding mode for each separate frequency band is extracted from the side information of the encoded signal, and wherein the joint signal processor (800) is configured to apply inverse center / edge processing or left / right processing to the frequency band as indicated for the bit associated with the corresponding frequency band.

39. The multi-signal decoder according to claim 29, wherein: The coded signal is a coded multi-channel signal, wherein the multi-signal decoder is a multi-channel decoder, wherein the coded signal is a coded multi-channel signal, wherein the signal decoder (700) is a channel decoder, wherein the coded signal is a coded channel, wherein the joint signal processing is a joint channel processing, wherein the at least three processed decoded signals are at least three processed decoded signals, wherein the post-processed signals are channels, or wherein the encoded signal is an encoded multi-component signal, the encoded multi-component signal representing audio component signals of a sound field description, wherein the multi-signal decoder is a multi-component decoder, wherein the encoded signal is an encoded multi-component signal, wherein the signal decoder (700) is a component decoder, wherein the encoded signal is an encoded component, wherein the joint signal processing is a joint component processing, wherein the at least three processed decoded signals are at least three processed decoded components, and wherein the post-processed signal is a component audio signal.

40. A method for performing multi-signal encoding on at least three audio signals, comprising: preprocessing each audio signal individually to obtain at least three preprocessed audio signals, wherein the preprocessing is performed such that the preprocessed audio signals are whitened relative to the signals before preprocessing; Processing the at least three preprocessed audio signals after energy level normalization to obtain at least three jointly processed signals or an unprocessed signal and at least two jointly processed signals, wherein the energy level normalization comprises: calculating (212) information about the average energy of the preprocessed audio signal; calculating (211) information about the energy of each preprocessed audio signal; Calculating a broadband energy normalization value of the specific preprocessed audio signal based on the information about the average energy and the information about the energy of the specific preprocessed audio signal (534); and performing wideband energy normalization (210) of the at least three preprocessed audio signals using the wideband energy normalization value (534) of the particular preprocessed audio signal, wherein each preprocessed audio signal has a normalized energy; encoding the at least three jointly processed signals or the unprocessed signal and the at least two jointly processed signals to obtain one or more encoded signals; and An encoded multi-signal audio signal is transmitted or stored, the encoded multi-signal audio signal comprising the one or more encoded signals (510), side information related to the pre-processing (520), and side information related to the processing (530), the side information related to the processing (530) comprising wideband energy normalized values ​​(534) of the pre-processed audio signals.

41. A method for multi-signal decoding of an encoded signal, comprising: individually decoding at least three encoded signals; Perform joint stereo signal processing including: extracting (610) energy normalized values ​​(534, 534a, 534b) of the joint stereo decoded signal from the encoded signal; performing joint signal processing including pairwise processing (820) of the decoded signals using a joint stereo mode indicated by side information (530) included in the encoded signals; and energy rescaling (830) the joint stereo decoded signal using the energy normalization values ​​(534, 534a, 534b) to obtain at least three processed decoded signals; and The at least three processed decoded signals are post-processed according to side information (530) included in the encoded signal, wherein the post-processing is performed such that the post-processed signals are less whitened than the signals before post-processing, and wherein the post-processed signals represent decoded audio signals.

42. A computer program product storing a computer program for performing the method according to claim 40 or the method according to claim 41 when the computer program is run on a computer or a processor.

Images