Switching between stereo coding modes in a multi-channel sound codec

Abstract

A method and apparatus for encoding a stereo sound signal includes a stereo encoder using stereo modes operating in a time domain (TD), a frequency domain (FD), or a modified discrete Fourier transform (MDCT) domain. A controller controls switching between the TD, FD, and MDCT stereo modes. When switching from one stereo mode to another, the switching controller can (a) recalculate at least one length of downmix / mixed signals in a current frame of the stereo sound signal, (b) reconstruct the downmix / mixed signals and other signals related to the other stereo mode in the current frame, (c) adapt a data structure and / or memory coding the stereo sound signal in the current frame using the other stereo mode, and / or (d) change a TD stereo channel downmix to maintain correct phase of left and right channels of the stereo sound signal. Corresponding stereo sound signal decoding methods and apparatus are described.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to stereo sound encoding, in particular but not exclusively switching between "stereo coding modes" (hereinafter also referred to as "stereo modes") in a multi-channel sound codec, in particular but not exclusively capable of producing good stereo quality at low bit rates and low delay in e.g. complex audio scenes.

[0002] In the present disclosure and the appended claims:

[0003] The term "sound" can relate to speech, audio and any other sound;

[0004] The term "stereo" is an abbreviation of "stereophonic"; and

[0005] The term "mono" is an abbreviation of "monophonic". BACKGROUND

[0006] Historically, conversational telephony has been implemented with a handset having only one transducer, which can only output sound to one of the user's ears. In the last decade, users have started to use their portable handsets in combination with earphones, receiving sound through both of their ears, mainly for listening to music, but sometimes also for listening to speech. However, when the portable handset is used for sending and receiving conversational speech, the content is still mono, but presented to both ears of the user when the earphones are used.

[0007] The quality of the coded sound, e.g. speech and / or audio, transmitted and received by a portable handset has been significantly improved using the latest 3GPP speech coding (code) standard as described in reference [1], the entire contents of which are incorporated herein by reference. The next natural step is to transmit stereo information so that the receiver is as close as possible to the real-life audio scene captured at the other end of the communication link.

[0008] In audio codecs, e.g. as described in reference [2], the entire contents of which are incorporated herein by reference, the transmission of stereo information is commonly used.

[0009] For conversational speech codecs, mono signals are the norm. When sending stereo signals, the bit rate usually needs to be doubled since both the left and right channels of the stereo signal are coded using mono codecs. This works well in most scenarios, but has the disadvantage of doubling the bit rate and failing to exploit any possible redundancy between the two channels (left and right of the stereo signal). Furthermore, in order to keep the overall bit rate at a reasonable level, very low bit rates are used for each channel, impacting the overall sound quality. In order to reduce the bit rate, efficient stereo coding techniques have been developed and used. As non-limiting examples, the use of three stereo coding techniques that can be used efficiently at low bit rates is discussed in the following paragraphs.

[0010] The first stereo coding technique is called parametric stereo. Parametric stereo coding uses a common mono codec plus a certain amount of stereo side information (corresponding to stereo parameters) representing the stereo image to encode the left and right channels into a mono signal. The two input left and right channels are downmixed into a mono signal, then the stereo parameters are usually computed in a transform domain (e.g. the Discrete Fourier Transform (DFT) domain) and are related to so-called binaural or inter-channel cues. The binaural cues (reference [3], which is incorporated by reference herein in its entirety) include the inter-aural level difference (ILD), the inter-aural time difference (ITD) and the inter-aural correlation (IC). Depending on the signal characteristics, the stereo scene configuration, etc., some or all of the binaural cues are coded and sent to the decoder. Information about what binaural cues are coded and sent is sent as signaling information, which is usually part of the stereo side information. Different coding techniques can also be used to quantize the specific binaural cues, which leads to the use of a variable number of bits. Then, in addition to the quantized binaural cues, the stereo side information can contain, at medium and higher bit rates, a quantized residual signal resulting from the downmixing. The residual signal can be coded using an entropy coding technique, such as an arithmetic coder. In the present disclosure, parametric stereo coding with stereo parameters computed in a transform domain will be referred to as "DFT stereo" coding.

[0011] Another stereo coding technique is a technique that operates in the time domain (TD). This stereo coding technique mixes the two input left and right channels into so-called primary and secondary channels. For example, following the approach described in reference [4], the entire contents of which are incorporated herein by reference, the time domain mixing can be based on a mixing ratio that determines the respective contributions of the two input left and right channels when generating the primary and secondary channels. The mixing ratio is derived from several measures, for example, the normalized correlation of the input left and right channels with respect to a mono signal version or the long-term correlation difference between the two input left and right channels. The primary channel can be coded by a common mono codec, while the secondary channel can be coded by a lower bit rate codec. The secondary channel coding can exploit the coherence between the primary and secondary channels and can reuse some parameters from the primary channel. In the present disclosure, the time domain stereo coding will be referred to as "TD stereo" coding. Generally, TD stereo coding is most efficient at low and medium bit rates for coding speech signals.

[0012] A third stereo coding technique is a technique that operates in the modified discrete cosine transform (MDCT) domain. It is based on joint coding of the left and right channels, while computing a global ILD and mid / side (M / S) processing in the whitened spectrum domain. The third stereo coding technique uses several tools adapted from TCX (transform coded excitation) coding in MPEG (Motion Picture Experts Group) codecs, as described in, for example, references [6] and [7], the entire contents of which are incorporated herein by reference; these tools can include TCX core coding, TCX LTP (long term prediction) analysis, TCX noise filling, frequency domain noise shaping (FDNS), stereo intelligent gap filling (IGF), and / or adaptive bit allocation between channels. Generally, this third stereo coding technique is efficient for encoding all kinds of audio content at medium and high bit rates. In the present disclosure, the MDCT domain stereo coding technique will be referred to as "MDCT stereo coding". Generally, MDCT stereo coding is most efficient at medium and high bit rates for coding general audio signals.

[0013] In recent years, stereo coding has been further extended to multi-channel coding. There are several techniques that provide multi-channel coding, but the basic core of all these techniques is often based on a single or multiple instances of mono or stereo coding techniques. Therefore, the present disclosure proposes switching between stereo coding modes, which can be part of a multi-channel coding technique such as Metadata-Assisted Spatial Audio (MASA), as described in, for example, reference [8], the entire contents of which are hereby incorporated by reference. In the MASA approach, MASA metadata (e.g., direction, energy ratio, spread coherence, distance, surround coherence, all in several time-frequency bins) is generated, quantized, coded and passed into the bitstream in a MASA analyzer, while the MASA audio channel(s) are processed as mono or stereo transport signal(s) coded by the core codec(s). At the MASA decoder, the MASA metadata then guides the decode and rendering process to recreate the output spatial sound. SUMMARY

[0014] The present disclosure provides a stereo sound signal encoding apparatus and method as defined in the appended claims.

[0015] The foregoing and other objects, advantages and features of the stereo encoding and decoding apparatus and method will become more readily apparent upon reading the following non-limiting description of illustrative embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] In the drawings:

[0017] Figure 1 is a schematic block diagram of a possible context of a sound processing and communication system depicting implementations of a stereo encoding and decoding apparatus and method;

[0018] Figure 2 is a high-level block diagram simultaneously showing an Immersive Voice and Audio Services (IVAS) stereo encoding apparatus and a corresponding stereo encoding method, wherein the IVAS stereo encoding apparatus comprises a frequency domain (FD) stereo encoder, a time domain (TD) stereo encoder and a modified discrete cosine transform (MDCT) stereo encoder, wherein in the present illustrative embodiments and drawings, the FD stereo encoder implementation is based on a discrete Fourier transform (DFT) (hereinafter referred to as “DFT stereo encoder”);

[0019] Figure 3 is a block diagram simultaneously showing Figure 2 the DFT stereo encoder of and a corresponding DFT stereo encoding method;

[0020] Figure 4 is a block diagram simultaneously showingFigure 2 a block diagram of a TD stereo encoder and a corresponding TD stereo encoding method of

[0021] Figure 5 is a high level block diagram showing an IVAS stereo encoding device and a corresponding encoding method, wherein the IVAS stereo encoding device comprises a DFT stereo encoder, a TD stereo encoder and an MDCT stereo encoder; Figure 2 a block diagram of a MDCT stereo encoder and a corresponding MDCT stereo encoding method of

[0022] Figure 6 is a flow chart showing processing operations in an IVAS stereo encoding device and method when switching from a TD stereo mode to a DFT stereo mode;

[0023] Figure 7a-1 and Figure 7a-2 is a flow chart showing processing operations in an IVAS stereo encoding device and method when switching from a DFT stereo mode to a TD stereo mode;

[0024] Figure 7b is a flow chart showing processing operations related to a TD stereo past signal when switching from a DFT stereo mode to a TD stereo mode;

[0025] Figure 8 is a high level block diagram showing an IVAS stereo decoding device and a corresponding decoding method, wherein the IVAS stereo decoding device comprises a DFT stereo decoder, a TD stereo decoder and an MDCT stereo decoder;

[0026] Figure 9 is a flow chart showing processing operations in an IVAS stereo decoding device and method when switching from a TD stereo mode to a DFT stereo mode;

[0027] Figure 10 is a flow chart showing Figure 9 Example B) comprising updating a DFT stereo synthesis memory in a TD stereo frame on the decoder side;

[0028] Figure 11 is a flow chart showing Figure 9 Example C) comprising smoothing an output stereo synthesis in a first DFT stereo frame on the decoder side after switching from a TD stereo mode to a DFT stereo mode;

[0029] Figure 12 is a flow chart showing processing operations in an IVAS stereo decoding device and method when switching from a DFT stereo mode to a TD stereo mode;

[0030] Figure 13 is a flow chart showing Figure 12a flowchart of example A) of FIG. 1, including, on the decoder side, updating the TD stereo sync memory in the first TD stereo frame after switching from the DFT stereo mode to the TD stereo mode; and

[0031] Figure 14 is a simplified block diagram of an example configuration of hardware components implementing each of the IVAS stereo encoding and method and the IVAS stereo decoding device and method. DETAILED DESCRIPTION

[0032] As noted above, the present disclosure relates to stereo sound encoding, in particular but not exclusively to switching between stereo codec modes in sound (including speech and / or audio), the codec being in particular but not exclusively capable of producing good stereo quality at low bitrates and low delay, e.g. in complex audio scenarios. In the present disclosure, complex audio scenarios include cases where, for example but not exclusively: (a) the correlation between sound signals recorded by microphones is low, (b) there are significant fluctuations in background noise, and / or (c) there are interfering talkers. Non-limiting examples of complex audio scenarios include large anechoic conference rooms with A / B microphone configuration, small reverberant rooms with binaural microphones, and small reverberant rooms with mono / side microphone setup. All these room configurations can include fluctuating background noise and / or interfering talkers.

[0033] Figure 1 is a schematic block diagram of a stereo sound processing and communication system 100, depicting a possible context of implementation of the IVAS stereo encoding and method and the IVAS stereo decoding device and method.

[0034] Figure 1 The stereo sound processing and communication system 100 of FIG. 1 supports the transmission of stereo sound signals across a communication link 101. The communication link 101 can comprise, for example, a wire or fiber link. Alternatively, the communication link 101 can comprise at least in part a radio frequency link. Radio frequency links typically support multiple simultaneous communications that need to share bandwidth resources, such as can be found in cellular telephones. Although not shown, in a single device implementation of the system 100, the communication link 101 can be replaced by a storage device that records and stores the encoded stereo sound signals for later playback.

[0035] Still referring to Figure 1 For example, a pair of microphones 102 and 122 produces left 103 and right 123 channels of original analog stereo sound signals. As noted in the foregoing description, the sound signals can in particular but not exclusively include speech and / or audio.

[0036] The left 103 and right 123 channels of the original analog audio signal are provided to the analog-to-digital (A / D) converter 104 to convert them into the left 105 and right 125 channels of the original digital stereo audio signal. The left 105 and right 125 channels of the original digital stereo audio signal can also be recorded and provided from a storage device (not shown).

[0037] Stereo audio encoder 106 encodes and decodes the left 105 and right 125 channels of the original digital stereo audio signal, thereby generating a set of encoding and decoding parameters. These parameters are multiplexed in the form of a bitstream 107, which is delivered to an optional error correction encoder 108. The optional error correction encoder 108, when present, adds redundancy to the binary representation of the encoding and decoding parameters in the bitstream 107 before transmitting the resulting bitstream 111 via communication link 101.

[0038] On the receiving side, an optional error correction decoder 109 utilizes the aforementioned redundant information in the received digital bitstream 111 to detect and correct errors that may occur during transmission on the communication link 101, generating a bitstream 112 with the received encoding and decoding parameters. A stereo decoder 110 converts the received encoding and decoding parameters in the bitstream 112 to create synthesized left 113 and right 133 channels of a digital stereo audio signal. The left 113 and right 133 channels of the digital stereo audio signal reconstructed in the stereo decoder 110 are converted into synthesized left 114 and right 134 channels of an analog stereo audio signal in a digital-to-analog (D / A) converter 115.

[0039] The synthesized left 114 and right 134 channels of the analog stereo sound signal are played back in a pair of speaker units or binaural headphones 116 and 136, respectively. Alternatively, the left 113 and right 133 channels of the digital stereo sound signal from the stereo sound decoder 110 can also be provided to a storage device (not shown) and recorded in the storage device.

[0040] For example, (a) Figure 1 The left channel can be... Figures 2-13 The left channel is achieved, (b) Figure 1 The right channel can be controlled by Figures 2-13 The right channel is implemented, (c) Figure 1 The stereo encoder 106 can be made by Figure 2 -7 IVAS stereo encoding device implementation, and (d) Figure 1 The stereo decoder 110 can be made by Figures 8-13 The IVAS stereo decoding device is implemented.

[0041] 1. Switching between stereo modes in IVAS stereo encoding device 200 and method 250

[0042] Figure 2 This is a high-level block diagram that simultaneously shows the IVAS stereo encoding device 200 and the corresponding IVAS stereo encoding method 250. Figure 3 It is shown at the same time Figure 2 A block diagram of the FD stereo encoder 300 and the corresponding FD stereo encoding method 350 of the IVAS stereo encoding device 200. Figure 4 It is shown at the same time Figure 2 A block diagram of the TD stereo encoder 400 and the corresponding TD stereo encoding method 450 of the IVAS stereo encoding device 200, and Figure 5 It is shown at the same time Figure 2 A block diagram of the MDCT stereo encoder 500 and the corresponding MDCT stereo encoding method 550 of the IVAS stereo encoding device 200.

[0043] exist Figures 2-5 In the illustrative and non-restrictive implementation, the IVAS stereo coding device 200 (and correspondingly) Figure 8 The framework of the IVAS stereo decoding device 200 is based on a modified version of the Enhanced Voice Services (EVS) codec (see reference [1]). Specifically, the EVS codec is extended to encode and decode stereo and multi-channel audio and to handle Immersive Voice and Audio Services (IVAS). Therefore, in this disclosure, the encoding device 200 and method 250 are referred to as IVAS stereo encoding device and method. In the described exemplary implementation, the IVAS stereo encoding device 200 and method 250 use three stereo encoding and decoding modes as non-limiting examples: a frequency domain (FD) stereo mode based on DFT (Discrete Fourier Transform), referred to in this disclosure as the “DFT stereo mode”; a time domain (TD) stereo mode, referred to in this disclosure as the “TD stereo mode”; and a joint stereo encoding and decoding mode based on a modified discrete cosine transform (MDCT) stereo mode, referred to in this disclosure as the “MDCT stereo mode”. It should be remembered that other codec architectures can also be used as the basis for the framework of the IVAS stereo encoding device 200 (and corresponding IVAS stereo decoding device 800).

[0044] In the described non-limiting implementation, stereo mode switching in the IVAS codec (IVAS stereo encoding device 200 and IVAS stereo decoding device 800) refers to switching between DFT, TD and MDCT stereo modes.

[0045] 1.1 Differences between different stereo encoders and encoding methods

[0046] The following nomenclature is used in this disclosure and the accompanying drawings: lower case letters indicate time domain signals, upper case letters indicate transform domain signals, l / L denotes the left channel, r / R denotes the right channel, m / M denotes the mid channel, s / S denotes the side channel, PCh denotes the primary channel, SCh denotes the secondary channel. Furthermore, in the figures, numbers without units correspond to the number of samples at a 16 kHz sampling rate.

[0047] There are differences between (a) the DFT stereo encoder 300 and encoding method 350, (b) the TD stereo encoder 400 and encoding method 450, (c) the MDCT stereo encoder 500 and encoding method 550. Some of these differences will be summarized in the following paragraphs, at least some of which will be better explained in the following description.

[0048] The IVAS stereo encoding device 200 and encoding method 250 perform operations such as buffering one 20 ms frame of the stereo input signal (left and right channels) (it is well known in the art that stereo sound signals are processed in successive frames of a given duration containing a given number of samples of the sound signal), a small number of classification steps, downmixing, pre-processing and actual coding. A look-ahead of 8.75 ms is available, mainly for analysis, classification and overlap-add (OLA) operations in the transform domain, such as the Transform Coded Excitation (TCX) core, the High Quality (HQ) core and the Frequency Domain Bandwidth Extension (FD-BWE). These operations are described in reference [1], sections 5.3 and 5.2.6.2.

[0049] The look-ahead in the IVAS stereo encoding device 200 and encoding method 250 is 0.9375 ms shorter compared to the unmodified EVS encoder (corresponding to a Finite Impulse Response (FIR) filter resampling delay (see reference [1], section 5.1.3.1). This has an impact on the resampling process of the down-processed signal (downmixed signal for TD and DFT stereo modes) in each frame:

[0050] - DFT stereo encoder 300 and encoding method 350 : Resampling is performed in the DFT domain, therefore, no additional delay is introduced;

[0051] - TD stereo encoder 400 and encoding method 450 : FIR resampling (decimation) is performed with a delay of 0.9375 ms. Since this resampling delay is not available in the IVAS stereo encoding device 200, the resampling delay is compensated by zero-padding at the end of the downmixed signal. Therefore, the 0.9375 ms long compensated part of the downmixed signal needs to be recomputed (resampled) again in the next frame.

[0052] - MDCT stereo encoder 500 and encoding method 550 : Same as in the TD stereo encoder 400 and encoding method 450.

[0053] The resampling in the DFT stereo encoder 300, the TD stereo encoder 400 and the MDCT stereo encoder 500 is from the input sampling rate (typically 16, 32 or 48 kHz) to the internal sampling rate (typically 12.8, 16, 25.6 or 32 kHz). The resampled signals are then used in the pre-processing and core encoding.

[0054] Furthermore, the lookahead contains a part of the downmix signal (for the downmix signal of the TD and DFT stereo modes) which is not accurate but extrapolated or estimated, which also has an impact on the resampling process. The inaccuracy of the lookahead downmix signal (for the downmix signal of the TD and DFT stereo modes) depends on the current stereo coding mode:

[0055] - DFT stereo encoder 300 and encoding method 350 : The length of 8.75 ms of the lookahead corresponds to the windowed overlap part of the downmix signal related to the OLA part of the DFT analysis window, respectively the OLA part of the DFT synthesis window. In order to perform the pre-processing on as meaningful signal as possible, the correction (or de-windowing, i.e. applying the inverse window to the lookahead part) is performed on this part of the downmix signal of the lookahead. As a result, the corrected downmix signal of 8.75 ms length in the lookahead cannot be accurately reconstructed in the current frame;

[0056] - TD stereo encoder 400 and encoding method 450 : Before the time-domain (TD) downmix, an inter-channel alignment (ICA) is performed using the inter-channel time delay (ITD) synchronization between the two input channels I and r in the time domain. This is achieved by delaying one of the input channels (I or r) and by extrapolating the missing part of the downmix signal corresponding to the ITD delay length; the maximum value of the ITD delay is 7.5 ms. As a result, the extrapolated downmix signal of up to 7.5 ms length in the lookahead cannot be accurately reconstructed in the current frame.

[0057] - MDCT stereo encoder 500 and encoding method 550 : Typically no downmix or time shifting is performed, so the lookahead part of the input audio signal is typically accurate.

[0058] The corrected / extrapolated signal part in the lookahead is not affected by the actual coding but is used for analysis and classification. Therefore, the corrected / extrapolated signal part in the lookahead is recalculated in the next frame and the resulting downmix signal (for the downmix signal of the TD and DFT stereo modes) is then used for the actual coding. The length of the recalculated signal depends on the stereo mode and the coding process:

[0059] - DFT stereo encoder 300 and encoding method 350 : The 8.75 ms long signal is affected by the recalculation under both the input stereo signal sampling rate and the internal sampling rate;

[0060] - TD stereo encoder 400 and encoding method 450 : The 7.5 ms long signal is affected by the recalculation under the input stereo signal sampling rate, while the 7.5 + 0.9375 = 8.4375 ms long signal is affected by the recalculation under the internal sampling rate.

[0061] - MDCT stereo encoder 500 and encoding method 550 : No recalculation under the input stereo signal sampling rate is typically needed, while the 0.9375 ms long signal is affected by the recalculation under the internal sampling rate.

[0062] Note that here the lengths of the corrected, respectively extrapolated signal portions in the lookahead are mentioned as an illustration, while typically any other lengths can be implemented.

[0063] Additional information on the DFT stereo encoder 300 and the encoding method 350 can be found in references [2] and [3]. Additional information on the TD stereo encoder 400 and the encoding method 450 can be found in reference [4]. Additional information on the MDCT stereo encoder 500 and the encoding method 550 can be found in references [6] and [7].

[0064] 1.2 Structure of the IVAS stereo encoding device 200 and processing in the IVAS stereo encoding method 250

[0065] The following table I lists the processing operations per frame in order according to the current stereo coding mode (see also Figures 2-5 ).

[0066] Table I - Processing operations at the IVAS stereo encoding device 200.

[0067]

[0068] The IVAS stereo encoding method 250 comprises operations (not shown) to control the switching between the DFT, TD and MDCT stereo modes. To perform the switching control operations, the IVAS stereo encoding device 200 comprises a controller (not shown) to switch between the DFT, TD and MDCT stereo modes. The switching between the DFT and TD stereo modes in the IVAS stereo encoding device 200 and the encoding method 250 involves using a stereo mode switching controller (not shown) to maintain the continuity of the following input signals 1) to 5) so that these signals can be processed adequately in the IVAS stereo encoding device 200 and method 250:

[0069] 1) The input stereo signal comprising left l / L and right r / R channels, e.g. for time domain transient detection or inter-channel BWE (IC-BWE);

[0070] 2) stereo downmix signal at input stereo signal sampling rate (downmixed signal for TD and DFT stereo modes):

[0071] - DFT stereo encoder 300 and encoding method 350 : mid channel m / M;

[0072] - TD stereo encoder 400 and encoding method 450 : primary (PCh) and secondary (SCh) channels;

[0073] - MDCT stereo encoder 500 and encoding method 550 : original (no downmix) left and right channels l and r;

[0074] 3) downmix signal at 12.8 kHz sampling rate (downmixed signal for TD and DFT stereo modes) - for pre-processing;

[0075] 4) downmix signal at internal sampling rate (downmixed signal for TD and DFT stereo modes) - for core encoding;

[0076] 5) high band (HB) input signal - for bandwidth extension (BWE).

[0077] While maintaining continuity of the above signal 1) is simple, it is challenging for signals 2) - 5) due to several aspects, e.g. different downmixing, different lengths of lookahead re-computed parts, use of inter-channel alignment (ICA) only in TD stereo mode, etc.

[0078] 1.2.1 Stereo classification and stereo mode selection

[0079] The operation (not shown) that controls switching between DFT, TD and MDCT stereo modes includes the operation 255 of stereo classification and stereo mode selection, e.g. as described in reference [9], the entire contents of which are incorporated herein by reference. To perform the operation 255, the controller (not shown) that switches between DFT, TD and MDCT stereo modes includes the stereo classifier and stereo mode selector 205.

[0080] Switching between TD stereo mode, DFT stereo mode and MDCT stereo mode is responsive to stereo mode selection. Stereo classification (reference [9]) is responsive to left l and right r channels of the input stereo signal, and / or a requested codec bit rate. Stereo mode selection (reference [9]) includes selecting one of DFT, TD and MDCT stereo modes based on the stereo classification.

[0081] The stereo classifier and stereo mode selector 205 produces stereo mode signaling 270 for identifying the selected stereo codec mode.

[0082] 1.2.2 Memory allocation / release

[0083] The operation (not shown) that controls the switching between the DFT, TD, and MDCT stereo modes includes a memory allocation operation (not shown). To perform the memory allocation operation, the controller that switches between the DFT, TD, and MDCT stereo modes dynamically allocates / deallocates static memory data structures to / from the DFT, TD, and MDCT stereo modes depending on the current stereo mode. Such memory allocation keeps the static memory footprint of the IVAS stereo encoding device 200 as low as possible by only keeping those data structures that are employed in the current frame.

[0084] For example, in the first DFT stereo frame after a TD stereo frame, the data structures related to the TD stereo mode (e.g., TD stereo data processing, second core encoder data structures) are deallocated (released), and the data structures related to the DFT stereo mode (e.g., DFT stereo data structures) are allocated and initialized instead. Note that the deallocation of further unused data structures is performed first, and then the allocation of newly used data structures is performed. This order of operations is important to not increase the static memory footprint at any point in the encoding.

[0085] A summary of the main static memory data structures used in the various stereo modes is shown in Table II.

[0086] Table II - Allocation of data structures in different stereo modes.

[0087] “x” means allocation - “xx” means two allocations -

[0088] “-” means deallocation, “--” means two deallocations.

[0089]

[0090] Below is an example implementation of the memory allocation / deallocation encoder module in C source code.

[0091] void stereo_memory_enc(

[0092] CPE_ENC_HANDLE hCPE, / * i : CPE encoder structure * /

[0093] const int32_t input_Fs, / * i : input sampling rate * /

[0094] const int16_t max_bwidth, / * i : maximum audio bandwidth * /

[0095] float *tdm_last_ratio / * o : TD stereo last ratio * / )

[0097] {

[0098] Encoder_State *st;

[0099] / *-------------------------------------------------------------*

[0100] * Save parameters from the structure to be released

[0101] *-------------------------------------------------------------* /

[0102] if ( hCPE->ast_element_mode == IVAS_CPE_TD )

[0103] {

[0104] *tdm_last_ratio = hCPE->hStereoTD->tdm_last_ratio; / * Note: This must be set as a local variable before allocating / dealing the data structure. * /

[0105] }

[0106] if ( hCPE->hStereoTCA != NULL && hCPE->last_element_mode == IVAS_CPE_DFT )

[0107] {

[0108] set_s( hCPE->hStereoTCA->prevCorrLagStats, (int16_t) hCPE->hStereoDft->itd[1], 3 );

[0109] hCPE->hStereoTCA->prevRefChanIndx = ( hCPE->hStereoDft->itd[1] >= 0 ) ? ( L_CH_INDX ) : ( R_CH_INDX );

[0110] }

[0111] / *-------------------------------------------------------------*

[0112] * Allocate / Release Data Structures

[0113] *-------------------------------------------------------------* /

[0114] if ( hCPE->element_mode != hCPE->last_element_mode )

[0115] {

[0116] / *----------------------------------------------------------*

[0117] * Switch CPE mode to DFT stereo

[0118] *---------------------------------------------------------* /

[0119] if ( hCPE->element_mode == IVAS_CPE_DFT )

[0120] {

[0121] / * Data structures of the previous CPE mode before release * /

[0122] if ( hCPE->hStereoTD != NULL )

[0123] {

[0124] count_free( hCPE->hStereoTD );

[0125] hCPE->hStereoTD = NULL;

[0126] }

[0127] if ( hCPE->hStereoMdct != NULL )

[0128] {

[0129] count_free( hCPE->hStereoMdct );

[0130] hCPE->hStereoMdct = NULL;

[0131] }

[0132] / * Free the core coder secondary channel * /

[0133] deallocate_CoreCoder_enc( hCPE->hCoreCoder[1] );

[0134] / * Allocate the DFT stereo data structure * /

[0135] stereo_dft_enc_create( &( hCPE->hStereoDft ), input_Fs,max_bwidth );

[0136] / * Allocate the ICBWE structure * /

[0137] if ( hCPE->hStereoICBWE == NULL )

[0138] {

[0139] hCPE->hStereoICBWE = (STEREO_ICBWE_ENC_HANDLE) count_malloc( sizeof( STEREO_ICBWE_ENC_DATA ) );

[0140] stereo_icBWE_init_enc( hCPE->hStereoICBWE );

[0141] }

[0142] / * Allocate the HQ core in the M channel * /

[0143] st = hCPE->hCoreCoder[0];

[0144] if ( st->hHQ_core == NULL )

[0145] {

[0146] st->hHQ_core = (HQ_ENC_HANDLE) count_malloc( sizeof(HQ_ENC_DATA ) );

[0147] HQ_core_enc_init( st->hHQ_core );

[0148] }

[0149] }

[0150] / *----------------------------------------------------------*

[0151] * Switch CPE mode to TD stereo

[0152] *---------------------------------------------------------* /

[0153] if ( hCPE->element_mode == IVAS_CPE_TD )

[0154] {

[0155] / * Free data structures of previous CPE mode * /

[0156] if ( hCPE->hStereoDft!= NULL )

[0157] {

[0158] stereo_dft_enc_destroy( &( hCPE->hStereoDft ) );

[0159] hCPE->hStereoDft = NULL;

[0160] }

[0161] if ( hCPE->hStereoMdct!= NULL )

[0162] {

[0163] count_free( hCPE->hStereoMdct );

[0164] hCPE->hStereoMdct = NULL;

[0165] }

[0166] / * Free TCX / IGF structure of second channel * /

[0167] deallocate_CoreCoder_TCX_enc( hCPE->hCoreCoder[1] );

[0168] / * Allocate TD stereo data structure * /

[0169] hCPE->hStereoTD = (STEREO_TD_ENC_DATA_HANDLE) count_malloc( sizeof( STEREO_TD_ENC_DATA ) );

[0170] stereo_td_init_enc( hCPE->hStereoTD, hCPE->element_brate, hCPE->last_element_mode );

[0171] / * Allocate secondary channel * /

[0172] allocate_CoreCoder_enc( hCPE->hCoreCoder[1] );

[0173] }

[0174] / *----------------------------------------------------------*

[0175] * Allocate DFT / TD stereo structure after MDCT stereo frame

[0176] *---------------------------------------------------------* /

[0177] if ( hCPE->last_element_mode == IVAS_CPE_MDCT && ( hCPE->element_mode == IVAS_CPE_DFT || hCPE->element_mode == IVAS_CPE_TD ) )

[0178] {

[0179] / * Allocate TCA data structure * /

[0180] hCPE->hStereoTCA = (STEREO_TCA_ENC_HANDLE) count_malloc(sizeof( STEREO_TCA_ENC_DATA ) );

[0181] stereo_tca_init_enc( hCPE->hStereoTCA, input_Fs );

[0182] st = hCPE->hCoreCoder[0];

[0183] / * Allocate the main channel structure * /

[0184] allocate_CoreCoder_enc( st );

[0185] / * Allocate the CLDFB for the main channel * /

[0186] if ( st->cldfbAnaEnc == NULL )

[0187] {

[0188] openCldfb( &st->cldfbAnaEnc, CLDFB_ANALYSIS, input_Fs, CLDFB_PROTOTYPE_1_25MS );

[0189] }

[0190] / * Allocate the BWE for the main channel * /

[0191] if ( st->hBWE_TD == NULL )

[0192] {

[0193] st->hBWE_TD = (TD_BWE_ENC_HANDLE) count_malloc(sizeof( TD_BWE_ENC_DATA ) );

[0194] if ( st->cldfbSynTd == NULL )

[0195] {

[0196] openCldfb( &st->cldfbSynTd, CLDFB_SYNTHESIS,16000, CLDFB_PROTOTYPE_1_25MS );

[0197] }

[0198] InitSWBencBuffer( st->hBWE_TD );

[0199] ResetSHBbuffer_Enc( st->hBWE_TD );

[0200] st->hBWE_FD = (FD_BWE_ENC_HANDLE) count_malloc(sizeof( FD_BWE_ENC_DATA ) );

[0201] fd_bwe_enc_init( st->hBWE_FD );

[0202] }

[0203] }

[0204] / *----------------------------------------------------------*

[0205] * Switch CPE mode to MDCT stereo

[0206] *---------------------------------------------------------* /

[0207] if ( hCPE->element_mode == IVAS_CPE_MDCT )

[0208] {

[0209] int16_t i;

[0210] / * Free data structure of previous CPE mode* /

[0211] if ( hCPE->hStereoDft!= NULL )

[0212] {

[0213] stereo_dft_enc_destroy( &( hCPE->hStereoDft ) );

[0214] hCPE->hStereoDft = NULL;

[0215] }

[0216] if ( hCPE->hStereoTD != NULL )

[0217] {

[0218] count_free( hCPE->hStereoTD );

[0219] hCPE->hStereoTD = NULL;

[0220] }

[0221] if ( hCPE->hStereoTCA != NULL )

[0222] {

[0223] count_free( hCPE->hStereoTCA );

[0224] hCPE->hStereoTCA = NULL;

[0225] }

[0226] if ( hCPE->hStereoICBWE != NULL )

[0227] {

[0228] count_free( hCPE->hStereoICBWE );

[0229] hCPE->hStereoICBWE = NULL;

[0230] }

[0231] for ( i = 0; i < CPE_CHANNELS; i++ )[[ID=*]] [[ID=*]]

[0232] {

[0233] st = hCPE->hCoreCoder[i];

[0234] / * Release the core channel structure * /

[0235] deallocate_CoreCoder_enc( hCPE->hCoreCoder[i] );

[0236] }

[0237] if ( hCPE->last_element_mode == IVAS_CPE_DFT )

[0238] {

[0239] / * Allocate secondary channel * /

[0240] allocate_CoreCoder_enc( hCPE->hCoreCoder[1] );

[0241] }

[0242] / * Allocate TCX / IGF structure for secondary channel * /

[0243] st = hCPE->hCoreCoder[1];

[0244] st->hTcxEnc = (TCX_ENC_HANDLE) count_malloc( sizeof( TCX_ENC_DATA ) );

[0245] st->hTcxEnc->spectrum[0] = st->hTcxEnc->spectrum_long;

[0246] st->hTcxEnc->spectrum[1] = st->hTcxEnc->spectrum_long +N_TCX10_MAX;

[0247] set_f( st->hTcxEnc->old_out, 0, L_FRAME32k );

[0248] set_f( st->hTcxEnc->spectrum_long, 0, N_MAX );

[0249] if ( hCPE->last_element_mode == IVAS_CPE_DFT )

[0250] {

[0251] st->last_core = ACELP_CORE; / * Need to set TCX core in SetTCXModeInfo() * /

[0252] }

[0253] st->hTcxCfg = (TCX_CONFIG_HANDLE) count_malloc( sizeof(TCX_config ) );

[0254] st->hIGFEnc = (IGF_ENC_INSTANCE_HANDLE) count_malloc(sizeof( IGF_ENC_INSTANCE ) );

[0255] st->igf = getIgfPresent( st->element_mode, st->total_brate, st->bwidth, st->rf_mode );

[0256] / * Allocate and initialize the MDCT stereo structure * /

[0257] hCPE->hStereoMdct = (STEREO_MDCT_ENC_DATA_HANDLE) count_malloc( sizeof( STEREO_MDCT_ENC_DATA ) );

[0258] initMdctStereoEncData( hCPE->hStereoMdct, hCPE->element_brate, hCPE->hCoreCoder[0]->max_bwidth, SMDCT_MS_DECISION, 0, NULL );

[0259] }

[0260] }

[0261] return;

[0262] }

[0263] 1.2.3 Setting the TD stereo mode

[0264] The TD stereo mode can include two sub-modes. One is the so-called normal TD stereo sub-mode, whose TD stereo mix ratio is higher than 0 and lower than 1. The other is the so-called LRTD stereo sub-mode, whose TD stereo mix ratio is 0 or 1; thus, LRTD is one extreme case of the TD stereo mode, in which the TD downmix actually does not mix the content of the time-domain left l and right r channels to form the primary PCh and secondary SCh channels, but directly obtains them from the channels l and r.

[0265] When both sub-modes of the TD stereo mode (normal and LRTD) are available, the stereo mode switching operation (not shown) includes the TD stereo mode setup (not shown). To perform the TD stereo mode setup which forms part of the memory allocation, the stereo mode switching controller (not shown) of the IVAS stereo encoding device 200 allocates / releases certain static memory data structures when switching between the normal TD stereo mode and the LRTD stereo mode. For example, the IC-BWE data structure (see Table II) is allocated only in frames using the normal TD stereo mode, while several data structures (complex low delay filter bank (CLDFB) and BWE for the secondary channel SCh) are allocated only in frames using the LRTD stereo mode (see Table II). Below is an example implementation of the memory allocation / release encoder modules in C source code:

[0266] / * Normal TD / LRTD switching * /

[0267] if ( hCPE->hStereoTD->tdm_LRTD_flag == 0 )

[0268] {

[0269] Encoder_State *st;

[0270] st = hCPE->hCoreCoder[1];

[0271] / * Release CLDFB ana for secondary channel * /

[0272] if ( st->cldfbAnaEnc!= NULL )

[0273] {

[0274] deleteCldfb( &st->cldfbAnaEnc );

[0275] }

[0276] / * Release BWE for secondary channel * /

[0277] if ( st->hBWE_TD!= NULL )

[0278] {

[0279] if ( st->hBWE_TD!= NULL )

[0280] {

[0281] count_free( st->hBWE_TD );

[0282] st->hBWE_TD = NULL;

[0283] }

[0284] deleteCldfb( &st->cldfbSynTd );

[0285] if ( st->hBWE_FD != NULL )

[0286] {

[0287] count_free( st->hBWE_FD );

[0288] st->hBWE_FD = NULL;

[0289] }

[0290] }

[0291] / * Allocate ICBWE structure * /

[0292] if ( hCPE->hStereoICBWE == NULL )

[0293] {

[0294] ( hCPE->hStereoICBWE = (STEREO_ICBWE_ENC_HANDLE)count_malloc( sizeof( STEREO_ICBWE_ENC_DATA ) );

[0295] stereo_icBWE_init_enc( hCPE->hStereoICBWE );

[0296] }

[0297] }

[0298] else / * tdm_LRTD_flag == 1 * /

[0299] {

[0300] Encoder_State *st;

[0301] st = hCPE->hCoreCoder[1];

[0302] / * Release ICBWE structure * /

[0303] if ( hCPE->hStereoICBWE != NULL )

[0304] {

[0305] / * Copy the past input signal for use in BWE * /

[0306] mvr2r( hCPE->hStereoICBWE->dataChan[1], hCPE->hCoreCoder[1]->old_input_signal, st->input_Fs / 50 );

[0307] count_free( hCPE->hStereoICBWE );

[0308] hCPE->hStereoICBWE = NULL;

[0309] }

[0310] / * Allocate the CLDFB ana for the secondary channel * /

[0311] if ( st->cldfbAnaEnc == NULL )

[0312] {

[0313] openCldfb( &st->cldfbAnaEnc, CLDFB_ANALYSIS, st->input_Fs, CLDFB_PROTOTYPE_1_25MS );

[0314] }

[0315] / * Allocate the BWE for the secondary channel * /

[0316] if ( st->hBWE_TD == NULL )

[0317] {

[0318] st->hBWE_TD = (TD_BWE_ENC_HANDLE) count_malloc(sizeof( TD_BWE_ENC_DATA ) );

[0319] openCldfb( &st->cldfbSynTd, CLDFB_SYNTHESIS, 16000,CLDFB_PROTOTYPE_1_25MS );

[0320] InitSWBencBuffer( st->hBWE_TD );

[0321] ResetSHBbuffer_Enc( st->hBWE_TD );

[0322] st->hBWE_FD = (FD_BWE_ENC_HANDLE) count_malloc(sizeof( FD_BWE_ENC_DATA ) );

[0323] fd_bwe_enc_init( st->hBWE_FD );

[0324] }

[0325] }

[0326] Primarily, only the normal TD stereo mode (further referred to as simply TD stereo mode for simplicity) will be described in detail in this disclosure. The LRTD stereo mode is mentioned as one possible implementation.

[0327] 1.2.4 Stereo mode switch update

[0328] The stereo mode switch control operation (not shown) comprises a stereo switch update operation (not shown). To perform this stereo switch update operation, the stereo mode switch controller (not shown) updates the long-term parameters and updates or resets the past buffer memories.

[0329] When switching from the DFT stereo mode to the TD stereo mode, the stereo mode switch controller (not shown) resets the TD stereo and ICA static memory data structures. These data structures store the parameters and memories of the TD stereo analysis and weighting (201 in Figure 2 Figure 4 of the ICA algorithm (401 in the DFT stereo processing and downmix module (300 in Fig. 3). Then, the stereo mode switch controller (not shown) sets the TD stereo past frame mixing ratio index according to the normal TD stereo mode or the LRTD stereo mode. As a non-limiting illustrative example:

[0330] - the previous frame mixing ratio index is set to 15, indicating that in the normal TD stereo mode the mid / middle channel m / M of the downmix is coded as the primary channel PCh with a mixing ratio of 0.5; or

[0331] - the previous frame mixing ratio index is set to 31, indicating that in the LRTD stereo mode the left channel I is coded as the primary channel PCh.

[0332] When switching from the TD stereo mode to the DFT stereo mode, the stereo mode switch controller (not shown) resets the DFT stereo data structure. This DFT stereo data structure stores the parameters and memories of the DFT stereo processing and downmix module (300 in Fig. 3). Then, the stereo mode switch controller (not shown) sets the DFT stereo past frame mixing ratio index according to the DFT stereo mode. As a non-limiting illustrative example:​Figure 3 parameters and memories related to the DFT stereo mode (see

[0333] In addition, a stereo mode switching controller (not shown) transfers some stereo related parameters between the data structures. As an example, parameters related to the time shift and energy between the sound channels l and r, i.e. the side gain (or ILD parameter) and the ITD parameter of the DFT stereo mode, are used to update the target gain and the correlation lag (ICA parameter 202) of the TD stereo mode and vice versa. These target gains and correlation lags are further described in the next section 1.2.5 of the present disclosure.

[0334] The updates / resetting related to the core encoder (see Figure 3 and 4 ) are described later in section 1.4 of the present disclosure. An example implementation of the processing of some memories in the encoder is shown below.

[0335] void stereo_switching_enc(

[0336] CPE_ENC_HANDLE hCPE, / * i : CPE encoder structure * /

[0337] float old_input_signal_pri[], / * i : old input signal of primary channel * /

[0338] const int16_t input_frame / * i : input frame length* / )

[0340] {

[0341] int16_t i, n, dft_ovl, offset;

[0342] float tmpF;

[0343] Encoder_State **st;

[0344] st = hCPE->hCoreCoder;

[0345] dft_ovl = STEREO_DFT_OVL_MAX * input_frame / L_FRAME48k;

[0346] / * update DFT analysis overlap memory * /

[0347] if ( hCPE->element_mode > IVAS_CPE_DFT && hCPE->input_mem[0]!= NULL )

[0348] {

[0349] for ( n = 0; n < CPE_CHANNELS; n++ )

[0350] {

[0351] mvr2r( st[n]->input + input_frame - dft_ovl, hCPE->input_mem[n], dft_ovl );

[0352] }

[0353] }

[0354] / * TD / MDCT -> DFT stereo switch * /

[0355] if ( hCPE->element_mode == IVAS_CPE_DFT && hCPE->last_element_mode!= IVAS_CPE_DFT )

[0356] {

[0357] / * window DFT synthesis overlap memory @input_fs, primary channel * /

[0358] for ( i = 0; i < dft_ovl; i++ )

[0359] {

[0360] hCPE->hStereoDft->output_mem_dmx[i] = old_input_signal_pri[input_frame - dft_ovl + i] * hCPE->hStereoDft->win[dft_ovl - 1 - i];

[0361] }

[0362] / * reset 48kHz BWE overlap memory * /

[0363] set_f( hCPE->hStereoDft->output_mem_dmx_32k, 0, STEREO_DFT_OVL_32k );

[0364] stereo_dft_enc_reset( hCPE->hStereoDft );

[0365] / * Update ITD parameters * /

[0366] if ( hCPE->element_mode == IVAS_CPE_DFT && hCPE->last_element_mode == IVAS_CPE_TD )

[0367] {

[0368] set_f( hCPE->hStereoDft->itd, hCPE->hStereoTCA->prevCorrLagStats[2], STEREO_DFT_ENC_DFT_NB );

[0369] }

[0370] / * Update side_gain[] parameters * /

[0371] if ( hCPE->hStereoTCA!= NULL && hCPE->last_element_mode!=IVAS_CPE_MDCT )

[0372] {

[0373] tmpF = usdequant( hCPE->hStereoTCA->indx_ica_gD, STEREO_TCA_GDMIN, STEREO_TCA_GDSTEP );

[0374] for ( i = 0; i < STEREO_DFT_BAND_MAX; i++ )

[0375] {

[0376] hCPE->hStereoDft->side_gain[STEREO_DFT_BAND_MAX + i]= tmpF;

[0377] }

[0378] }

[0379] / * Do not allow differential coding for DFT side parameters * /

[0380] hCPE->hStereoDft->ipd_counter = STEREO_DFT_FEC_THRESHOLD;

[0381] hCPE->hStereoDft->res_pred_counter = STEREO_DFT_FEC_THRESHOLD;

[0382] / * Update DFT synthesis overlap memory @12.8kHz * /

[0383] for ( i = 0; i < STEREO_DFT_OVL_12k8; i++ )

[0384] {

[0385] hCPE->hStereoDft->output_mem_dmx_12k8[i] = st[0]->buf_speech_enc[L_FRAME32k + L_FRAME - STEREO_DFT_OVL_12k8 + i] * hCPE->hStereoDft->win_12k8[STEREO_DFT_OVL_12k8 - 1 - i];

[0386] }

[0387] / * Update only DFT synthesis overlap memory @16kHz, primary channel * /

[0388] lerp( hCPE->hStereoDft->output_mem_dmx, hCPE->hStereoDft->output_mem_dmx_16k, STEREO_DFT_OVL_16k, dft_ovl );

[0389] / * Reset DFT synthesis overlap memory @8kHz, secondary channel * /

[0390] set_f( hCPE->hStereoDft->output_mem_res_8k, 0, STEREO_DFT_OVL_8k );

[0391] hCPE->vad_flag[1] = 0;

[0392] }

[0393] / * DFT / MDCT -> TD stereo switch * /

[0394] if ( hCPE->element_mode == IVAS_CPE_TD && hCPE->last_element_mode!= IVAS_CPE_TD )

[0395] {

[0396] hCPE->hStereoTD->tdm_last_ratio_idx = LRTD_STEREO_MID_IS_PRIM;

[0397] hCPE->hStereoTD->tdm_last_ratio_idx_SM = LRTD_STEREO_MID_IS_PRIM;

[0398] hCPE->hStereoTD->tdm_last_SM_flag = 0;

[0399] hCPE->hStereoTD->tdm_last_inst_ratio_idx = LRTD_STEREO_MID_IS_PRIM;

[0400] / * DFT frame after the first frame and the content is irrelevant or xtalk -> primary channel forced to the left * /

[0401] if ( hCPE->hStereoClassif->lrtd_mode == 1 )

[0402] {

[0403] hCPE->hStereoTD->tdm_last_ratio = ratio_table[LRTD_STEREO_LEFT_IS_PRIM];

[0404] hCPE->hStereoTD->tdm_last_ratio_idx = LRTD_STEREO_LEFT_IS_PRIM;

[0405] if ( hCPE->hStereoTCA->instTargetGain < 0.05f && ( hCPE->vad_flag[0] || hCPE->vad_flag[1] ) ) / * but if no content in left channel -> force primary channel to the right * /

[0406] {

[0407] hCPE->hStereoTD->tdm_last_ratio = ratio_table[LRTD_STEREO_RIGHT_IS_PRIM];

[0408] hCPE->hStereoTD->tdm_last_ratio_idx = LRTD_STEREO_RIGHT_IS_PRIM;

[0409] }

[0410] }

[0411] }

[0412] / * DFT -> TD stereo switch * /

[0413] if ( hCPE->element_mode == IVAS_CPE_TD && hCPE->last_element_mode== IVAS_CPE_DFT )

[0414] {

[0415] offset = st[0]->cldfbAnaEnc->p_filter_length - st[0]->cldfbAnaEnc->no_channels;

[0416] mvr2r( old_input_signal_pri + input_frame - offset - NS2SA(input_frame * 50, L_MEM_RECALC_TBE_NS ), st[0]->cldfbAnaEnc->cldfb_state,offset );

[0417] cldfb_reset_memory( st[0]->cldfbSynTd );

[0418] st[0]->currEnergyLookAhead = 6.1e-5f;

[0419] if ( hCPE->hStereoICBWE == NULL )

[0420] {

[0421] offset = st[1]->cldfbAnaEnc->p_filter_length - st[1]->cldfbAnaEnc->no_channels;

[0422] if ( hCPE->hStereoTD->tdm_last_ratio_idx == LRTD_STEREO_LEFT_IS_PRIM )

[0423] {

[0424] v_multc( hCPE->hCoreCoder[1]->old_input_signal +input_frame - offset - NS2SA( input_frame * 50, L_MEM_RECALC_TBE_NS ), -1.0f,st[1]->cldfbAnaEnc->cldfb_state, offset );

[0425] }

[0426] else

[0427] {

[0428] mvr2r( hCPE->hCoreCoder[1]->old_input_signal + input_frame - offset - NS2SA( input_frame * 50, L_MEM_RECALC_TBE_NS ), st[1]->cldfbAnaEnc->cldfb_state, offset );

[0429] }

[0430] cldfb_reset_memory( st[​​​​​​​​​​​​​​set_zero( st[1]->old_inp_12k8, L_INP_MEM );

[0436] / * set_zero( st[1]->old_inp_16k, L_INP_MEM ); * /

[0437] set_zero( st[1]->mem_decim, 2 * L_FILT_MAX );

[0438] / * set_zero( st[1]->mem_decim16k, 2*L_FILT_MAX ); * /

[0439] st[1]->mem_preemph = 0;

[0440] / * st[1]->mem_preemph16k = 0; * /

[0441] set_zero( st[1]->buf_speech_enc, L_PAST_MAX_32k + L_FRAME32k+ L_NEXT_MAX_32k );

[0442] set_zero( st[1]->buf_speech_enc_pe, L_PAST_MAX_32k + L_FRAME32k + L_NEXT_MAX_32k );

[0443] if ( st[1]->hTcxEnc!= NULL )

[0444] {

[0445] set_zero( st[1]->hTcxEnc->buf_speech_ltp, L_PAST_MAX_32k+ L_FRAME32k + L_NEXT_MAX_32k );

[0446] }

[0447] set_zero( st[1]->buf_wspeech_enc, L_FRAME16k + L_SUBFR + L_FRAME16k + L_NEXT_MAX_16k );

[0448] set_zero( st[1]->buf_synth, OLD_SYNTH_SIZE_ENC + L_FRAME32k);

[0449] st[1]->mem_wsp = 0.0f;

[0450] st[1]->mem_wsp_enc = 0.0f;

[0451] init_gp_clip( st[1]->clip_var );

[0452] set_f( st[1]->Bin_E, 0, L_FFT );

[0453] set_f( st[1]->Bin_E_old, 0, L_FFT / 2 );

[0454] / * The allocation handled by st[1]->hLPDmem has been reset * /

[0455] st[1]->last_L_frame = st[0]->last_L_frame;

[0456] pitch_ol_init( &st[1]->old_thres, &st[1]->old_pitch, &st[1]->delta_pit, &st[1]->old_corr );

[0457] set_zero( st[1]->old_wsp, L_WSP_MEM );

[0458] set_zero( st[1]->old_wsp2, ( L_WSP_MEM - L_INTERPOL ) / OPL_DECIM );

[0459] set_zero( st[1]->mem_decim2, 3);

[0460] st[1]->Nb_ACELP_frames = 0;

[0461] / * Fill PCh memory into SCh * /

[0462] mvr2r( st[0]->hLPDmem->old_exc, st[1]->hLPDmem->old_exc, L_EXC_MEM );

[0463] mvr2r( st[0]->lsf_old, st[1]->lsf_old, M );

[0464] mvr2r( st[0]->lsp_old, st[1]->lsp_old, M );

[0465] mvr2r( st[0]->lsf_old1, st[1]->lsf_old1, M );

[0466] mvr2r( st[0]->lsp_old1, st[1]->lsp_old1, M );

[0467] st[1]->GSC_noisy_speech = 0;

[0468] }

[0469] else if ( hCPE->element_mode == IVAS_CPE_TD && hCPE->last_element_mode == IVAS_CPE_MDCT )

[0470] {

[0471] set_f( st[0]->hLPDmem->old_exc, 0.0f, L_EXC_MEM );

[0472] set_f( st[1]->hLPDmem->old_exc, 0.0f, L_EXC_MEM );

[0473] }

[0474] 1.2.5 ICA encoder

[0475] In a TD stereo frame, the stereo mode switch control operation (not shown) includes a time inter-channel alignment (ICA) operation 251. To perform operation 251, the stereo mode switch controller (not shown) includes an ICA encoder 201 to time align channels l and r of the input stereo signal and then scale channel r.

[0476] As described in the foregoing description, before mixing in TD, ICA is performed using ITD synchronization between the two input channels l and r in the time domain. This is achieved by delaying one of the input channels (l or r) and by extrapolating the missing part of the downmix signal corresponding to the length of the ITD delay; the maximum value of the ITD delay is 7.5 ms. First, a time alignment, i.e. an ICA time shift, is applied, which changes the bulk of the current TD stereo frame. The extrapolated part of the downmix signal is re-computed and is thus time adjusted in the next frame based on the ITD estimated in this next frame.

[0477] When no stereo mode switch is expected, the 7.5 ms long extrapolated signal is re-computed in the ICA encoder 201. However, when a stereo mode switch can occur, i.e. from DFT stereo mode to TD stereo mode, a longer signal will be re-computed. This length then corresponds to the length of the DFT stereo correction signal plus the FIR resampling delay, i.e. 8.75 ms + 0.9375 ms = 9.6875 ms. These features are explained in more detail in section 1.4.

[0478] Another purpose of the ICA encoder 201 is to scale the input channel r. Whatever the DFT or TD stereo mode used, the scaling gain, i.e. the above-mentioned target gain, is estimated as the log ratio of the l and r channel energies smoothed with the target gain of the previous frame at each frame. The target gain estimated in the current frame (20 ms) is applied to the last 15 ms of the current input channel r, while the first 5 ms of the current channel r is scaled in a cross-fade manner by a combination of the previous and current frame target gains.

[0479] The ICA encoder 201 produces ICA parameters 202, such as the ITD delay, the target gain and the target channel index.

[0480] 1.2.6 Time domain transient detector

[0481] In the same way, the stereo mode switch control operation (not shown) comprises an operation 254 of detecting a time-domain transient in the channel r from the ICA encoder 201. To perform operation 254, the stereo mode switch controller (not shown) comprises a detector 204 to detect a time-domain transient in the channel r.

[0482] In the same way, the stereo mode switch control operation (not shown) comprises an operation 254 of detecting a time-domain transient in the channel r from the ICA encoder 201. To perform operation 254, the stereo mode switch controller (not shown) comprises a detector 204 to detect a time-domain transient in the channel r.

[0483] The time-domain transient detection in the time-domain channels l and r is a pre-processing step that enables to detect and thus to properly handle and encode such transients in the transform-domain core coding modules (TCX core, HQ core, FD-BWE).

[0484] Further information on the time-domain transient detectors 203 and 204 and the time-domain transient detection operations 253 and 254 can be found for example in the reference [1], clause 5.1.8.

[0485] 1.2.7 Stereo encoder configuration

[0486] To perform the stereo encoder configuration, the IVAS stereo encoding device 200 sets the parameters of the stereo encoders 300, 400 and 500. For example, the nominal bit rate of the core encoders is set.

[0487] 1.2.8 DFT analysis, stereo processing in DFT domain and downmix, and IDFT synthesis

[0488] Reference Figure 3 The DFT stereo encoding method 350 comprises an operation 351 for applying a DFT transform to the channel l from the time-domain transient detector 203 of the Figure 2 To perform the operation 351, the DFT stereo encoder 300 comprises a calculator 301 of a DFT transform (DFT analysis) of the channel l to produce the channel L in the DFT domain.

[0489] The DFT stereo encoding method 350 further comprises an operation 352 for applying a DFT transform to the channel r from the time-domain transient detector 204 of the Figure 2 To perform the operation 352, the DFT stereo encoder 300 comprises a calculator 302 of a DFT transform (DFT analysis) of the channel r to produce the channel R in the DFT domain.

[0490] The DFT stereo encoding method 350 further comprises an operation 353 of stereo processing and downmixing in the DFT domain. For performing the operation 353, the DFT stereo encoder 300 comprises a stereo processor and downmixer 303 to produce side information on a side channel S. The downmixing of the channels L and R also produces a residual signal on the side channel S. The side information and the residual signal from the side channel S are coded, e.g., using a coding operation 354 and a corresponding encoder 304, and then multiplexed in the output bitstream 310 of the DFT stereo encoder 300. The stereo processor and downmixer 303 also downmix the left L and right R channels from the DFT calculators 301 and 302 to produce a mid channel M in the DFT domain. Further information on the operation 353 of stereo processing and downmixing, the stereo processor and downmixer 303, the mid channel M and the side information and the residual signal from the side channel S can be found, e.g., in reference [3].

[0491] In an inverse DFT (IDFT) synthesis operation 355 of the DFT stereo encoding method 350, the calculator 305 of the DFT stereo encoder 300 computes an IDFT transform m of the mid channel M at the sampling rate of the input stereo signal, e.g., 12.8 kHz. In the same way, in an inverse DFT (IDFT) synthesis operation 356 of the DFT stereo encoding method 350, the calculator 306 of the DFT stereo encoder 300 computes an IDFT transform m of the channel M at the internal sampling rate.

[0492] 1.2.9 TD analysis and downmix in TD domain

[0493] Reference Figure 4 The TD stereo encoding method 450 comprises an operation 451 of time domain analysis and weighted downmixing in the TD domain. For performing the operation 451, the TD stereo encoder 400 comprises a time domain analyzer and downmixer 401 to compute stereo side parameters 402, such as submode flags, mixing ratio indices or linear prediction reuse flags, which are multiplexed in the output bitstream 410 of the TD stereo encoder 400. The time domain analyzer and downmixer 401 also performs a weighted downmixer of the channels l and r from the detectors 203 and 204 (see Fig. 2) to produce a primary channel PCh and a secondary channel SCh using the estimated mixing ratios aligned with the ICA scaling. Further information on the time domain analyzer and downmixer 401 and the operation 451 can be found, e.g., in reference [4]. Figure 2

[0494] ​Downmixing using the current frame mixing ratio is performed for example on the last 15 ms of the current frame of input channels l and r, while the first 5 ms of the current frame are downmixed in a cross-fade manner using a combination of the previous and current frame mixing ratios to smooth the transition from one channel to the other. The two channels (primary channel PCh and secondary channel SCh) which are downsampled at the stereo input channel sampling rate (e.g. 32 kHz) are resampled at 12.8 kHz and an internal sampling rate using FIR decimation filters to their representation.

[0495] In the TD stereo mode, not only the stereo input signal of the current frame is downmixed. In addition, the stored downmixed signal corresponding to the previous frame is downmixed again. The length of the previous signal re-computed by this re-computation corresponds to the length of the time-shifted signal re-computed in the ICA module, i.e. 8.75 ms + 0.9375 ms = 9.6875 ms.

[0496] 1.2.10 Pre- pre-processing

[0497] In the IVAS codec (IVAS stereo encoding device 200 and IVAS stereo decoding device 800), there is a reconstruction of the traditional pre-processing so that some classification decisions are made on the overall codec bit rate, while other decisions are made according to the core encoding bit rate. Thus, the traditional pre-processing used for example in the EVS codec (reference [1]) is split into two parts to ensure that the best possible codec configuration is used in each processed frame. Thus, the codec configuration can change from frame to frame, while certain changes in the configuration can be made as quickly as possible, for example those based on signal activity or signal class. On the other hand, some changes in the codec configuration should not occur too often, for example the selection of the coded audio bandwidth, the selection of the internal sampling rate or the allocation of the bit budget between the low and high band coding; too frequent changes in such codec configurations would lead to unstable codec signal quality and even audible artifacts.

[0498] The first part of the pre-processing, the front-end pre-processing, can include pre-processing and classification modules such as resampling at the pre-processing sampling rate, spectral analysis, bandwidth detection (BWD), sound activity detection (SAD), linear prediction (LP) analysis, open-loop pitch search, signal classification, speech / music classification. Note that the decisions in the front-end pre-processing are entirely dependent on the overall codec bit rate. Further information on the operations performed during the above-mentioned pre-processing can be found for example in reference [1].

[0499] In the DFT stereo mode ( Figure 3In the DFT stereo encoder 300, preprocessing is performed on the middle channel m in the time domain by the preprocessing processor 307 and the corresponding preprocessing operation 357 at the internal sampling rate from the IDFT calculator 306.

[0500] In TD stereo mode, pre-processing is performed by (a) pre-processor 403 and corresponding pre-processing operation 453 on the main channel PCh from the time domain analyzer and downmixer 401, and (b) pre-processor 404 and corresponding pre-processing operation 454 on the sub-channel SCh from the time domain analyzer and downmixer 401.

[0501] In MDCT stereo mode, preprocessing is performed by preprocessor 503 and corresponding preprocessing operation 553 on the transient detector 203 (in the time domain). Figure 2 (a) The input left channel l is executed, and (b) the preprocessor 504 and the corresponding preprocessing operation 554 are applied to the input from the time-domain transient detector 204. Figure 2 The input to the right channel r is executed.

[0502] 1.2.11 Core encoder configuration

[0503] The core encoder configuration is based on the overall bit rate of the codec and preprocessing.

[0504] Specifically, in the DFT stereo encoder 300 and the corresponding DFT stereo encoding method 350 ( Figure 3 In the core encoder configuration, core encoder configurator 308 and corresponding core encoder configuration operation 358 respond to the output of the middle channel m in the time domain from IDFT calculator 305 and from preprocessor 307 to configure core encoder 311 and corresponding core encoding operation 361. Core encoder configurator 308 is responsible for, for example, setting the internal sample rate and / or modifying the core encoder type classification. Further information on core encoder configuration in the DFT domain can be found, for example, in references [1] and [2].

[0505] In the TD stereo encoder 400 and the corresponding TD stereo encoding method 450 ( Figure 4) the core encoder configurator 405 and the corresponding core encoder configuration operation 455 are responsive to the pre-processor primary channel PCh and secondary channel SCh from the pre-processors 403 and 404, respectively, to perform the configuration of the core encoder 406 and the corresponding core encoding operation 456 of the primary channel PCh and the corresponding core encoding operation 457 of the secondary channel SCh. The core encoder configurator 405 is responsible, for example, for setting the internal sampling rate and / or modifying the core encoder type classification. Further information on core encoder configuration in the TD domain can be found, for example, in the references [1] and [4].

[0506] 1.2.12 Further pre-processing

[0507] The DFT encoding method 350 comprises a further pre-processing operation 362. To perform operation 362, the so-called further pre-processor 312 of the DFT stereo encoder 300 performs a second part of the pre-processing, which can include classification, core selection, pre-processing at the encoding internal sampling rate, etc. The decision in the pre-processor 307 is dependent on the core encoding bit rate, which typically fluctuates during a session. Additional information on the operations performed during such further pre-processing in the DFT domain can be found, for example, in the reference [1].

[0508] The TD encoding method 450 comprises a further pre-processing operation 458. To perform operation 458, the so-called further pre-processor 408 of the TD stereo encoder 400 performs a second part of the pre-processing, which can include classification, core selection, pre-processing at the encoding internal sampling rate, etc., before core encoding of the primary channel PCh. The decision in the further pre-processor 408 is dependent on the core encoding bit rate, which typically fluctuates during a session.

[0509] And, the TD encoding method 450 comprises a further pre-processing operation 459. To perform operation 459, the TD stereo encoder 400 comprises a so-called further pre-processor 409 for performing a second part of the pre-processing, which can include classification, core selection, pre-processing at the encoding internal sampling rate, etc., before core encoding of the secondary channel SCh. The decision in the further pre-processor 409 is dependent on the core encoding bit rate, which typically fluctuates during a session.

[0510] Additional information on such further pre-processing in the TD domain can be found, for example, in the reference [1].

[0511] The MDCT encoding method 550 comprises an operation 555 of further pre-processing the left channel l. To perform operation 555, a so-called further pre-processor 505 of the MDCT stereo encoder 500 performs a second part of the pre-processing of the left channel l, which can comprise classification, core selection, pre-processing at the encoding internal sampling rate, etc., prior to the operation 556 of joint core encoding of the left channel l and the right channel r performed by the joint core encoder 506 of the MDCT stereo encoder 500.

[0512] The MDCT encoding method 550 comprises an operation 557 of further pre-processing the right channel r. To perform operation 557, a so-called further pre-processor 507 of the MDCT stereo encoder 500 performs a second part of the pre-processing of the left channel l, which can comprise classification, core selection, pre-processing at the encoding internal sampling rate, etc., prior to the operation 556 of joint core encoding of the left channel l and the right channel r performed by the joint core encoder 506 of the MDCT stereo encoder 500.

[0513] Additional information on such further pre-processing in the MDCT domain can be found, for example, in reference [1].

[0514] 1.2.13 Core encoding

[0515] Generally, the core encoders 311 (performing core encoding operation 361) in the DFT stereo encoder 300 and 406 (performing core encoding operation 456) and 407 (performing core encoding operation 457) in the TD stereo encoder 400 can be any variable bit rate mono codec. In the illustrative implementation of the present disclosure, the EVS codec (see reference [1]) with fluctuating bit rate capability (see reference [5]) is used. Of course, other suitable codecs can also be considered and implemented. In the MDCT stereo encoder 500, a joint core encoder 506 is employed, which generally can be a stereo codec module with stereo tools that process and quantize the l and r channels in a joint manner.

[0516] 1.2.14 General stereo update

[0517] Finally, a general stereo update is performed. Further information on the general stereo update can be found, for example, in reference [1].

[0518] 1.2.15 Bitstream

[0519] References Figure 2 and Figure 3, stereo mode signaling 270 from the stereo classifier and stereo mode selector 205, bitstream 313 from the side information, residual signal encoder 304, and bitstream 314 from the core encoder 311 are multiplexed to form the DFT stereo encoder bitstream 310 (which then forms the output bitstream 206 of the IVAS stereo encoding device 200 ( Figure 2 ) ).

[0520] Referring to Figure 2 and Figure 4 , stereo mode signaling 270 from the stereo classifier and stereo mode selector 205, side parameters 402 from the time domain analyzer and downmixer 401, ICA parameters 202 from the ICA encoder 201, bitstream 411 from the core encoder 406, and bitstream 412 from the core encoder 407 are multiplexed to form the TD stereo encoder bitstream 410 (which then forms the output bitstream 206 of the IVAS stereo encoding device 200 ( Figure 2 ) ).

[0521] Referring to Figure 2 and Figure 5 , stereo mode signaling 270 from the stereo classifier and stereo mode selector 205 and bitstream 509 from the joint core encoder 506 are multiplexed to form the MDCT stereo encoder bitstream 508 (which then forms the output bitstream 206 of the IVAS stereo encoding device 200 ( Figure 2 ) ).

[0522] 1.3 Switching from TD stereo mode to DFT stereo mode in the IVAS stereo encoding device 200

[0523] Switching from TD stereo mode (TD stereo encoder 400) to DFT stereo mode (DFT stereo encoder 300) is relatively simple, as shown in Figure 6 .

[0524] In particular, Figure 6 is a flowchart showing the processing operations in the IVAS stereo encoding device 200 and method 250 when switching from TD stereo mode to DFT stereo mode. As can be seen, Figure 5 shows two frames of the stereo input signal, namely, a TD stereo frame 601 followed by a DFT stereo frame 602, with different processing operations and related time instances when switching from TD stereo mode to DFT stereo mode.

[0525] A sufficiently long look-ahead is available, resampling in the DFT domain (thus no FIR decimation filter memory handling), and there is a transition from the two core encoders 406 and 407 in the last TD stereo frame 501 to one core encoder 311 in the first DFT stereo frame 502.

[0526] In response to the stereo mode selection, the above stereo mode switch controller (not shown) performs the following operations performed when switching from the TD stereo mode (TD stereo encoder 400) to the DFT stereo mode (DFT stereo encoder 300).

[0527] Figure 6 Example A) refers to the update of the DFT analysis memory, in particular the DFT stereo OLA analysis memory as part of the DFT stereo data structure, which is windowed before the DFT computation operations 351 and 352. This update is done by the stereo mode switch controller (not shown) before the inter-channel alignment (ICA) (see Figure 2 in 251) and includes storing samples related to the last 8.75 ms of the current TD stereo frame 601 of the channel l and r of the input stereo signal. This update is done in each TD stereo frame in both channels l and r. Further information on the DFT analysis memory can be found e.g. in the references [1] and [2].

[0528] Figure 6 Example B) refers to the update of the DFT synthesis memory, in particular the OLA synthesis memory as part of the DFT stereo data structure, caused by windowing after the IDFT computation operations 355 and 356 when switching from the TD stereo mode to the DFT stereo mode. The stereo mode switch controller (not shown) performs this update in the first DFT stereo frame 602 after the TD stereo frame 601 and, for this update, uses the TD stereo memory as part of the TD stereo data structure and for the TD stereo processing corresponding to the downmix primary channel PCH. Further information on the DFT synthesis memory can be found e.g. in the references [1] and [2], further information on the TD stereo memory can be found e.g. in the reference [4].

[0529] Starting from the first DFT stereo frame 602, certain TD stereo related data structures, e.g. the TD stereo data structure (as used in the TD stereo encoder 400) and the data structure of the core encoder 407 related to the secondary channel SCh, are no longer needed and thus, released, i.e. deallocated, by the stereo mode switch controller (not shown).

[0530] In the DFT stereo frame 602 following the TD stereo frame 601, the stereo mode switch controller (not shown) continues the core encoding operation 361 in the core encoder 311 of the DFT stereo encoder 300 with the memory of the primary PCh channel core encoder 406 (e.g. synthesis memory, pre-emphasis memory, past signals and parameters, etc.) used in the preceding TD stereo frame 601, while controlling the time instance difference between TD and DFT stereo mode to ensure continuity of several core encoder buffers, e.g. pre-emphasis input signal buffer, HB input buffer, etc., which are later used for the low-band encoder, FD-BWE high-band encoder, respectively. Further information on the core encoding operation 361, the memory of the PCh channel core encoder 406, the pre-emphasis input signal buffer, the HB input buffer, etc. can be found for example in reference [1].

[0531] 1.4 Switching from DFT stereo mode to TD stereo mode in the IVAS stereo encoding device 200

[0532] Switching from DFT stereo mode to TD stereo mode is more complex than switching from TD stereo mode to DFT stereo mode due to the more complex structure of the TD stereo encoder 400. In response to the stereo mode selection, the stereo mode switch controller (not shown) performs the following operations performed when switching from DFT stereo mode (DFT stereo encoder 300) to TD stereo mode (TD stereo encoder 400).

[0533] Figure 7a-1 and Figure 7a-2 are flowcharts illustrating the processing operations in the IVAS stereo encoding device 200 and method 250 when switching from DFT stereo mode to TD stereo mode. In particular, Figure 7a-1 and Figure 7a-2 illustrate the two frames of the stereo input signal at different processing operations with related time instances, i.e. DFT stereo frame 701 followed by TD stereo frame 702, when switching from DFT stereo mode to TD stereo mode.

[0534] Figure 7a-1Example A) refers to the update of the FIR resampling filter memory used in the main channel PCh of the TD stereo codec mode (as employed in FIR resampling from the input stereo signal sampling rate to the 12.8 kHz sampling rate and the internal core encoder sampling rate). The stereo mode switching controller (not shown) performs this update in each DFT stereo frame using the undermixed middle channel m, corresponding to a 2 × 0.9375 ms long segment 703 (see 704) before the last 7.5 ms long segment in DFT stereo frame 701, thereby ensuring the continuity of the FIR resampling memory of the main channel PCh.

[0535] Due to the DFT stereo coding method 350's side channel s ( Figure 3 The sub-channel SCh FIR resampling filter memory is unavailable, although it is used at, for example, a 12.8 kHz sampling rate, the input stereo signal sampling rate, and the internal sampling rate. Therefore, the stereo mode switching controller (not shown) fills the sub-channel SCh FIR resampling filter memory differently. In order to reconstruct the full length of the sub-mixed signal at the internal sampling rate of the core encoder 407, the 8.75 ms segment of the sub-mixed signal from the previous frame (see 705) is recalculated in the TD stereo frame 702. Therefore, the update of the sub-channel SCh FIR resampling filter memory corresponds to the 2 × 0.9375 ms segment 708 (see 705) of the sub-mixed middle channel m before the last 8.75 ms segment; this is done in the first TD stereo frame 702 after switching from the previous DFT stereo frame 701. The sub-channel SCh FIR resampling filter memory update is performed by Figure 7a-2 See Example C in the document. As can be seen, the stereo mode switching controller (not shown) recalculates the length of the downmix signal in the TD stereo frame (see 706), which is longer in the sub-channel SCh than the length of the downmix signal recalculated in the main channel PCh (see 707).

[0536] Figure 7a-2 Instance B) involves updating (recalculating) the primary PCh and secondary SCh channels in the first TD stereo frame 702 following the DFT stereo frame 701. The operation of Instance B) performed by the stereo mode switching controller (not shown) is as follows: Figure 7b This is shown in more detail below. As described above, Figure 7b This is a flowchart illustrating the processing operations when switching from DFT stereo mode to TD stereo mode.

[0537] refer to Figure 7b In operation 710, the stereo mode switching controller (not shown) recalculates the ICA memory, as used in ICA analysis and calculation (see...). Figure 2and the subsequent pre-processing and core encoder (see operations 453-454 and 456-459) of length 9.6875 ms (as discussed in sections 1.2.7-1.2.9 of the present disclosure) of the channels l and r corresponding to the previous DFT stereo frame 701.

[0538] Thus, in operations 712 and 713, the stereo mode switch controller (not shown) recomputes the primary PCh and secondary SCh channels of the DFT stereo frame 701 by downmixing the ICA processed channels l and r using the stereo mixing ratios of the DFT stereo frame 701.

[0539] For the secondary channel SCh, the length of the past segment recomputed by the stereo mode switch controller (not shown) in operation 712 (see 714) is 9.6875 ms, although the length of the segment recomputed is only 7.5 ms (see 715) when there is no stereo codec mode switch. For the primary channel PCh (see operation 713), the length of the segment recomputed by the stereo mode switch controller (not shown) using the TD stereo mixing ratios of the past frame 701 is always 7.5 ms (see 715). This ensures continuity of the primary PCh and secondary SCh channels.

[0540] When switching from the mid channel m of the DFT stereo frame 701 to the primary channel PCh of the TD stereo frame 702, a continuous downmix signal is employed. To this end, the stereo mode switch controller (not shown) cross-fades (717) the 7.5 ms long segment (see 715) of the DFT mid channel m and the recomputed primary channel PCh (713) of the DFT stereo frame 701 in order to smoothly transition and equalize the different downmix signal energies between the DFT stereo mode and the TD stereo mode. The reconstruction of the secondary channel SCh in operation 712 uses the mixing ratios of the frame 701, while no further smoothing is applied due to the unavailability of the secondary channel SCh from the DFT stereo frame 701.

[0541] Then, core encoding in the first TD stereo frame 702 after the DFT stereo frame 701 continues with resampling the downmix signals using FIR filters, pre-emphasis of the signals, computation of the HB signals, etc. Further information on these operations can be found in reference [1].

[0542] With respect to the pre-emphasis filter implemented as a first order high-pass filter for emphasizing higher frequencies of the input signal (see reference [1], clause 5.1.4), the stereo mode switch controller (not shown) stores two values of the pre-emphasis filter memory in each DFT stereo frame. These stored values correspond to different re-computation length based time instances of the DFT and TD stereo modes. This mechanism ensures an optimal re-computation of the pre-emphasized signal with minimum signal length in the channel m and the primary channel PCh, respectively. For the secondary channel SCh of the TD stereo mode, the pre-emphasis filter memory is set to zero before the first TD stereo frame is processed.

[0543] Starting from the first TD stereo frame 702 after the DFT stereo frame 701, certain DFT stereo related data structures (e.g., the DFT stereo data structures mentioned above) are not needed and thus released / deallocated by the stereo mode switch controller (not shown). On the other hand, the core encoding (operation 457) for the secondary channel SCh allocates and initializes a second instance of the core encoder data structures. Most of the secondary channel SCh core encoder data structures are reset, although some of them are estimated for a smoother switching transition. For example, the previous excitation buffer (adaptive codebook of ACELP core), the previous LSF parameters and LSP parameters (see reference [1]) of the secondary channel SCh are filled from the corresponding parts in the primary channel PCh. The reset or estimation of the previous buffers of the secondary channel SCh can be a source of a significant amount of artifacts. While many of these artifacts are significantly suppressed in the decoder’s smoothing-based process, some of them can still be a source of subjective artifacts.

[0544] 1.5 Switching from TD stereo mode to MDCT stereo mode in the IVAS stereo encoding device 200

[0545] Switching from TD stereo mode to MDCT stereo mode is relatively simple, since both stereo modes process two input channels and employ two core encoder instances. The main obstacle is to maintain the correct phase of the input left and right channels.

[0546] In order to maintain the correct phase of the input left and right channels of the stereo sound signal, the stereo mode switch controller (not shown) changes the TD stereo downmix. In the last TD stereo frame before the first MDCT stereo frame, the TD stereo mixing ratio is set to β = 1.0 and an inverse downmix of the left and right channels of the stereo sound signal is achieved using, for example, the following formula for the TD stereo downmix:

[0547]

[0548]

[0549] where PCh(i) is the TD primary channel, SCh(i) is the TD secondary channel, l(i) is the left channel, r(i) is the right channel, β is the TD stereo mix ratio, and i is the discrete time index.

[0550] In turn, this means that the TD stereo primary channel PCh(i) is identical to the MDCT stereo past left channel lpast(i), and the TD stereo secondary channel SCh(i) is identical to the MDCT stereo past right channel rpast(i), where i is the discrete time index. For completeness, note that the stereo mode switch controller (not shown) can use, in the last TD stereo frame, the following formula, using the default TD stereo downmix:

[0551]

[0552]

[0553] Next, in the usual (no stereo mode switch) MDCT stereo processing, the pre- processing (pre-processor 503 and 504 and pre-processing operation 553 and 554) does not recompute the lookahead of the left l and right r channels of the stereo sound signal except for its last 0.9375 ms long segment. In practice, however, the 7.5 + 0.9375 ms long lookahead is affected by the recomputation at the internal sampling rate (in this non-limiting illustrative implementation 12.8 kHz). Therefore, no special processing is needed to maintain continuity of the input signal at the input sampling rate.

[0554] Then, in the usual (no stereo mode switch) MDCT stereo processing, the further pre-processing (further pre-processor 505 and 507 and further processing operation 555 and 557) does not recompute the lookahead of the left l and right r channels of the stereo sound signal except for its last 0.9375 ms long segment. In contrast to the pre-processing, in the further pre-processing the input signal (left l and right r channels of the stereo sound signal) at the internal sampling rate (in this non-limiting illustrative implementation 12.8 kHz) is recomputed for a length of only 0.9375 ms.

[0555] In other words:

[0556] The MDCT stereo encoder 500 comprises (a) a pre-preprocessor 503 and 504 which, in the second MDCT stereo mode, recomputes a lookahead of a first duration of the left I and right r channels of the stereo sound signal at the internal sampling rate, and (b) a further pre-processor which, in the second MDCT stereo mode, recomputes a last segment of a given duration of the lookahead of the left I and right r channels of the stereo sound signal at the internal sampling rate, wherein the first and second durations are different.

[0557] The MDCT stereo encoding operation 550 comprises, in the second MDCT stereo mode: (a) recomputing a lookahead of a first duration of the left I and right r channels of the stereo sound signal at the internal sampling rate, and (b) recomputing a last segment of a given duration of the lookahead of the left I and right r channels of the stereo sound signal at the internal sampling rate, wherein the first and second durations are different.

[0558] 1.6 Switching from MDCT stereo mode to TD stereo mode in the IVAS stereo encoding device 200

[0559] Similar to the switching from TD stereo mode to MDCT stereo mode, in this scenario, always two input channels are available and always two core encoder instances are employed. The main obstacle is again to maintain the correct phase of the input left and right channels. Therefore, in the first TD stereo frame after the last MDCT stereo frame, the stereo mode switch controller (not shown) sets the TD stereo mix ratio to β = 1.0 and changes the TD stereo downmix by using the phase-inverted mix scheme similar to described in section 1.5.

[0560] Another specific operation regarding the switching from MDCT stereo mode to TD stereo mode is that the stereo mode switch controller (not shown) appropriately reconstructs past segments of the input channels of the stereo sound signal at the internal sampling rate in the first TD frame. Therefore, a part of the lookahead corresponding to 8.75 - 7.5 = 1.25 ms is reconstructed (resampled and pre-emphasized) in the first TD stereo frame.

[0561] 1.7 Switching from DFT stereo mode to MDCT stereo mode in the IVAS stereo encoding device 200

[0562] In this scenario, a mechanism similar to the switching from DFT stereo mode to TD stereo mode as described above is used, wherein the primary PCh and secondary SCh channels of the TD stereo mode are replaced by the left I and right r channels of the MDCT stereo mode.

[0563] 1.8 Switching from MDCT stereo mode to DFT stereo mode in the IVAS stereo encoding device 200

[0564] In this scenario, a mechanism similar to the one described above for switching from TD stereo mode to DFT stereo mode is used, wherein the primary PCh and secondary SCh channels of the TD stereo mode are replaced by the left l and right r channels of the MDCT stereo mode.

[0565] 2. Switching between stereo modes in the IVAS stereo decoding device 800 and method 850

[0566] Figure 8 is a high-level block diagram showing both the IVAS stereo decoding device 800 and the corresponding decoding method 850, wherein the IVAS stereo decoding device 800 comprises a DFT stereo decoder 801 and the corresponding DFT stereo decoding method 851, a TD stereo decoder 802 and the corresponding TD stereo decoding method 852, and an MDCT stereo decoder 803 and the corresponding MDCT stereo decoding method 853. For simplicity, only the DFT, TD, and MDCT stereo modes are shown and described; however, using and implementing other types of stereo modes is also within the scope of the present disclosure.

[0567] The IVAS stereo decoding device 800 and the corresponding decoding method 850 receive the bitstream 830 transmitted from the IVAS stereo encoding device 200. In general, the IVAS stereo decoding device 800 and the corresponding decoding method 850 decode successive frames of the coded stereo signal from the bitstream 830, e.g. 20 ms long frames in the case of the EVS codec, perform the upmixing of the decoded frames, and finally produce a stereo output signal comprising the channels L and R.

[0568] 2.1 Differences between different stereo decoders and decoding methods

[0569] The core decoding performed at the internal sampling rate is basically the same, independent of the actual stereo mode; however, for DFT stereo frames, the core decoding is performed once (intermediate channel m), for TD stereo frames (primary PCh and secondary SCh channels) or for MDCT stereo frames (left l and right r channels) twice. One problem is that the memory for the secondary channel SCh of the TD stereo frame is kept (updated) when switching from a DFT stereo frame to a TD stereo frame, respectively, and the memory for the r channel of the MDCT stereo frame is kept (updated) when switching from a DFT stereo frame to a MDCT stereo frame.

[0570] Furthermore, the further decoding operations after the core decoding strongly depend on the actual stereo mode, which thus complicates the switching between stereo modes. The most fundamental differences are as follows:

[0571] DFT stereo decoder 801 and decoding method 851 :

[0572] - Resampling of the decoded core synthesis from the internal sampling rate to the output stereo signal sampling rate is done in the DFT domain with a DFT analysis and synthesis overlap window length of 3.125 ms.

[0573] - Low-band (LB) bass post-filtering (in ACELP frames) adaptation is done in the DFT domain.

[0574] - Core switching (ACELP core <-> TCX / HQ core) is done in the DFT domain with a delay of 3.125 ms.

[0575] - Synchronization between LB synthesis and HB synthesis (in ACELP frames) does not require additional delay.

[0576] - Stereo upmixing is done in the DFT domain with a delay of 3.125 ms.

[0577] - Time synchronization matching the overall decoder delay (which is 3.25 ms) is applied as a length of 0.125 ms.

[0578] TD stereo decoder 802 and decoding method 852 : (Further information on the TD stereo decoder can be found in e.g. reference [4])

[0579] - Resampling of the decoded core synthesis from the internal sampling rate to the output stereo signal sampling rate is done with a CLDFB filter with a delay of 1.25 ms.

[0580] - LB bass post-filtering (in ACELP frames) adaptation is done in the CLDFB domain.

[0581] - Core switching (ACELP core <-> TCX / HQ core) is done in the time domain with a delay of 1.25 ms.

[0582] - Synchronization between LB synthesis and HB synthesis (in ACELP frames) introduces an additional delay.

[0583] - Stereo upmixing is done in the TD domain with zero delay.

[0584] - Time synchronization matching the overall decoder delay is applied as a length of 2.0 ms.

[0585] MDCT stereo decoder 803 and decoding method 853 :

[0586] - Since only a TCX-based core decoder is employed, only a delay adjustment of 1.25 ms is used to synchronize the core synthesis signals between different cores.

[0587] - LB low-band post-filtering is skipped (in ACELP frames).

[0588] - Core-switch (ACELP core <-> TCX / HQ core) is done in time-domain only in the first MDCT stereo frame after a TD or DFT stereo frame, with a delay of 1.25 ms.

[0589] - Synchronization of LB synthesis and HB synthesis is not relevant.

[0590] - Stereo up-mixing is skipped.

[0591] - Time synchronization matching the overall decoder delay is applied as a length of 2.0 ms.

[0592] In the procedures described herein below for switching between DFT and TD stereo modes, the different operations during decoding are carefully considered, mainly the DFT "vs" TD domain processing, and the different delay schemes between DFT stereo mode and TD stereo mode.

[0593] 2.2 Processing and decoding method 850 in IVAS stereo decoding device 800

[0594] The following table III lists the processing operations in IVAS stereo decoding device 800 for each frame in order according to the current DFT, TD or MDCT stereo mode (see also Figure 8 ).

[0595] Table III - Processing steps in IVAS stereo decoding device 800

[0596]

[0597] The IVAS stereo decoding method 850 includes operations (not shown) to control the switching between DFT, TD and MDCT stereo modes. To perform the switching control operations, the IVAS stereo decoding device 800 includes a controller (not shown) to switch between DFT, TD and MDCT stereo modes. The switching between DFT, TD and MDCT stereo modes in the IVAS stereo decoding device 800 and decoding method 850 involves using the stereo mode switching controller (not shown) to maintain continuity of several decoder signals and memories 1) to 6) below, to enable adequate processing of these signals and use of these memories in the IVAS stereo decoding device 800 and method 850:

[0598] 1) Memories of down-mix signals and core post-filters at the internal sampling rate, for core decoding;

[0599] - DFT stereo decoder 801 : mid channel m;

[0600] - TD stereo decoder 802 : primary channel PCh and secondary channel SCh;

[0601] - MDCT stereo decoder 803 : left channel l and right channel r (non-downmix).

[0602] 2) TCX-LTP (Transform Coded Excitation - Long Term Prediction) post filter memory. The TCX-LTP post filter is used to interpolate between past synthesized samples using a polyphase FIR interpolation filter (see reference [1], clause 6.9.2);

[0603] 3) DFT OLA analysis memory used in the OLA part of the window in the previous and current frame, at the internal sampling rate and the output stereo signal sampling rate, before the DFT operation 854;

[0604] 4) DFT OLA synthesis memory used in the OLA part of the window in the previous and current frame, at the output stereo signal sampling rate, after the IDFT operations 855 and 856;

[0605] 5) output stereo signals, including channels l and r; and

[0606] 6) HB signal memory used in BWE and IC-BWE (see reference [1], clause 6.1.5), channels l and r.

[0607] While maintaining the continuity of one channel (the mid channel m in DFT stereo mode, the primary channel PCh in TD stereo mode or the l channel in MDCT stereo mode, respectively) in item 1) above is relatively simple, it is challenging for the secondary channel SCh in item 1) above and for the signals / memory in items 2) - 6) due to several aspects, e.g. complete loss of past signals and memories for the secondary channel SCh, different downmixing, different default delays between DFT stereo mode and TD stereo mode, etc. In addition, the shorter decoder delay (3.25 ms) compared to the encoder delay (8.75 ms) further complicates the decoding process.

[0608] 2.2.1 Reading stereo mode and audio bandwidth information

[0609] The IVAS stereo decoding method 850 starts with reading (not shown) the stereo mode and audio bandwidth information from the transmitted bitstream 830. Based on the currently read stereo mode, the relevant decoding operations are performed for each specific stereo mode (see Table III), while maintaining the memories and buffers for the other stereo modes.

[0610] 2.2.2 Memory allocation

[0611] Similar to the IVAS stereo encoding device 200, in a memory allocation operation (not shown), a stereo mode switching controller (not shown) dynamically allocates / deallocates data structures (static memory) according to the current stereo mode. The stereo mode switching controller (not shown) keeps the codec’s static memory footprint as low as possible by only keeping those parts of the static memory that are used in the current frame. Refer to Table II for an overview of the data structures allocated in a specific stereo mode.

[0612] Furthermore, the stereo mode switching controller (not shown) reads the LRTD stereo submode flag to distinguish between normal TD stereo mode and LRTD stereo mode. Based on the submode flag, the stereo mode switching controller (not shown) allocates / deallocates the relevant data structures within the TD stereo mode as shown in Table II.

[0613] 2.2.3 Stereo mode switch update

[0614] Similar to the IVAS stereo encoding device 200, the stereo mode switching controller (not shown) handles the memory in case of a switch from one of the DFT, TD, and MDCT stereo modes to another stereo mode. This keeps the updated long-term parameters and updates or resets the past buffer memory.

[0615] Upon reception of the first DFT stereo frame after a TD stereo frame or an MDCT stereo frame, the stereo mode switching controller (not shown) performs the operation of resetting the DFT stereo data structures (already defined with respect to the DFT stereo encoder 300). Upon reception of the first TD stereo frame after a DFT or MDCT stereo frame, the stereo mode switching controller performs the operation of resetting the TD stereo data structures (already described with respect to the TD stereo decoder 400). Finally, upon reception of the first MDCT stereo frame after a DFT or TD stereo frame, the stereo mode switching controller (not shown) performs the operation of resetting the MDCT stereo data structures. Again, when switching from one of the DFT and TD stereo modes to another stereo mode, the stereo mode switching controller (not shown) performs the operation of transferring some stereo-related parameters between the data structures as described with respect to the IVAS stereo encoding device 200 (see section 1.2.4 above).

[0616] The update / resetting of the secondary channels SCh related to the core decoding is described in section 2.4.

[0617] Further information on the operation of the stereo decoder configurations in Table III, core decoder configurations, TD stereo decoder configurations, core decoding, core switching in the DFT domain, core switching in the TD domain can be found in, for example, references [1] and [2].

[0618] 2.2.4 Update of DFT stereo mode overlap memory

[0619] The stereo mode switch controller (not shown) maintains or updates the DFT OLA memory in each TD or MDCT stereo frame (see Table III “Update of DFT stereo mode overlap memory”, “Update of MDCT stereo TCX overlap buffer”, and “Reset / update of DFT stereo overlap memory”). In this way, the updated DFT OLA memory is available for the next DFT stereo frame. The actual maintenance / update mechanism and related memory buffers are described later in Section 2.3 of this disclosure. An example implementation of the DFT stereo OLA memory update performed in a TD or MDCT stereo frame in C source code is given below.

[0620] if ( st[n]->element_mode!= IVAS_CPE_DFT )

[0621] {

[0622] ivas_post_proc(... );

[0623] / * Update OLA buffer - switch to DFT stereo needed * /

[0624] stereo_td2dft_update( hCPE, n, output[n], synth[n], hb_synth[n],output_frame );

[0625] / * Update ovl buffer for possible switch from TD stereo SCh ACELP frame to MDCT stereo TCX frame * /

[0626] if ( st[n]->element_mode == IVAS_CPE_TD && n == 1 && st[n]->hTcxDec == NULL )

[0627] {

[0628] mvr2r( output[n] + st[n]->L_frame / 2, hCPE->hStereoTD->TCX_old_syn_Overl, st[n]->L_frame / 2 );

[0629] }

[0630] }

[0631] void stereo_td2dft_update(

[0632] CPE_DEC_HANDLE hCPE, / * i / o: CPE decoder structure * /

[0633] const int16_t n, / * i : number of audio channels * /

[0634] float output[], / * i / o: synthesize @internal Fs * /

[0635] float synth[], / * i / o: synthesize @output Fs * /

[0636] float hb_synth[], / * i / o: hb synthesis * /

[0637] const int16_t output_frame / * i : frame length * / )

[0639] {

[0640] int16_t ovl, ovl_TCX, dft32ms_ovl, hq_delay_comp;

[0641] Decoder_State **st;

[0642] / * Initialization * /

[0643] st = hCPE->hCoreCoder;

[0644] ovl = NS2SA( st[n]->L_frame * 50, STEREO_DFT32MS_OVL_NS );

[0645] dft32ms_ovl = (STEREO_DFT32MS_OVL_MAX * st[0]->output_Fs) / 48000;

[0646] hq_delay_comp = NS2SA( st[0]->output_Fs, DELAY_CLDFB_NS );

[0647] if ( hCPE->element_mode >= IVAS_CPE_DFT && hCPE->element_mode !=IVAS_CPE_MDCT )

[0648] {

[0649] if ( st[n]->core == ACELP_CORE )

[0650] {

[0651] if ( n == 0 )

[0652] {

[0653] / * Update DFT analysis overlap memory @internal_fs: core synthesis * /

[0654] mvr2r( output + st[n]->L_frame - ovl, hCPE->input_mem_LB[n], ovl );

[0655] / * Update DFT analysis overlap memory @internal_fs: BPF * /

[0656] if ( st[n]->p_bpf_noise_buf )

[0657] {

[0658] mvr2r( st[n]->p_bpf_noise_buf + st[n]->L_frame -ovl, hCPE->input_mem_BPF[n], ovl );

[0659] }

[0660] / * Update DFT analysis overlap memory @output_fs: BWE * /

[0661] if ( st[n]->extl != -1 || ( st[n]->bws_cnt > 0 && st[n]->core == ACELP_CORE ) )

[0662] {

[0663] mvr2r( hb_synth + output_frame - dft32ms_ovl,hCPE->input_mem[n], dft32ms_ovl );

[0664] }

[0665] }

[0666] else

[0667] {

[0668] / * Update DFT analysis overlay memory @internal_fs: core synthesis, sub-channel * /

[0669] mvr2r( output + st[n]->L_frame - ovl, hCPE->input_mem_LB[n], ovl );

[0670] }

[0671] }

[0672] else / * TCX core * /

[0673] {

[0674] / * LB-TCX Synthesis * /

[0675] mvr2r( output + st[n]->L_frame - ovl, hCPE->input_mem_LB[n], ovl );

[0676] / * BPF * /

[0677] if ( n == 0 && st[n]->p_bpf_noise_buf )

[0678] {

[0679] mvr2r( st[n]->p_bpf_noise_buf + st[n]->L_frame - ovl,hCPE->input_mem_BPF[n], ovl );

[0680] }

[0681] / * TCX synthesis (it has been delayed in TD stereo in core_switching_post_dec() * /

[0682] if ( st[n]->hTcxDec!= NULL )

[0683] {

[0684] ovl_TCX = NS2SA( st[n]->hTcxDec->L_frameTCX * 50,STEREO_DFT32MS_OVL_NS );

[0685] mvr2r( synth + st[n]->hTcxDec->L_frameTCX + hq_delay_comp - ovl_TCX, hCPE->input_mem[n], ovl_TCX - hq_delay_comp );

[0686] mvr2r( st[n]->delay_buf_out, hCPE->input_mem[n] +ovl_TCX - hq_delay_comp, hq_delay_comp );

[0687] }

[0688] }

[0689] }

[0690] else if ( hCPE->element_mode == IVAS_CPE_MDCT && hCPE->input_mem[0]!= NULL )

[0691] {

[0692] / * Reset DFT stereo OLA memory * /

[0693] set_zero( hCPE->input_mem[n], NS2SA( st[0]->output_Fs,STEREO_DFT32MS_OVL_NS ) );

[0694] set_zero( hCPE->input_mem_LB[n], STEREO_DFT32MS_OVL_16k );

[0695] if ( n == 0 )

[0696] {

[0697] set_zero( hCPE->input_mem_BPF[n], STEREO_DFT32MS_OVL_16k);

[0698] }

[0699] }

[0700] return;

[0701] }

[0702] 2.2.5 DFT stereo decoder 801 and decoding method 851

[0703] The DFT decoding method 851 includes an operation 857 of core decoding the mid channel m. To perform operation 857, the core decoder 807 decodes the mid channel m in the time domain in response to the received bitstream 830. The core decoder 807 in the DFT stereo decoder 801 (performing core decoding operation 857) can be any variable bit rate mono codec. In the illustrative implementation of the present disclosure, the EVS codec (see reference [1]) with fluctuating bit rate capability (see reference [5]) is used. Of course, other suitable codecs can also be considered and implemented.

[0704] In the DFT computation operation 854 (DFT analysis of Table III) of the DFT decoding method 851, the calculator 804 computes the DFT of the mid channel m to recover the mid channel M in the DFT domain.

[0705] The DFT decoding method 851 also includes an operation 858 (residual decoding of Table III) of decoding the stereo side information and the residual signal S. To perform operation 858, the decoder 808 recovers the stereo side information and the residual signal S in response to the bitstream 830.

[0706] In the DFT stereo decoding (DFT stereo decoding of Table III) and upmixing (upmixing in the DFT domain of Table III) operation 859, the DFT stereo decoder and upmixer 809 produces the channels L and R in the DFT domain in response to the mid channel M and the side information and the residual signal S. Generally, the DFT stereo decoding and upmixing operation 859 is the inverse of the DFT stereo processing and downmixing operation 353 of Figure 3

[0707] ​In IDFT computation operation 855 (DFT synthesis of Table III), calculator 805 computes the IDFT of channel L to recover channel / in time domain. Likewise, in IDFT computation operation 856 (DFT synthesis of Table III), calculator 806 computes the IDFT of channel R to recover channel r in time domain.

[0708] 2.2.6 TD stereo decoder 802 and decoding method 852

[0709] TD decoding method 852 includes operation 860 of core decoding the primary channel PCh. To perform operation 860, core decoder 810 decodes the primary channel PCh in response to the received bitstream 830.

[0710] TD decoding method 852 also includes operation 861 of core decoding the secondary channel SCh. To perform operation 861, core decoder 811 decodes the secondary channel SCh in response to the received bitstream 830.

[0711] Likewise, core decoder 810 (performing core decoding operation 860 in TD stereo decoder 802) and core decoder 811 (performing core decoding operation 861 in TD stereo decoder 802) can be any variable bit rate single mode codec. In the illustrative implementation of the present disclosure, the EVS codec (see reference [1]) with fluctuating bit rate capability (see reference [5]) is used. Of course, other suitable codecs can also be considered and implemented.

[0712] In time domain (TD) mixing operation 862 (upmix in TD domain of Table III), upmixer 812 receives the primary PCh and secondary SCh channels and performs upmixing to recover the time domain channels / and r of the stereo signal based on the TD stereo mixing factors.

[0713] 2.2.7 MDCT stereo decoder 803 and decoding method 853

[0714] MDCT decoding method 853 includes operation 863 of joint core decoding the left channel / and the right channel r (joint stereo decoding of Table III). To perform operation 863, joint core decoder 813 decodes the left channel / and the right channel r in response to the received bitstream 830. Note that no upmix operation is performed and no upmixer is employed in the MDCT stereo mode.

[0715] 2.2.8 Synthesis synchronization

[0716] To perform the stereo synthesis time synchronization (synthesis synchronization of Table III) and stereo switching operation 864, the stereo mode switching controller (not shown) includes a time synchronizer and stereo switch 814 to receive the channels l and r from the DFT stereo decoder 801, TD stereo decoder 802, or MDCT stereo decoder 803 and synchronize the upmixed output stereo channels l and r. The time synchronizer and stereo switch 814 delays the upmixed output stereo channels l and r to match the codec overall delay value and handles the transition between DFT stereo output channels, TD stereo output channels, and MDCT stereo output channels.

[0717] By default, in the DFT stereo mode, the time synchronizer and stereo switch 814 introduces a delay of 3.125 ms at the DFT stereo decoder 801. To match the codec overall delay of 32 ms (frame length 20 ms, encoder delay 8.75 ms, decoder delay 3.25 ms), the time synchronizer and stereo switch 814 applies a delay synchronization of 0.125 ms. In the case of TD or MDCT stereo mode, the time synchronizer and stereo switch 814 applies a delay consisting of a 1.25 ms resampling delay and a 2 ms delay for the synchronization between LB and HB synthesis to match the overall codec delay of 32 ms.

[0718] After performing the time synchronization and stereo switching (see Figure 8 synthesis time synchronization and stereo switching operation 864 and time synchronizer and stereo switch 814) of Table III), the HB synthesis (from BWE or IC-BWE) is added to the core synthesis (IC-BWE, addition of HB synthesis of Table III; see also Figure 8 BWE or IC-BWE computation operation 865 and BWE or IC-BWE calculator 815) of Table III) and ICA decoding is performed (ICA decoder - time adjustment of Table III that de-synchronizes the two output channels l and r) (see time ICA operation 866 and corresponding ICA decoder 816) before outputting the final stereo synthesis of channels l and r from the IVAS stereo decoding device 800. Operations 865 and 866 are skipped in the MDCT stereo mode.

[0719] Finally, the general stereo update is performed as shown in Table III.

[0720] 2.3 Switching from TD stereo mode to DFT stereo mode at the IVAS stereo decoding device

[0721] Further information on the elements, operations, and signals mentioned in sections 2.3 and 2.4 can be found in references [1] and [2].

[0722] The mechanism of switching from TD stereo mode to DFT stereo mode at the IVAS stereo decoding device 800 is complicated by the fact that the decoding steps between these two stereo modes differ radically (see section 2.1 above for details), including the transition from two core decoders 810 and 811 in the last TD stereo frame to one core decoder 807 in the first DFT stereo frame.

[0723] Figure 9 is a flowchart illustrating the processing operations in the IVAS stereo decoding device 800 and method 850 when switching from TD stereo mode to DFT stereo mode. Specifically, Figure 9 illustrates the two frames of decoded stereo signals at different processing operations with the relevant time instance when switching from a TD stereo frame 901 to a DFT stereo frame 902.

[0724] First, the core decoders 810 and 811 of the TD stereo decoder 802 are used for the primary PCh and secondary SCh channels, and each outputs a corresponding decoded core synthesis at the internal sampling rate. In the TD stereo frame 901, the decoded core syntheses from the two core decoders 810 and 811 are used to update the DFT stereo OLA memory buffers (one memory buffer per channel, i.e. two OLA memory buffers in total; see the DFT OLA analysis and synthesis memories described above). These OLA memory buffers are updated in each TD stereo frame to remain up-to-date in case the next frame is a DFT stereo frame.

[0725] Figure 9 Instance A) refers to the operation (not shown) of updating the DFT stereo analysis memories (these memories are used in the windowed OLA sections in the previous and current frames prior to the DFT computation operation 854) using the stereo mode switch controller (not shown) at the internal sampling rate input mem LB [] upon receiving the first DFT stereo frame 902 after the TD stereo frame 901. To this end, the stereo mode switch controller (not shown) uses the number L ovl of the last samples 903 of the TD stereo synthesis of the primary channel PCh and secondary channel SCh in the TD stereo frame 901 to update the DFT stereo analysis memories for the DFT stereo mid channel m and side channel s, respectively. The length L ovl of the overlap segment 903 corresponds to the 3.125 ms long overlap section of the DFT analysis window 905, e.g. L ovl = 40 samples at the 12.8 kHz internal sampling rate.

[0726] Similarly, the stereo mode switch controller (not shown) updates the DFT stereo bass post filter (BPF) analysis memory of the mid channel m (which is used in the windowed OLA portion in the previous and current frame before the DFT computation operation 854) with the L ovl Additionally, the DFT stereo full band (FB) analysis memory of the mid channel m under the output stereo signal sampling rate input_mem[] is updated with the 3.125 ms last samples of the TD stereo PCh HB synthesis (ACELP core), PCh TCX synthesis, respectively (this memory is used in the windowed OLA portion in the previous and current frame before the DFT computation operation 854). The DFT stereo BPF and FB analysis memories are not used for the side information channel s, so these memories are not updated with the secondary channel SCh core synthesis.

[0727] Next, in the TD stereo frame 901, the decoded ACELP core synthesis (both primary PCh and secondary SCh channels) is resampled at the internal sampling rate using the CLD FB domain filtering with the introduced 1.25 ms delay. In the case of TCX / HQ core frames, a compensating delay of 1.25 ms is used to synchronize the core synthesis between different cores. The TCX-LTP post filter is then applied to both core channels PCh and SCH.

[0728] In the next operation, the primary PCh and secondary SCh channels of the TD stereo synthesis from the TD stereo frame 901 under the output stereo signal sampling rate are subject to TD stereo upmixing (using the TD stereo mixing ratios in the TD upmixer 812 to combine the primary PCh and secondary SCh channels (see reference [4]), resulting in the upmixed stereo channels l and r in the time domain. Since the upmixing operation 862 is performed in the time domain, no upmixing delay is introduced.

[0729] Then, the left l and right r upmixed channels of the TD stereo frame 901 from the upmixer 812 of the TD stereo decoder 802 are used in an operation (not shown) to update the DFT stereo synthesis memories (these memories are used in the windowed OLA portion in the previous and current frame after the IDFT computation operation 855). Again, in the case of the next frame being a DFT stereo frame, the stereo mode switch controller (not shown) completes this update in every TD stereo frame. Figure 9Example B) depicts that the number of available last samples of the TD stereo left l and right r channel synthesis is insufficient for a direct update of the DFT stereo synthesis memory. The 3.125 ms long DFT stereo synthesis memory is thus used to reconstruct approximately two segments. The first segment corresponds to the available (3.125 - 1.25) ms long signal (i.e. the upmix synthesis at the output stereo signal sampling rate), while the second segment corresponds to the remaining 1.25 ms long signal which is not available due to the core decoder resampling delay.

[0730] In particular, the DFT stereo synthesis memory is updated by the stereo mode switch controller (not shown) using the following sub-operations, as shown in Figure 10 Figure 10 is a flowchart showing Figure 9 Example B) including the update of the DFT stereo synthesis memory in the TD stereo frame at the decoder side:

[0731] (a) The two channels l and r of the DFT stereo analysis memory at the internal sampling rate input_mem_LB[] which were reconstructed earlier during the decoding method 850 (they are identical to the core synthesis at the internal sampling rate) are subject to further processing depending on the actual decoding core:

[0732] - ACELP core : The last L ovl samples 1001 of the LB core synthesis of the primary PCh and secondary SCh channels at the internal sampling rate are resampled to the output stereo signal sampling rate (see 1003) using a simple linear interpolation with zero delay.

[0733] - TCX / HQ core : The last L ovl samples 1001 of the LB core synthesis of the primary PCh and secondary SCh channels at the internal sampling rate are similarly resampled to the output stereo signal sampling rate (see 1003) using a simple linear interpolation with zero delay. However, then, the TCX sync memory (the last 1.25 ms segment of the TCX synthesis from the previous frame) is used to update the last 1.25 ms of the resampled core synthesis.

[0734] (b) The linearly resampled LB signal corresponding to the 3.125 ms long portion of the primary PCh and secondary SCh channels of the TD stereo frame 901 is upmixed (see 1003) to form left l and right r channels using the general TD stereo upmix routine while using the TD stereo mixing ratios from the current frame (see TD upmix operation 862). The resulting signal is further referred to as "reconstructed synthesis" 1002.

[0735] ​​​(c) The reconstruction of the first (3.125 - 1.25 ms) long part of the DFT stereo synthesis memory depends on the actual decoding core:

[0736] - ACELP core : During the first (3.125 - 1.25) ms long part of the channels of the TD stereo frame 901, both CLD FB based resampling and TD upmix synthesis 1005 at the output stereo signal sampling rate and crossfading 1004 between the reconstructed synthesis 1002 (from the previous sub-operation (b)) and the upmix synthesis 1005 are performed for both channels I and r.

[0737] - TCX / HQ core : The first (3.125 - 1.25) ms long part of the DFT stereo synthesis memory is updated using the upmix synthesis 1005.

[0738] (d) The last 1.25 ms long part of the DFT stereo synthesis memory is filled with the last part of the reconstructed synthesis 1002.

[0739] (e) Only in the first DFT stereo frame 902 (if a switch from TD to DFT stereo mode occurs), the DFT synthesis window (904) in Figure 9 is applied to the DFT OLA synthesis memory (defined above). Note that due to the DFT synthesis window shape 904 converging to zero, the importance of the last 1.25 ms part of the DFT OLA synthesis memory is limited and it thus masks the approximated samples of the reconstructed synthesis 1002 resulting from a simple linear interpolation based resampling.

[0740] Finally, the upmixed reconstructed synthesis 1002 of the TD stereo frame 901 is aligned, i.e. delayed by 2 ms in the time synchronizer and stereo switch 814, in order to match the codec overall delay. In particular:

[0741] - In case of a switch from a TD stereo frame to a DFT stereo frame, the other DFT stereo memories (except the overlap memory) are reset by the stereo mode switch controller (not shown), i.e. the past frame parameters and buffers of the DFT stereo decoder.

[0742] - Then, the DFT stereo decoding (see 859), upmixing (see 859) and DFT synthesis (see 855 and 856) are performed and the stereo output synthesis (channels I and r) is aligned, i.e. delayed by 0.125 ms in the time synchronizer and stereo switch 814, in order to match the codec overall delay.

[0743] Figure 11 is shown Figure 9The flowchart of Instance C) includes, on the decoder side, smoothing the output stereo synthesis in the first DFT stereo frame 902 after the stereo mode switch.

[0744] refer to Figure 11 Once the DFT stereo synthesis is aligned and synchronized with the overall delay of the codec in the first DFT stereo frame 902, the stereo mode switching controller (not shown) performs a crossfading operation 1151 between TD stereo alignment and synchronization synthesis 1101 (from operation 864) and DFT stereo alignment and synchronization synthesis 1102 (from operation 864) to smooth the switching transition. The crossfading is performed on a 1.875 ms long segment 1103, which begins 0.125 ms after a delay 1104 at the beginning of both output channels l and r (all signals are at the output stereo signal sampling rate). This example corresponds to... Figure 9 (Example C in the text).

[0745] Then, the IC-BWE calculator 815, ICA decoder 816, and universal stereo decoder are used to continue decoding, regardless of the current stereo mode.

[0746] 2.4 Switch from DFT stereo mode to TD stereo mode at the IVAS stereo decoding device.

[0747] The fundamentally different decoding operations between DFT stereo mode and TD stereo mode, as well as the presence of two core decoders 810 and 811 in the TD stereo decoder 802, make switching from DFT stereo mode to TD stereo mode in the IVAS stereo decoder device 800 challenging. Figure 12 This is a flowchart illustrating the processing operations in the IVAS stereo decoding device 800 and method 850 when switching from DFT stereo mode to TD stereo mode. Specifically, Figure 12 The diagram shows two frames of the decoded stereo signal at different processing operations with relevant time instances when switching from DFT stereo frame 1201 to TD stereo frame 1202.

[0748] The core decoder can use the same processing regardless of the actual stereo mode, with two exceptions.

[0749] First exception In a DFT stereo frame, resampling from the internal sampling rate to the output stereo signal sampling rate is performed in the DFT domain, but CLDFB resampling is run in parallel to maintain / update the CLDFB analysis and synthesis memory if the next frame is a TD stereo frame.

[0750] Second exception: Then, a BPF (a bass post-filter, see [1], clause 6.1.4.2) is applied in the DFT domain of the DFT stereo frame, while the BPF analysis and computation is done in the time domain on the error signal, regardless of the stereo mode.

[0751] Otherwise, when switching from the DFT mid channel m to the TD primary channel PCh, all internal states and memories of the core decoder are simply continuously and well maintained.

[0752] Then, in the DFT stereo frame 1201, the decoding continues with core decoding of the mid channel m (857), computation of the DFT transform of the mid channel m in the time domain (854) to obtain the mid channel m in the DFT domain, and stereo decoding and upmixing of the channels M and S (859) to the channels L and R in the DFT domain, including decoding of the residual signal (858). The DFT domain analysis and synthesis introduces an OLA latency of 3.125 ms. The synthesis conversion is then processed in the time synchronizer and stereo switch 814.

[0753] The fact that only one core decoder 807 is in the DFT stereo decoder 801 when switching from the DFT stereo frame 1201 to the TD stereo frame 1202 complicates the core decoding of the TD secondary channel SCh, because the internal states and memories of the second core decoder 811 of the TD stereo decoder 802 are not continuously maintained (in contrast, the internal states and memories of the core decoder 807 of the DFT stereo decoder 801 are continuously maintained for the internal states and memories of the first core decoder 810). Therefore, the memories of the second core decoder 811 are typically reset by the stereo mode switch controller (not shown) in the stereo mode switch update (see Table III). There are, however, few exceptions in the case where the primary channel SCh memory is filled by the memory of certain PCh buffers (e.g. previous excitation, previous LSF parameters and previous LSP parameters). In any case, the synthesis at the beginning of the first TD secondary channel SCh frame after switching from the DFT stereo frame 1201 to the TD stereo frame 1202 is thus subject to an imperfect reconstruction. Therefore, while the synthesis from the first core decoder 810 is well and smoothly decoded during the stereo mode switch, the limited quality synthesis from the second core decoder 811 introduces discontinuities during the stereo upmixing and final synthesis (862). These discontinuities are suppressed by employing the DFT stereo OLA memory during the first TD stereo output synthesis reconstruction, as described later.

[0754] The stereo mode switch controller (not shown) suppresses possible discontinuities and differences between the DFT stereo and TD stereo upmixed channels by a simple equalization of the signal energies. If the ICA target gain gICA Below 1.0, in the first TD stereo frame 1202 after stereo mode switching, the following relationship is used to change the channels l, y after upmixing (862) and before time synchronization (864) L（i） :

[0755]

[0756] where L eq is the length of the signal used for equalization, which corresponds to a long segment of 8.75 ms in the IVAS stereo decoding device 800 (e.g., corresponding to Leq= 140 samples at a 16 kHz output stereo signal sampling rate). Then, the gain factor is obtained using the following relationship:

[0757] .

[0758] Referring to Figure 12 , example A) involves the missing portion 1203 of the TD stereo upmix synchronization synthesis (from operation 864) of the TD stereo frame 1202 corresponding to the same as the previous DFT stereo upmix synchronization synthesis memory from the DFT stereo frame 1201. This memory of length (3.25 - 1.25) ms is not available except for its first 0.125 ms long segment 1204 when switching from the DFT stereo frame 1201 to the TD stereo frame 1202.

[0759] Figure 13 is a flowchart illustrating example A) of Figure 12 , including updating the TD stereo upmix synchronization synthesis memory in the first TD stereo frame after switching from DFT stereo mode to TD stereo mode at the decoder side.

[0760] Referring to Figure 12 and Figure 13 , the stereo mode switching controller (not shown) reconstructs the 3.25 ms of the TD stereo upmix synchronization synthesis 1205 using the following operations (a) to (e) for the left l and right r channels:

[0761] (a) Correct the DFT stereo OLA synthesis memory (defined above) (i.e., apply the inverse synthesis window to the OLA synthesis memory; see 1301).

[0762] (b) The first 0.125 ms portion 1302 of the TD stereo upmix synchronization synthesis 1303 (see Figure 12The first part of the TD stereo upmix synthesis 1303 (see 1302) is identical to the previous DFT stereo upmix synthesis 1204 and is therefore re-used to form the first part of the TD stereo upmix synthesis 1303.

[0763] (c) The second part of the TD stereo upmix synthesis 1303 (see 1305) with a length of (3.125 - 1.25) ms is approximated with the modified DFT stereo OLA synthesis memory 1301. Figure 12

[0764] (d) The parts of the TD stereo upmix synthesis 1303 of length 2 ms from the previous two steps (b) and (c) are then padded to the output stereo synthesis in the first TD stereo frame 1202.

[0765] (e) At the beginning of the TD stereo sync upmix synthesis 1305, a transition between the previous DFT stereo OLA synthesis memory 1301 and the TD sync upmix synthesis 1305 from the operation 864 of the current TD stereo frame 1202 is smoothed. The transition segment is 1.25 ms long (see 1306) and is obtained using a cross-fade 1307 between the modified DFT stereo OLA synthesis memory 1301 and the TD stereo sync upmix synthesis 1305.

[0766] 2.5 Switching from TD stereo mode to MDCT stereo mode in the IVAS stereo decoding device

[0767] Switching from TD stereo mode to MDCT stereo mode is relatively simple, since both stereo modes process two transport channels and employ two core decoder instances.

[0768] Since the inverted downmixing scheme is employed in the TD stereo encoder 400, the stereo mode switch controller (not shown) similarly changes the TD stereo channel upmixing to maintain the correct phase of the left and right channels of the stereo sound signal in the last TD stereo frame before the first MDCT stereo frame. Specifically, the stereo mode switch controller (not shown) sets the mixing ratio β = 1.0 and implements inverted upmixing of the TD stereo primary channel PCh(i) and the TD stereo secondary channel SCh(i) (as opposed to the inverted downmixing employed in the TD stereo encoder 400) to compute the MDCT stereo past left channel l past (i) and the MDCT stereo past right channel r pas t(i). Thus, the TD stereo primary channel PCh(i) is in phase with the MDCT stereo past left channel l past ​(i) Same, the TD stereo secondary channel SCh(i) signal is the same as the MDCT stereo past right channel rpast(i).

[0769] 2.6 Switching from MDCT stereo mode to TD stereo mode in an IVAS stereo decoding device

[0770] Similar to the switching from TD stereo mode to MDCT stereo mode, in this scenario two transport channels are available and two core decoder instances are employed. In order to maintain the correct phase of the left and right channels of the stereo sound signal, a stereo mode switching controller (not shown) sets the TD stereo mix ratio to 1.0 in the first TD stereo frame after the last MDCT stereo frame and again uses the anti-phase upmixing scheme.

[0771] 2.7 Switching from DFT stereo mode to MDCT stereo mode in an IVAS stereo decoding device

[0772] In this scenario a similar mechanism as the decoder side switching from DFT stereo mode to TD stereo mode is used, where the primary PCh and secondary SCh channels of the TD stereo mode are replaced by the left l and right r channels of the MDCT stereo mode.

[0773] 2.8 Switching from MDCT stereo mode to DFT stereo mode in an IVAS stereo decoding device

[0774] In this scenario a similar mechanism as the decoder side switching from TD stereo mode to DFT stereo mode is used, where the primary PCh and secondary SCh channels of the TD stereo mode are replaced by the left l and right r channels of the MDCT stereo mode.

[0775] Finally, regardless of the current stereo mode, the decoding continues with IC-BWE decoding 865 (skipped in MDCT stereo mode), addition of HB synthesis (skipped in MDCT stereo mode), time ICA alignment 866 (skipped in MDCT stereo mode) and general stereo decoder updates.

[0776] 2.9 Hardware implementation

[0777] Figure 14 is a simplified block diagram of an example configuration of the hardware components forming each of the above described IVAS stereo encoding device 200 and IVAS stereo decoding device 800.

[0778] Each of the IVAS stereo encoding device 200 and the IVAS stereo decoding device 800 can be implemented as part of a mobile terminal, as part of a portable media player, or in any similar device. Each of the IVAS stereo encoding device 200 and the IVAS stereo decoding device 800 (identified as 1400 in Figure 14 includes an input 1402, an output 1404, a processor 1406, and a memory 1408.

[0779] The input 1402 is configured to receive the left, l, and right, r, channels of an input stereo sound signal in digital or analog form in the case of the IVAS stereo encoding device 200, or to receive the bitstream 830 in the case of the IVAS stereo decoding device 800. The output 1404 is configured to provide the multiplexed bitstream 206 in the case of the IVAS stereo encoding device 200, or to provide the decoded left, l, and right, r, channels in the case of the IVAS stereo decoding device 800. The input 1402 and the output 1404 can be implemented in a general purpose module, such as a serial input / output device.

[0780] The processor 1406 is operatively connected to the input 1402, the output 1404, and the memory 1408. The processor 1406 is implemented as one or more processors for executing code instructions to support the functionality of the various elements and operations of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 described above, as shown in the figures and / or as described in the present disclosure.

[0781] The memory 1408 can include non-transitory memory for storing code instructions executable by the processor 1406, specifically, processor-readable memory storing non-transitory instructions that, when executed, cause the processor to implement the elements and operations of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850. The memory 1408 can also include random access memory or one or more buffers to store intermediate processing data for the various functions performed by the processor 1406.

[0782] One of ordinary skill in the art will recognize that the description of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 is merely illustrative and is not intended to be limiting in any way. Other embodiments will readily suggest themselves to persons of ordinary skill in the art having the benefit of the present disclosure. Further, the disclosed IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 can be customized to provide valuable solutions to existing needs and problems of encoding and decoding stereo.

[0783] For the sake of clarity, not all routine features of implementations of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 are shown and described. It will of course be understood that, where an actual implementation of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 is developed, many implementation-specific decisions would have to be made to achieve the developer's specific goals, such as compliance with application-, system-, network- and business-related constraints, and these specific goals will vary from one implementation to another and from one developer to another. Further, it will be understood that a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of the present disclosure.

[0784] In accordance with the present disclosure, the elements, processing operations and / or data structures described herein can be implemented using various types of operating systems, computing platforms, network devices, computer programs, and / or general purpose machines. In addition, those of ordinary skill in the art will recognize that devices of less general purpose can also be used, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or like devices, to implement the methods described herein. When the methods described herein are implemented by a processor, computer, or machine, and these operations and sub-operations can be stored as a series of non-transitory code instructions that are readable by the processor, computer, or machine, they can be stored on a tangible and / or non-transitory medium.

[0785] The elements and processing operations of the IVAS stereo encoding device 200, the IVAS stereo encoding method 250, the IVAS stereo decoding device 800, and the IVAS stereo decoding method 850 as described herein can include software, firmware, hardware, or any combination of software, firmware, or hardware suitable for the purposes described herein.

[0786] In the IVAS stereo encoding method 250 and the IVAS stereo decoding method 850 described herein, various processing operations and sub-operations can be performed in various orders, and some processing operations and sub-operations can be optional.

[0787] Although the present disclosure has been described above with the aid of non-limiting illustrative embodiments, these embodiments can be modified in a variety of ways without departing from the spirit and nature of the application.

[0788] The present disclosure refers to the following references, the entire contents of which are hereby incorporated by reference:

[0789] [1] 3GPP TS 26.445, v. 12.0.0, “Codec for Enhanced Voice Services (EVS); Detailed Algorithmic Description”, Sep 2014.

[0790] [2] M. Neuendorf, M. Multrus, N. Rettelbach, G. Fuchs, J. Robillard, J. Lecompte, S. Wilde, S. Bayer, S. Disch, C. Helmrich, R. Lefevbre, P. Gournay, et al., “The ISO / MPEG Unified Speech and Audio Coding Standard - Consistent High Quality for All Content Types and at All Bit Rates”, J. Audio Eng. Soc., vol. 61, no. 12, pp. 956-977, Dec. 2013.

[0791] [3] F. Baumgarte, C. Faller, “Binaural cue coding - Part I: Psychoacoustic fundamentals and design principles,” IEEE Trans. Speech Audio Processing, vol. 11, pp. 509-519, Nov. 2003.

[0792] [4] T. Vaillancourt, “Method and system using a long-term correlation difference between left and right channels for time domain down mixing a stereo sound signal into primary and secondary channels,” PCT Application WO2017 / 049397A1.

[0793] [5] V. Eksler, “Method and Device for Allocating a Bit-Budget between Sub- Frames in a CELP Codec,” PCT Application WO2019 / 056107A1.

[0794] [6] M. Neuendorf et al., “MPEG Unified Speech and Audio Coding - The ISO / MPEG Standard for High-Efficiency Audio Coding of all Content Types”, Journal of the Audio Engineering Society, vol. 61, n° 12, pp. 956-977, December 2013.

[0795] [7] J. Herre et al., “MPEG-H Audio - The New Standard for Universal Spatial / 3D Audio Coding”, in 137th International AES Convention, Paper 9095, Los Angeles, October 9-12, 2014.

[0796] [8] 3GPP SA4 contribution S4-180462, “On spatial metadata for IVAS spatial audio input format”, SA4 meeting #98, April 9-13, 2018, https: / / www.3gpp.org / ftp / tsg_sa / WG4_CODEC / TSGS4_98 / Docs / S4-180462.zip

[0797] [9] V. Malenovsky, T. Vaillancourt, “Method and Device for Classification of Uncorrelated Stereo Content, Cross-Talk Detection, and Stereo Mode Selection in a Sound Codec,” US Provisional Patent Application 63 / 075,984 filed on September 9, 2020.

Claims

1. A stereo audio signal encoding device, comprising: A first stereo encoder using a first stereo mode operating in the time domain (TD) of a stereo audio signal, wherein the first TD stereo mode in a TD frame of the stereo audio signal: (a) generates a first downmixed signal, and (b) uses a first data structure and memory. A second stereo encoder is used for a stereo audio signal operating in a second stereo mode in the frequency domain (FD), wherein the second FD stereo mode in the FD frame of the stereo audio signal: (a) generates a second downmixed signal, and (b) uses a second data structure and memory. The controller switches between (i) a first TD stereo mode and a first stereo encoder and (ii) a second FD stereo mode and a second stereo encoder to encode stereo audio signals in the time domain or the frequency domain. When switching from one stereo mode (first TD stereo mode) to the other stereo mode (second FD stereo mode), the stereo mode switching controller recalculates at least one length of the downmixed signal in the current frame of the stereo audio signal, wherein the recalculated downmixed signal length in the first TD stereo mode is different from the recalculated downmixed signal length in the second FD stereo mode.

2. The stereo audio signal encoding device as described in claim 1, wherein the second FD stereo mode is a Discrete Fourier Transform (DFT) stereo mode.

3. The stereo audio signal encoding device as described in claim 2, wherein, When switching from one stereo mode to another stereo mode between the first TD and the second DFT stereo modes, the stereo mode switching controller allocates / releases data structures to / from the first TD and the second DFT stereo modes according to the current stereo mode in order to reduce memory impact by retaining only those data structures used in the current frame.

4. The stereo audio signal encoding device as described in claim 3, wherein, When switching from the first TD stereo mode to the second DFT stereo mode, the stereo mode switching controller releases the TD stereo related data structure.

5. The stereo audio signal encoding device as described in claim 4, wherein, The TD stereo related data structure includes the TD stereo data structure and / or the data structure of the core encoder of the first stereo encoder.

6. The stereo audio signal encoding device as described in any one of claims 2 to 5, wherein, When switching from the first TD stereo mode to the second DFT stereo mode, the second stereo encoder continues the core encoding operation in the DFT stereo frame following the TD stereo frame using the main channel PCh core encoder memory.

7. The stereo audio signal encoding device according to any one of claims 2 to 5, wherein the stereo mode switching controller updates the stereo-related parameters of the other stereo mode using stereo-related parameters from the one stereo mode when switching from the one stereo mode to the other stereo mode.

8. The stereo audio signal encoding device as claimed in claim 7, wherein the stereo correlation parameters include the side gain and inter-channel time delay (ITD) parameters of the second DFT stereo mode and the target gain and correlation hysteresis of the first TD stereo mode.

9. The stereo audio signal encoding apparatus according to any one of claims 2 to 5, wherein the stereo mode switching controller updates the DFT analysis memory in each TD frame by storing samples related to the last time period of the current TD frame.

10. The stereo audio signal encoding apparatus according to any one of claims 2 to 5, wherein the stereo mode switching controller maintains the DFT-related memory during TD frames.

11. The stereo audio signal encoding device according to any one of claims 2 to 5, wherein when the stereo mode switching controller switches from a first TD stereo mode to a second DFT stereo mode, it updates the DFT synthesis memory in the DFT frame following the TD frame using the TD stereo memory of the main channel PCh corresponding to the TD frame.

12. The stereo audio signal encoding apparatus of any one of claims 2 to 5, wherein the stereo mode switching controller maintains a finite impulse response (FIR) resampling filter memory during a DFT frame of the stereo audio signal, and wherein the stereo mode switching controller updates the FIR resampling filter memory used in the master channel PCh of the first stereo encoder in each DFT frame using a segment of the middle channel m preceding the last segment of a first length in the DFT frame.

13. The stereo audio signal encoding apparatus of claim 12, wherein the stereo mode switching controller fills the FIR resampling filter memory used in the sub-channel SCh of the first stereo encoder, which is different from the update of the FIR resampling filter memory used in the main channel PCh of the first stereo encoder.

14. The stereo audio signal encoding apparatus of claim 13, wherein the stereo mode switching controller updates the FIR resampling filter memory used in the sub-channel SCh of the first stereo encoder in the current TD frame by filling the FIR resampling filter memory with a segment of the middle channel m preceding the last segment of the second length of the middle channel m in the DFT frame.

15. The stereo audio signal encoding device according to any one of claims 2 to 5, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, the stereo mode switching controller recalculates the length of the sub-mix signal in the secondary channel SCh that is longer than the recalculated length of the sub-mix signal in the primary channel PCh in the current TD frame.

16. The stereo audio signal encoding device according to any one of claims 2 to 5, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, the stereo mode switching controller cross-fades the recalculated master channel PCh and DFT middle channel m of the DFT stereo channels to recalculate the master sub-mix channel PCh in the first TD frame after the DFT frame.

17. The stereo audio signal encoding device according to any one of claims 2 to 5, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, the stereo mode switching controller recalculates the ICA memory of the left l and right r channels corresponding to the DFT frames preceding the TD frames.

18. The stereo audio signal encoding apparatus of claim 17, wherein the stereo mode switching controller recalculates the primary PCh and secondary SCh channels of the DFT frame by downmixing the channels l and r processed by the ICA using the stereo mixing ratio of the DFT frame.

19. The stereo audio signal encoding device as described in claim 18, wherein, When there is no stereo mode switching, the stereo mode switching controller recalculates the shorter length of the sub-channel SCh.

20. The stereo audio signal encoding apparatus of claim 18, wherein the stereo mode switching controller recalculates a first length of the main channel PCh and a second length of the secondary channel SCh in a DFT frame preceding the TD frame, and wherein the first length is shorter than the second length.

21. The stereo audio signal encoding apparatus according to any one of claims 2 to 5, wherein the stereo mode switching controller stores two values ​​of the pre-emphasis filter memory in each DFT frame of the stereo audio signal.

22. The stereo audio signal encoding device as described in any one of claims 2 to 5, comprising a secondary channel Sch core encoder data structure, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, the stereo mode switching controller resets or estimates the sub-channel SCh core encoder data structure based on the main PCh channel core encoder data structure.

23. A stereo audio signal decoding device, comprising: A first stereo decoder uses a stereo audio signal in a first stereo mode that operates in the time domain (TD), wherein the first stereo decoder in the TD frame of the stereo audio signal: (a) decodes the undermixed signal, and (b) uses a first data structure and memory. A second stereo decoder is used for a stereo audio signal in a second stereo mode that operates in the frequency domain (FD), wherein the second stereo decoder in the FD frame of the stereo audio signal: (a) decodes a second downmixed signal, and (b) uses a second data structure and memory; The controller switches between (i) a first TD stereo mode and a first stereo decoder and (ii) a second FD stereo mode and a second stereo decoder; When switching from one stereo mode (first TD stereo mode) to the other stereo mode (second FD stereo mode), the stereo mode switching controller recalculates at least one length of the downmixed signal in the current frame of the stereo audio signal, wherein the recalculated downmixed signal length in the first TD stereo mode is different from the recalculated downmixed signal length in the second FD stereo mode.

24. The stereo audio signal decoding device as claimed in claim 23, wherein the second FD stereo mode is a discrete Fourier transform (DFT) stereo mode.

25. The stereo audio signal decoding device of claim 24, wherein the first TD stereo mode uses a first processing delay, the second DFT stereo mode uses a second processing delay, and the first and second processing delays are different and include resampling and upmixing processing delays.

26. The stereo audio signal decoding device of any one of claims 24 to 25, wherein the stereo mode switching controller allocates / releases data structures to / from the first TD and the second DFT stereo modes according to the current stereo mode, so as to reduce the impact of static memory by retaining only those data structures used in the current frame.

27. The stereo audio signal decoding device as described in any one of claims 24 to 25, wherein, When the first DFT frame following the TD frame is received, the stereo mode switching controller resets the DFT stereo data structure.

28. The stereo audio signal decoding device as described in any one of claims 24 to 25, wherein, When the first TD frame following the DFT frame is received, the stereo mode switching controller resets the TD stereo data structure.

29. The stereo audio signal decoding device according to any one of claims 24 to 25, wherein the stereo mode switching controller updates the DFT stereo OLA memory buffer in each TD stereo frame.

30. The stereo audio signal decoding apparatus of any one of claims 24 to 25, wherein the stereo mode switching controller updates the DFT stereo analysis memory, and wherein, Upon receiving the first DFT frame after the TD frame, the stereo mode switching controller uses several last samples of the main channel PCh and the secondary channel SCh from the TD frame to update the DFT stereo analysis memory of the DFT stereo center channel m and the side channel s in the DFT frame, respectively.

31. The stereo audio signal decoding device according to any one of claims 24 to 25, wherein the stereo mode switching controller updates the DFT stereo synthesis memory in each TD stereo frame.

32. The stereo audio signal decoding device as described in claim 31, wherein, In order to update the DFT stereo synthesis memory and for the ACELP core, the stereo mode switching controller reconstructs the first part of the DFT stereo synthesis memory in each TD frame by crossfading (a) CLDFB-based resampling and TD-mixed left and right channel synthesis and (b) reconstructed resampling and TD-mixed left and right channel synthesis.

33. The stereo audio signal decoding device according to any one of claims 24 to 25, wherein the stereo mode switching controller cross-fading TD alignment and synchronization synthesis and DFT stereo alignment and synchronization synthesis are used to smoothly transition from TD frames to DFT frames.

34. The stereo audio signal decoding device as described in any one of claims 24 to 25, wherein, If the next frame is a TD frame, the stereo mode switching controller updates the TD stereo synthesis memory during the DFT frame.

35. The stereo audio signal decoding device as described in any one of claims 24 to 25, wherein, When switching from a DFT frame to a TD frame, the stereo mode switching controller resets the memory of the core decoder of the sub-channel SCh in the first stereo decoder.

36. The stereo audio signal decoding device as described in any one of claims 24 to 25, wherein, When switching from a DFT frame to a TD frame, the stereo mode switching controller uses signal energy equalization to suppress discontinuities and differences between the mixed channels of DFT and TD stereo.

37. The stereo audio signal decoding apparatus of any one of claims 24 to 25, wherein the stereo mode switching controller reconstructs TD stereo on-mix sync synthesis, and wherein the stereo mode switching controller uses the following operations (a) to (e) for both the left and right channels to reconstruct TD stereo on-mix sync synthesis: (a) Modify the DFT stereo OLA synthesis memory; (b) Reuse the DFT stereo on-mix sync synthesis memory as the first part of the TD stereo on-mix sync synthesis; (c) The second part of the TD stereo on-mix synchronous synthesis is approximated using a modified DFT stereo OLA synthesis memory; and (d) At the start of TD stereo sync on-mix synthesis, the transition between DFT stereo on-mix sync synthesis memory and TD stereo sync on-mix synthesis is smoothed by cross-fading modified DFT stereo OLA synthesis memory and TD stereo sync on-mix synthesis.

38. A stereo audio signal encoding method, comprising: A first stereo encoder is provided that uses a first stereo mode operating in the time domain (TD) for a stereo audio signal, wherein the first TD stereo mode in a TD frame of the stereo audio signal: (a) generates a first downmixed signal, and (b) uses a first data structure and memory. A second stereo encoder is provided that uses a second stereo mode operating in the frequency domain (FD) for a stereo audio signal, wherein the second FD stereo mode in the FD frame of the stereo audio signal: (a) generates a second downmixed signal, and (b) uses a second data structure and memory; The control switches between (i) a first TD stereo mode and a first stereo encoder and (ii) a second FD stereo mode and a second stereo encoder to encode stereo audio signals in the time domain or frequency domain. When switching between the first TD stereo mode and the second FD stereo mode, the control of the stereo mode switching includes recalculating at least one length of the downmix signal in the current frame of the stereo sound signal, wherein the recalculated downmix signal length in the first TD stereo mode is different from the recalculated downmix signal length in the second FD stereo mode.

39. The stereo audio signal encoding method as described in claim 38, wherein the second FD stereo mode is a discrete Fourier transform (DFT) stereo mode.

40. The stereo audio signal encoding method as described in claim 39, wherein, When switching from one stereo mode to the other stereo mode between the first TD and the second DFT stereo modes, controlling the stereo mode switching includes maintaining the continuity of at least one of the following signals: Including the input stereo signal of the left and right channels; The middle channel used in the second DFT stereo mode; The primary and secondary channels used in the first TD stereo mode; The downmixed signal used in preprocessing; and The downmixed signal used in the core encoding.

41. The stereo audio signal encoding method as described in claim 39 or 40, wherein, When switching from one stereo mode to the other stereo mode in the first TD and second DFT stereo modes, controlling the stereo mode switching includes allocating / releasing data structures to / from the first TD and second DFT stereo modes according to the current stereo mode, in order to reduce memory impact by retaining only those data structures used in the current frame.

42. The stereo audio signal encoding method as described in claim 41, wherein, When switching from the first TD stereo mode to the second DFT stereo mode, controlling the stereo mode switching includes releasing the TD stereo related data structure, wherein the TD stereo related data structure includes the TD stereo data structure and / or the data structure of the core encoder of the first stereo encoder.

43. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein, When switching from the first TD stereo mode to the second DFT stereo mode, the second stereo encoder continues the core encoding operation in the DFT frame following the TD frame using the main channel PCh core encoder memory.

44. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling stereo mode switching includes updating the stereo correlation parameters of the other stereo mode using stereo correlation parameters from the one stereo mode when switching from the one stereo mode to the other stereo mode.

45. The stereo audio signal encoding method of claim 44, wherein controlling stereo mode switching includes transmitting stereo-related parameters between data structures, and wherein the stereo-related parameters include side gain and inter-channel time delay (ITD) parameters of a second DFT stereo mode and target gain and related hysteresis of a first TD stereo mode.

46. ​​The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling the stereo mode switching includes updating the DFT analysis memory in each TD stereo frame by storing samples related to the last time period of the current TD stereo frame.

47. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling stereo mode switching includes maintaining DFT-related memory during TD stereo frames.

48. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling the stereo mode switching includes updating the DFT synthesis memory using the TD stereo memory corresponding to the main channel PCh of the TD frame in the DFT frame after the TD frame when switching from a first TD stereo mode to a second DFT stereo mode.

49. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling the stereo mode switching includes maintaining a finite impulse response (FIR) resampling filter memory during DFT frames.

50. The stereo audio signal encoding method of claim 49, wherein controlling the stereo mode switching includes updating the FIR resampling filter memory used in the master channel PCh in the first stereo encoder using a segment of the middle channel m before the last segment of the first length of the middle channel m in each DFT frame.

51. The stereo audio signal encoding method of claim 50, wherein controlling the stereo mode switching includes filling the FIR resampling filter memory used in the secondary channel SCh of the first stereo encoder, which is different from the update of the FIR resampling filter memory used in the main channel PCh of the first stereo encoder.

52. The stereo audio signal encoding method of claim 51, wherein controlling the stereo mode switching includes updating the FIR resampling filter memory used in the sub-channel SCh in the first stereo encoder in the current TD frame by filling the FIR resampling filter memory with a segment of the middle channel m before the last segment of the second length of the middle channel m in the DFT frame.

53. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, controlling the stereo mode switching includes recalculating the length of the sub-mix signal in the secondary channel SCh that is longer than the recalculated length of the sub-mix signal in the primary channel PCh in the current TD frame.

54. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, the control of the stereo mode switching includes recalculating the main channel PCh and the DFT intermediate channel m of the crossfading DFT channel, so as to recalculate the main submix channel PCh in the first TD frame after the DFT frame.

55. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, controlling the stereo mode switching includes recalculating the ICA memory of the left l and right r channels corresponding to the DFT frames preceding the TD frame.

56. The stereo audio signal encoding method of claim 55, wherein controlling the stereo mode switching includes recalculating the primary PCh and secondary SCh channels of the DFT frame by downmixing the channels l and r processed by the DFT frame using the stereo mixing ratio of the DFT frame.

57. The stereo audio signal encoding method as described in claim 56, wherein, When there is no stereo codec mode switching, controlling stereo mode switching includes recalculating the shorter length of the sub-channel SCh.

58. The stereo audio signal encoding method of claim 56, wherein controlling the stereo mode switching includes recalculating a first length of the main channel PCh and a second length of the secondary channel SCh in a DFT frame prior to the TD frame, and wherein the first length is shorter than the second length.

59. The stereo audio signal encoding method according to any one of claims 39 to 40, wherein controlling the stereo mode switching includes storing two values ​​of the pre-emphasis filter memory in each DFT frame.

60. The stereo audio signal encoding method according to any one of claims 39 to 40, comprising a sub-channel Sch core encoder data structure, wherein, When switching from the second DFT stereo mode to the first TD stereo mode, controlling the stereo mode switching includes resetting or estimating the sub-channel SCh core encoder data structure based on the main PCh channel core encoder data structure.

61. A method for decoding stereo audio signals, comprising: A first stereo decoder is provided that uses a stereo audio signal operating in a first stereo mode in the time domain (TD), wherein the first stereo decoder in the TD frame of the stereo audio signal: (a) decodes the undermixed signal, and (b) uses a first data structure and memory; A second stereo decoder is provided that uses a second stereo mode of a stereo audio signal operating in the frequency domain (FD), wherein the second stereo decoder in the FD frame of the stereo audio signal: (a) decodes a second downmixed signal, and (b) uses a second data structure and memory; Controls switching between (i) a first TD stereo mode and a first stereo decoder and (ii) a second FD stereo mode and a second stereo decoder; When switching between the first TD stereo mode and the second FD stereo mode, the control of the stereo mode switching includes recalculating at least one length of the downmix signal in the current frame of the stereo sound signal, wherein the recalculated downmix signal length in the first TD stereo mode is different from the recalculated downmix signal length in the second FD stereo mode.

62. The stereo audio signal decoding method as described in claim 61, wherein the second FD stereo mode is a discrete Fourier transform (DFT) stereo mode.

63. The stereo audio signal decoding method of claim 62, wherein the first TD stereo mode uses a first processing delay, the second FD stereo mode uses a second processing delay, and the first and second processing delays are different and include resampling and upmixing processing delays.

64. The stereo audio signal decoding method as described in claim 62 or 63, wherein, When switching from one stereo mode (first TD and second DFT stereo modes) to another stereo mode (first TD and second DFT stereo modes), controlling the stereo mode switching includes maintaining the continuity of at least one of the following signals and memories: The middle channel m used in the second DFT stereo mode; The primary channel PCh and secondary channel SCh used in the first TD stereo mode; TCX-LTP post-filter memory; DFT OLA analysis memory at internal sampling rate and output stereo signal sampling rate; DFT OLA synthesis memory at the output stereo signal sampling rate; Output stereo signal, including channels l and r; and HB signal memory, and the audio channels l and r used in BWE and IC-BWE.

65. The stereo audio signal decoding method of any one of claims 62 to 63, wherein controlling the stereo mode switching includes allocating / releasing data structures to / from a first TD and a second DFT stereo mode according to the current stereo mode, in order to reduce the impact of static memory by retaining only those data structures used in the current frame.

66. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein, When the first DFT frame following the TD frame is received, controlling the stereo mode switching includes resetting the DFT stereo data structure.

67. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein, When the first TD frame following the DFT frame is received, controlling the stereo mode switching includes resetting the TD stereo data structure.

68. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein controlling the stereo mode switching includes updating the DFT stereo OLA memory buffer in each TD frame.

69. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein controlling the stereo mode switching includes updating the DFT stereo analysis memory.

70. The stereo audio signal decoding method as described in claim 69, wherein, Upon receiving the first DFT frame after the TD frame, controlling the stereo mode switching involves using several last samples of the main channel PCh and secondary channel SCh from the TD frame to update the DFT stereo analysis memory of the DFT stereo center channel m and side channel s, respectively, in the DFT frame.

71. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein controlling the stereo mode switching includes updating the DFT stereo synthesis memory in each TD frame, and wherein, In order to update the DFT stereo synthesis memory and for the ACELP core, controlling the stereo mode switching includes reconstructing the first part of the DFT stereo synthesis memory in each TD frame by crossfading (a) CLDFB-based resampling and TD-mixed left and right channel synthesis and (b) reconstructed resampling and TD-mixed left and right channel synthesis.

72. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein controlling the stereo mode switching includes crossfading TD alignment and synchronous synthesis and DFT stereo alignment and synchronous synthesis to smoothly transition when switching from a TD frame to a DFT frame.

73. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein, If the next frame is a TD frame, controlling the stereo mode switching includes updating the TD stereo synthesis memory during the DFT frame.

74. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein, When switching from a DFT frame to a TD frame, controlling the stereo mode switching includes resetting the memory of the core decoder of the sub-channel SCh in the first stereo decoder.

75. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein, When switching from a DFT frame to a TD frame, controlling the stereo mode switching includes using signal energy equalization to suppress discontinuities and differences between the DFT and TD stereo upmixed channels, and wherein, in order to suppress discontinuities and differences between the DFT and TD stereo upmixed channels, if the ICA target gain g ICA Below 1.0, controlling stereo mode switching involves using the following relationship in the TD frame to change the left channel l, y after overmixing and before time synchronization. L (i): L eq It is the length of the signal to be balanced, and The value of the gain factor is obtained using the following relationship: 。 76. The stereo audio signal decoding method according to any one of claims 62 to 63, wherein controlling the stereo mode switching includes reconstructing the TD stereo upmixing synchronous synthesis, and wherein controlling the stereo mode switching includes reconstructing the TD stereo upmixing synchronous synthesis by applying the following operations (a) to (e) to both the left and right channels: (a) Modify the DFT stereo OLA synthesis memory; (b) Reuse the DFT stereo on-mix sync synthesis memory as the first part of the TD stereo on-mix sync synthesis; (c) The second part of the TD stereo on-mix synchronous synthesis is approximated using a modified DFT stereo OLA synthesis memory; and (d) At the start of TD stereo sync on-mix synthesis, the transition between DFT stereo on-mix sync synthesis memory and TD stereo sync on-mix synthesis is smoothed by cross-fading modified DFT stereo OLA synthesis memory and TD stereo sync on-mix synthesis.

Images