Audio signal decoding and encoding device and method using adaptive spectrum tile selection
By employing adaptive spectrum piling selection and spectrum envelope shaping techniques, the problem of bandwidth expansion in existing audio coding at low bit rates is solved, thereby improving audio quality and coding efficiency, reducing artifacts, and ensuring accurate reconstruction of high-frequency details.
Patent Information
- Application Number
- CN202010010552.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2013-10-18
- Filing Date
- 2014-07-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2034-12-08
AI Technical Summary
Existing audio coding techniques struggle to effectively extend audio bandwidth at low bit rates, leading to loss of high-frequency details and sound quality degradation. Meanwhile, spectrum repair methods are computationally complex and unstable, especially affecting non-tonal signals such as silent speech.
An adaptive spectrum piling selection scheme is adopted. The best matching source region is analyzed and identified on the encoder side. Reconstructed spectrum data is generated on the decoder side through frequency piling filling operation. Combined with spectrum envelope shaping and time noise shaping technology, it is ensured that the important content in the high-frequency part is encoded by the core encoder.
Improve audio quality at low bit rates, reduce artifacts and spatial isolation artifacts, maintain the accuracy of pitch and time envelope matching, and achieve efficient encoding and decoding across the entire frequency band.
Smart Images

Figure CN111179963B_ABST
Abstract
Description
[0001] This patent application is a divisional application of a patent application with an international application date of July 15, 2014, a national application number of 201480041566.7, and an invention name of "Device and method for decoding and encoding audio signals using adaptive spectrum tiling selection". Technical Field
[0002] The present invention relates to audio encoding / decoding and in particular to audio encoding using intelligent gap filling. Background Art
[0003] Audio coding is a signal compression field that uses psychoacoustic knowledge to process redundant and irrelevant parts in audio signals. Today, audio codecs usually require about 60kbps / channel to perform perceptually transparent encoding of almost any type of audio signal. Newer codecs are designed to reduce the encoding bit rate by utilizing the spectral similarity in the signal using technologies such as bandwidth extension (BWE). The BWE scheme uses a low bit rate parameter set to represent the high frequency (HF) component of the audio signal. The HF spectrum is filled with spectral content from the low frequency (LF) region, and the spectral shape, tilt and time continuity are adjusted to maintain the sound quality and timbre of the original signal. Such a BWE method enables the audio codec to maintain good quality even at a low bit rate of about 24kbps / channel.
[0004] The storage or transmission of audio signals is often subject to strict bit rate constraints. In the past, encoders were forced to drastically reduce the transmission audio bandwidth only when very low bit rates were available.
[0005] Modern audio codecs are now able to encode wideband signals by using bandwidth extension (BWE) methods [1]. These algorithms rely on a parameterized representation of the high frequency content (HF), which is generated by transferring the waveform-encoded low frequency part (LF) of the decoded signal into the HF spectral region ("patching"), and the application of parameter-driven post-processing. According to the BWE scheme, the reconstruction of the HF spectral region above a given so-called crossover frequency is usually based on spectral patching. Typically, the HF region comprises a plurality of adjacent patches, and each of these patches originates from a bandpass (BP) region of the LF spectrum below a given crossover frequency. Prior art systems effectively perform patching within a filter bank representation, such as a quadrature mirror filter bank (QMF), by copying a set of adjacent subband coefficients from a source region to a target region.
[0006] Another technique found in today's audio codecs to improve compression efficiency and thus enable extension of audio bandwidth at low bit rates is parameter-driven synthetic replacement of appropriate portions of the audio spectrum. For example, noise-like signal portions of the original audio signal can be replaced with artificial noise generated in the decoder and scaled by side information parameters, with substantially no loss in subjective quality. An example is the Perceptual Noise Substitution (PNS) tool included in MPEG-4 Advanced Audio Coding (AAC) [5].
[0007] Another technique that also enables bandwidth extension at low bit rates is the noise filling technique included in MPEG-D Unified Speech and Audio Coding (USAC) [7]. The spectral gaps (zeros) resulting from the dead zones of the quantizer due to too coarse quantization are then filled with artificial noise in the decoder and scaled by parameter-driven post-processing.
[0008] Another prior art system is known as exact spectral replacement (ASR) [2-4]. In addition to the waveform codec, ASR employs a dedicated signal synthesis stage which restores the perceptually important sinusoidal parts of the signal at the decoder. Furthermore, the system described in [5] relies on sinusoidal modeling in the HF region of the waveform coder to enable extended audio bandwidth with decent perceptual quality at low bit rates. All of these approaches involve transforming the data to a second domain other than the modified discrete cosine transform (MDCT) and also involve fairly complex analysis / synthesis stages for preserving the HF sinusoidal components.
[0009] FIG. 13A shows a schematic diagram of an audio encoder for a bandwidth extension technique such as used in, for example, high-efficiency advanced audio coding (HE-ACC). The audio signal at line 1300 is input to a filter system including a low-pass filter 1302 and a high-pass filter 1304. The signal output by the high-pass filter 1304 is input to a parameter extractor / encoder 1306. For example, the parameter extractor / encoder 1306 is configured to calculate and encode parameters such as spectral envelope parameters, noise addition parameters, lost harmonic parameters or inverse filtering parameters. These extracted parameters are input to a bitstream multiplexer 1308. The low-pass output signal is input to a processor that generally includes the functions of a downsampler 1310 and a core encoder 1312. The bandwidth to be encoded by the low-pass filter 1302 is limited to a bandwidth that is significantly smaller than the bandwidth that appears in the original input audio signal on line 1300. This provides a significant coding gain due to the fact that all functions that appear in the core encoder only have to operate on signals with reduced bandwidth. When, for example, the bandwidth of the audio signal on line 1300 is 20 kHz and when the low-pass filter 1302 has a bandwidth of, for example, 4 kHz, in order to satisfy the sampling theorem, it is theoretically sufficient that the sampling frequency of the signal after the downsampler is 8 kHz, which is essentially reduced to the required sampling rate of the audio signal 1300, which must be at least 40 kHz.
[0010] Figure 13 B shows the schematic diagram of corresponding bandwidth extension decoder.Decoder comprises bitstream demultiplexer 1320.Bitstream demultiplexer 1320 extracts the input signal of core decoder 1322 and the input signal of parameter decoder 1324.In the example above, the sampling rate of core decoder output signal is 8kHz and therefore bandwidth is 4kHz, yet, in order to complete bandwidth reconstruction, the output signal of high frequency reconstruction device 1330 must be in 20kHz, and it requires the sampling rate of at least 40kHz.In order to make this possible, need to have the decoder processor of the function of up sampler 1325 and filter bank 1326.Then, high frequency reconstruction device 1330 receives the low frequency signal through frequency analysis output by filter bank 1326, and uses the parameterization of high frequency band to represent that the frequency range limited by the high pass filter 1304 of Figure 13 A is reconstructed. The high frequency reconstructor 1330 has several functions such as using the source range in the low frequency range to regenerate the upper frequency range, spectrum envelope adjustment, noise addition function and the function of introducing lost harmonics in the upper frequency range, and in order to explain the fact that the higher frequency range usually has a different tone from the lower frequency range, if applied and calculated in the encoder of Figure 13A, an inverse filtering operation is also included. In HE-ACC, the lost harmonics are resynthesized on the decoder side and are accurately placed in the middle of the reconstructed frequency band. Therefore, all the lost harmonic lines determined in a certain reconstructed frequency band are not placed at their frequency values in the original signal. Instead, those lost harmonic lines are placed at the frequency at the center of a certain frequency band. Thus, when the lost harmonic line in the original signal is placed very close to the reconstructed frequency band boundary in the original signal, the error of the frequency introduced by placing the lost harmonic line in the reconstructed signal at the center of the frequency band is close to 50% of the individual reconstructed frequency bands for which parameters are generated and sent.
[0011] Furthermore, while a typical audio core encoder operates in the spectral domain, the core decoder generates a time domain signal which is then converted back to the spectral domain by the filter bank 1326 function. This introduces additional processing delays, may introduce artifacts due to the cascade of first transforming from the spectral domain to the frequency domain and back again to the generally different frequency domain, and of course requires a great deal of computational complexity and therefore electrical power, which is particularly an issue when bandwidth extension techniques are applied to mobile devices such as mobile phones, tablets or laptops etc.
[0012] Current audio codecs use BWE as an integral part of the coding scheme to perform low bitrate audio coding. However, BWE techniques are limited to replacing only high frequency (HF) content. Furthermore, they cannot enable perceptually important content above a given crossover frequency to be waveform encoded. Therefore, since the precise alignment of the pitch harmonics of the signal is not considered in most systems, contemporary audio codecs lose HF details or timbre when implementing BWE.
[0013] Another disadvantage of current prior art BWE systems is the need to transform the audio signal into a new domain to implement BWE (eg, from the MDCT domain to the QMF domain). This results in synchronization complications, additional computational complexity, and increased storage requirements.
[0014] Typically, bandwidth extension schemes use spectrum patching for the purpose of reconstructing a high-frequency spectrum region above a given so-called crossover frequency. The HF region includes a plurality of adjacent patches, and each of these patches originates from the same bandpass region of the low-frequency spectrum below a given crossover frequency. Within the filter bank representation of the signal, such a system copies a group of adjacent subband coefficients in the low-frequency spectrum into the HF region. The boundaries of the selected group are usually system-related and have nothing to do with the signal. For some signal contents, this static patching selection may result in a bad timbre and coloration of the reconstructed signal.
[0015] Other methods transfer the LF signal to the HF region by signal adaptive single sideband (SSB) modulation. Compared with the replication process, such methods have high computational complexity because they operate on the time domain signal at a high sampling rate.
[0016] Furthermore, the patching may be unstable, especially for non-tonal signals such as unvoiced speech. Thus, known patching schemes may introduce impairments to the audio signal. Summary of the invention
[0017] It is an object of the present invention to provide an improved encoding / decoding concept resulting in better audio quality.
[0018] The object is achieved by a decoding device, an encoding device, an audio decoding method or a storage medium having a computer program stored thereon.
[0019] Another aspect is based on the following discovery: the damage of audio quality can be compensated by applying a signal adaptive frequency tile filling scheme. For this reason, analysis is performed on the encoder side to find the most matching candidate source area of a certain target area. Matching information for a certain source area identified by the target area is generated together with some additional information optionally, and is sent to the decoder as side information. Then, the decoder uses this matching information to apply the frequency tile filling operation. For this reason, the decoder reads matching information from the data stream or data file sent, and accesses the source area identified for a certain reconstruction band, and if indicated in the matching information, additionally performs some processing of the source area data to generate the original spectrum data of the reconstruction band. Then, the result of the frequency tile filling operation, i.e., the original spectrum data of the reconstruction band, is shaped using spectrum envelope information to finally obtain a reconstruction band including a first spectrum part such as a tone part. However, these tone parts are not generated by the adaptive tile filling scheme, but these first spectrum parts are directly output by an audio decoder or a core decoder.
[0020] The adaptive spectrum tiling selection scheme can operate at a low granularity. In this implementation, the source region is subdivided into usually overlapping source regions, and the target region or reconstructed band is given by non-overlapping frequency target regions. Then, the similarity between each source region and each target region is determined on the encoder side, and the best matching pair of source region and target region is identified by matching information, and on the decoder side, the source region identified in the matching information is used to generate the original spectral data of the reconstructed band.
[0021] For the purpose of obtaining higher granularity, each source region is allowed to shift to obtain a certain lag where the similarity is maximal. This lag can be as fine as a region element and allows for an even better match between the source and target regions.
[0022] Furthermore, in addition to identifying only the best matching pair, this correlation lag can also be sent within the match information, and additionally, even the sign can be sent. When the sign is determined to be negative on the encoder side, a corresponding sign flag is also sent within the match information, while on the decoder side, the source region spectral value is multiplied by "-1", or in complex representation, the source region spectral value is "rotated" by 180°.
[0023] Another implementation of the present invention uses tile whitening operation. The whitening of the spectrum removes the coarse spectrum envelope information and emphasizes the spectrum fine structure that is most interested in for evaluating tile similarity. Therefore, before calculating the cross-correlation measure, on the one hand, the frequency tile and / or on the other hand, the source signal is whitened. When using a predetermined process to whiten only the tile, a whitening flag indicating that the same predetermined whitening process should be applied to the frequencies in the IGF is sent to the decoder.
[0024] Regarding tile selection, preferably the regenerated spectrum is shifted by an integer number of transform elements on the spectrum using the hysteresis of the correlation. Depending on the basic transformation, the spectrum shift may require additive correlation. In the case of an odd number of hysteresis, the tile is additionally modulated by multiplying the alternating time sequence -1 / 1 to compensate for the frequency inversion representation of each other frequency band in the MDCT. In addition, when generating frequency tiles, the sign of the correlation result is applied.
[0025] Furthermore, tile pruning and stabilization are preferably used to ensure that artifacts caused by rapidly changing source regions of the same reconstructed or target region are avoided. To this end, a similarity analysis between different identified source regions is performed, and when a source tile is similar to other source tiles with a similarity above a threshold, then the source tile can be discarded from a set of potential source tiles because it is highly correlated with other source tiles. Furthermore, as a tile selection stabilization, if none of the source tiles in the current frame is correlated with the target tile in the current frame (better than a given threshold), then the tile order from the previous frame is preferably kept.
[0026] On the other hand, it is based on the following discovery: the audio quality of the reconstructed signal can be improved by IGF, because the entire spectrum can be accessed by the core encoder, so that, for example, the perceptually important tonal parts in the high spectral range can still be encoded by the core encoder instead of parametric substitution. In addition, a gap filling operation using frequency tiles from the first group of first spectral parts is performed, and the first group of first spectral parts is, for example, a group of tonal parts that are usually from a lower frequency range but can also be from a higher frequency range when available. However, for performing spectral envelope adjustment on the encoder side, the spectral parts from the first group of spectral parts located in the reconstructed frequency band are not further post-processed by, for example, spectral envelope adjustment. Only the remaining spectral values in the reconstructed frequency band that do not originate from the core encoder are to be envelope adjusted using envelope information. Preferably, the envelope information is full-band envelope information that takes into account the energy of the first group of first spectral parts in the reconstructed frequency band and the second group of second spectral parts in the same reconstructed frequency band, wherein the latter spectral values in the second group of second spectral parts are indicated as zero, and therefore, are not encoded by the core encoder, but are parametrically encoded using low-resolution energy information.
[0027] It has been found that both the absolute energy values normalized with respect to the bandwidth of the corresponding frequency band and the unnormalized absolute energy values are useful and very efficient for the application on the decoder side. This is particularly applicable when the gain factors have to be calculated based on the residual energy in the reconstructed frequency band, the lost energy in the reconstructed frequency band and the frequency tiling information in the reconstructed frequency band.
[0028] Furthermore, it is preferred that the coded bitstream comprises not only the energy information of the reconstructed bands but also additionally the scale factors of the scale factor bands extending up to the maximum frequency. This ensures that for each reconstructed band available for a specific tonal portion, i.e. the first spectral portion, the first group of first spectral portions can actually be decoded with a suitable amplitude. Furthermore, in addition to the scale factor of each reconstructed band, the energy of the reconstructed band is generated in the encoder and transmitted to the decoder. Furthermore, it is preferred that the reconstructed bands coincide with the scale factor bands or, in case of energy grouping, at least the boundaries of the reconstructed bands coincide with the boundaries of the scale factor bands.
[0029] On the other hand, it is based on the following discovery: the problems related to the separation of bandwidth extension on the one hand and core coding on the other hand can be solved and overcome by performing bandwidth extension in the same spectral domain where the core decoder operates. Therefore, a full-rate core decoder is provided for encoding and decoding the full audio signal range. This does not require a downsampler on the encoder side and an upsampler on the decoder side. Alternatively, all processing is performed in the full sampling rate or full bandwidth domain. In order to obtain a high coding gain, the audio signal is analyzed to find the first group of first spectrum parts that must be encoded at high resolution, wherein, in one embodiment, the first group of first spectrum parts may include the tonal part of the audio signal. On the other hand, the non-tonal or noisy components constituting the second group of second spectrum parts in the audio signal are parametrically encoded at low spectral resolution. The encoded audio signal only needs the first group of first spectrum parts encoded in a waveform-preserving manner at high spectral resolution and the second group of second spectrum parts parametrically encoded at low resolution using additional frequency "slices" derived from the first group. On the decoder side, the core decoder as a full-band decoder reconstructs the first group of first spectrum parts in a waveform-preserving manner, i.e., without knowing that there is any additional frequency regeneration. However, the spectrum thus generated has many spectral gaps. These gaps are subsequently filled with the inventive Intelligent Gap Filling (IGF) technique by using on the one hand frequency regeneration applying parameterized data and on the other hand the source spectral range, i.e. the first spectral portion reconstructed by the full rate audio decoder.
[0030] In another embodiment, the spectrum portions reconstructed only by noise filling and not bandwidth replication or frequency tiling include a third group of third spectrum portions. Due to the fact that the present coding concept operates in a single domain of core coding / decoding on the one hand and in a single domain of frequency regeneration on the other hand, the IGF is not only limited to filling higher frequency ranges, but can also fill lower frequency ranges by noise filling without frequency regeneration or by frequency regeneration using spectrum tiling at different frequency ranges.
[0031] Furthermore, it is emphasized that the information about the spectral energy, the information about the individual energy or the individual energy information, the information about the residual energy or the residual energy information, the information about the tile energy or the tile energy information, or the information about the lost energy or the lost energy information may include not only the energy value, but also (e.g., absolute) amplitude value, level value or any other value, from which the final energy value can be obtained. Thus, the information about the energy may include, for example, the energy value itself and / or the value of the level and / or the value of the amplitude and / or the value of the absolute amplitude.
[0032] Another aspect is based on the following discovery: the correlation situation is not only important for the source range, but also important for the target range. In addition, the present invention recognizes the following situation: different correlation situations can occur in the source range and the target range. For example, when considering a speech signal with high-frequency noise, the situation can be: when the speaker is placed in the middle, the low-frequency band of the voice signal including the voice signal with a small number of overtones is highly correlated in the left channel and the right channel. However, due to the fact that there may be different high-frequency noises on the left side compared with other high-frequency noises or no high-frequency noise on the right side, the high-frequency part can be strongly uncorrelated. Thus, when a simple gap filling operation that ignores this situation can be performed, the high-frequency part will also be correlated, and this can generate serious spatial isolation artifacts in the reconstructed signal. In order to solve this problem, the parameterized data of the reconstructed band is calculated or the parameterized data of the second group of the second spectrum part reconstructed by the first group of first spectrum parts must usually be used to identify the second spectrum part or the first binaural representation of the reconstructed band stated differently or the different second binaural representations. Therefore, at the encoder side, a binaural identification is calculated for the second spectral portion, i.e., for the portion where the energy information of the reconstructed frequency band is additionally calculated. Then, a frequency regenerator at the decoder side regenerates the second spectral portion based on the first set of first spectral portions, i.e., the first portion in the source range, and the parameterized data of the second portion, such as the spectral envelope energy information or any other spectral envelope data, and additionally based on the binaural identification of the second portion, i.e., the binaural identification of the reconstructed frequency band in the case of reconsideration.
[0033] Preferably, the binaural identification is sent as a flag for each reconstruction band and this data is sent from the encoder to the decoder which then decodes the core signal as indicated by the flag preferably calculated for the core band. Thus, in one implementation, the core signal is stored in two stereo representations (e.g. left / right and mid / side) and for IGF frequency tile filling, the source tile representation is selected to match the target tile representation as indicated by the binaural identification flag for the smart gap filling or reconstruction band, i.e. for the target range.
[0034] It is emphasized that this process works not only for stereo signals, i.e. for the left and right channels, but also for multi-channel signals. In the case of multi-channel signals, several different pairs of channels can be processed in this way, such as the left and right channels as a first pair, the left surround channel and the right surround channel as a second pair, and the center channel and the LFE channel as a third pair. For higher output channel formats such as 7.1, 11.1, etc., other pairs can be determined.
[0035] On the other hand, it is based on the following discovery: by combining the time noise shaping (TNS) or time tile shaping (TTS) technique with high frequency reconstruction, improved quality and reduced bit rate are obtained, especially for signals including transient parts that often appear in audio signals, the quality is improved and the bit rate is reduced. The TNS / TTS processing implemented by the prediction of the frequency on the encoder side reconstructs the time envelope of the audio signal. Depending on the implementation, that is, when the time noise shaping filter is determined in a frequency range that includes not only the source frequency range but also the target frequency range to be reconstructed in the frequency regeneration decoder, the time envelope is applied not only to the core audio signal up to the gap filling start frequency but also to the spectrum range of the reconstructed second spectrum part. Thus, the pre-echo and post-echo that appear without the use of time tile shaping are reduced or eliminated. This is done by applying the inverse prediction of the frequency not only in the core frequency range up to a certain gap filling start frequency but also in the frequency range above the core frequency range. To this end, frequency regeneration or frequency tile generation is performed on the decoder side before applying the prediction of the frequency. However, depending on whether the energy information calculation is performed on spectral residual values after filtering or on (full) spectral values before envelope shaping, the frequency-related prediction may be applied before or after spectral envelope shaping.
[0036] The TTS processing on one or more frequency tiles additionally establishes the continuity of the correlation between the source range and the reconstruction range or within two adjacent reconstruction ranges or frequency tiles.
[0037] According to an implementation, preferably complex TNS / TTS filtering is used. Thus, the (time) aliasing artifacts of the real number representation of critical sampling such as MDCT are avoided. By applying not only the modified discrete cosine transform but also the modified discrete sine transform, the complex TNS filter can be calculated on the encoder side in addition to obtaining the complex modified transform. However, only the real part of the modified discrete cosine transform value, i.e. the complex transform, is sent. However, on the decoder side, the MDCT spectrum of the previous or subsequent frame can be used to estimate the imaginary part of the transform so that the complex filter can be applied again on the decoder side in the inverse prediction about the frequency, especially in the prediction about the boundary between the source range and the reconstruction range and in the prediction about the boundary between the frequency tiles adjacent to the frequency within the reconstruction range.
[0038] The audio coding system of the present invention efficiently encodes arbitrary audio signals at a wide range of bit rates. However, for high bit rates, the system of the present invention converges on transparency, and for low bit rates, minimizes perceptual annoyance. Therefore, the main share of the available bit rate is used to waveform encode only the most perceptually relevant structures of the signal in the encoder, and the resulting spectral gaps are filled in the decoder with signal content that roughly approximates the original spectrum. A very limited bit budget is consumed to control parameter-driven so-called spectrally intelligent gap filling (IGF) through dedicated side information sent from the encoder to the decoder. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Next, preferred embodiments of the present invention will be described with reference to the accompanying drawings, in which:
[0040] Figure 1A An apparatus for encoding an audio signal is shown;
[0041] Figure 1B Shown with Figure 1A A decoder matched with the encoder for decoding the encoded audio signal;
[0042] Figure 2A A preferred implementation of a decoder is shown;
[0043] Figure 2B A preferred implementation of the encoder is shown;
[0044] Figure 3A Shown by Figure 1B Schematic representation of the spectrum generated by the spectral domain decoder of;
[0045] Figure 3B A table showing the relationship between the scale factor of the scale factor band and the energy used to reconstruct the band and the noise filling information of the noise filling band;
[0046] Figure 4A The functionality of a spectral domain encoder for applying the selection of spectral portions to a first group of spectral portions and a second group of spectral portions is shown;
[0047] Figure 4B Shows Figure 4A The realization of the function;
[0048] Figure 5A shows the functionality of the MDCT encoder;
[0049] Figure 5B shows the functionality of a decoder using MDCT technology;
[0050] Figure 5C shows the implementation of a frequency regenerator;
[0051] Fig. 6A An audio encoder with temporal noise shaping / temporal tiling shaping functionality is shown;
[0052] Figure 6B A decoder using temporal noise shaping / temporal tiling shaping techniques is shown;
[0053] Figure 6C A further function showing a temporal noise shaping / temporal tiling shaping function with a different order of spectral prediction filter and spectral shaper;
[0054] Fig. 7A The implementation of the Time Tile Shaping (TTS) function is shown;
[0055] Figure 7B Shown with Fig. 7A The encoder implementation matches the decoder implementation;
[0056] Figure 7C shows the spectrum representation of the original signal and the extended signal without TTS;
[0057] Fig.7D A frequency representation showing the correspondence between the smart gap filling frequency and the time tiling shaping energy is shown;
[0058] Fig. 7E shows the spectrogram of the original signal and the extended signal with TTS;
[0059] Fig. 8A A two-channel decoder with frequency regeneration is shown;
[0060] Figure 8B A table showing different combinations of representations and source / destination ranges is shown;
[0061] Figure 8C shows that Fig. 8A A flow chart of the frequency regeneration function of a two-channel decoder;
[0062] Fig.8D Shows Fig. 8A A more detailed implementation of the decoder;
[0063] Fig. 8E shows the Fig. 8A The decoder decodes the implementation of the encoder for binaural processing;
[0064] Fig. 9A A decoder having a frequency regeneration technique using energy values of a regeneration frequency range is shown;
[0065] Fig. 9B Shows Fig. 9A A more detailed implementation of the frequency regenerator;
[0066] Fig. 9C Shows Shows Fig. 9B Schematic diagram of the function;
[0067] Fig.9D Shows Fig. 9B Yet another implementation of the decoder;
[0068] Fig. 10A Shown with Fig. 9A The decoder is matched to the encoder block diagram;
[0069] Fig. 10B Shown for illustration Fig. 10A A block diagram of yet another function of a parameter calculator;
[0070] Fig. 10C Shows Shows Fig. 10A A block diagram of yet another function of a parameterized calculator;
[0071] Fig. 10D Shows Fig. 10A A block diagram of yet another function of a parameterized calculator;
[0072] Fig.11A Still another decoder with specific source range identification for spectral tiling operation in the decoder is shown;
[0073] Fig. 11B Shows Fig.11A Another function of the frequency regenerator;
[0074] Fig. 11C Shown for use with Fig.11A The decoder in the coder cooperates;
[0075] Fig.11D Shows Fig. 11C A block diagram of the implementation of the parameter calculator;
[0076] Fig. 12A and Fig. 12B A frequency schematic diagram for illustrating source range and target range is shown;
[0077] Fig. 12C A graph showing an example correlation of two signals;
[0078] FIG. 13A shows a prior art encoder with bandwidth extension; and
[0079] FIG. 13B shows a prior art decoder with bandwidth extension. DETAILED DESCRIPTION
[0080] Figure 1A An apparatus for encoding an audio signal 99 is shown. The audio signal 99 is input to a time-spectral converter 100 for converting an audio signal having a sampling rate into a spectral representation 101 output by the time-spectral converter. The spectrum 101 is input to a spectrum analyzer 102 for analyzing the spectral representation 101. The spectrum analyzer 101 is configured to determine a first group of first spectral portions 103 to be encoded at a first spectral resolution and a different second group of second spectral portions 105 to be encoded at a second spectral resolution. The second spectral resolution is smaller than the first spectral resolution. The second group of second spectral portions 105 is input to a parameter calculator or parametric encoder 104 for calculating spectral envelope information having a second spectral resolution. In addition, a spectral domain audio encoder 106 is provided for generating a first encoded representation 107 of the first group of first spectral portions having the first spectral resolution. In addition, the parameter calculator / parametric encoder 104 is configured for generating a second encoded representation 109 of the second group of second spectral portions. The first coded representation 107 and the second coded representation 109 are input to a bitstream multiplexer or bitstream former 108 and the block 108 finally outputs the coded audio signal for transmission or storage on a storage device.
[0081] Typically, the first spectrum portion is Figure 3A 306 will be surrounded by two second spectral portions such as 307A and 307B. This is not the case in HEAAC where the core encoder frequency range is band limited.
[0082] Figure 1B Shown with Figure 1A The first encoded representation 107 is input to a spectral domain audio decoder 112 for generating a first decoded representation of a first set of first spectral portions, the decoded representation having a first spectral resolution. Furthermore, the second encoded representation 109 is input to a parametric decoder 114 for generating a second decoded representation of a second set of second spectral portions having a second spectral resolution lower than the first spectral resolution.
[0083] The decoder also includes a frequency regenerator 116 for regenerating a reconstructed second spectral portion with a first spectral resolution using the first spectral portion. The frequency regenerator 116 performs a tile filling operation, i.e., uses a tile or a portion of the first set of first spectral portions and copies the first set of first spectral portions into a reconstruction range or reconstruction band with the second spectral portion, and typically performs spectral envelope shaping or other operations indicated by the second decoded representation output by the parameterized decoder 114, i.e., by using information about the second set of second spectral portions. As shown at the output of the frequency regenerator 116 on line 117, the decoded first set of first spectral portions and the reconstructed second set of spectral portions are input to a spectrum-time converter 118, which is configured to convert the first decoded representation and the reconstructed second spectral portion into a time representation 119, which has a certain high sampling rate.
[0084] Figure 2B Shows Figure 1A The audio input signal 99 is input to the Figure 1A The analysis filter bank 220 corresponding to the time-to-spectrum converter 100 is then subjected to a temporal noise shaping operation in the TNS block 222. Figure 2B The tone masking block 226 corresponds to Figure 1A The input of the spectrum analyzer 102 of may be the full spectrum value when the temporal noise shaping / temporal tiling shaping operation is not applied, and may be the spectrum residual value when the TNS operation of block 222 as shown in FIG2B is applied. For a two-channel signal or a multi-channel signal, a joint channel coding 228 may be additionally performed to make Figure 1A The spectral domain encoder 106 may include a joint channel encoding block 228. In addition, an entropy encoder 232 is provided for performing lossless data compression, which is also Figure 1A Part of the spectral domain encoder 106.
[0085] The spectrum analyzer / tone masker 226 separates the output of the TNS block 222 into a core frequency band and a tonal component corresponding to the first set of first spectrum portions 103 and a Figure 1A The block 224 denoted as IGF parameter extraction coding corresponds to the residual component of the second set of second spectral parts 105. Figure 1A The parameterized encoder 104 and the bitstream multiplexer 230 correspond to Figure 1A bit stream multiplexer 108.
[0086] Preferably, the analysis filter bank 222 is implemented as an MDCT (Modified Discrete Cosine Transform filter bank) and is used to transform the signal 99 into the time-frequency domain using a Modified Discrete Cosine Transform used as a frequency analysis tool.
[0087] Preferably, the spectrum analyzer 226 applies pitch masking. The pitch masking estimation segment is used to separate the pitch components in the signal from the noise-like components. This enables the core encoder 228 to encode all pitch components using the psychoacoustic module. The pitch masking estimation segment can be implemented in a number of different ways and is preferably implemented as a sinusoidal trajectory estimation segment [8, 9] similar in functionality to the sinusoidal modeling of speech / audio coding used in speech / audio coding or the HILN model-based audio encoder described in
[10] . Preferably, an implementation that is easy to implement without having to maintain birth and death trajectories is used, but any other pitch or noise detector may also be used.
[0088] The IGF module calculates the similarity between the source region and the target region. The target region will be represented by the spectrum from the source region. The cross-correlation method is used to measure the similarity between the source region and the target region. The target region is divided into nTar non-overlapping frequency tiles. For each tile in the target region, nSrc source tiles are created from a fixed starting frequency. These source tiles overlap with a factor between 0 and 1, where 0 represents 0% overlap and 1 represents 100% overlap. Each of these source tiles is related to the target tile at various lags to find the source tile that best matches the target tile. The best matching tile number is stored in tileNum[idx_tar], the lag most relevant to the target at it is stored in xcorr_lag[idx_tar][idx_src], and the sign of the correlation is stored in xcorr_sign[idx_tar][idx_src]. In the case of high negative correlation, the source tile needs to be multiplied by -1 before the tile filling process is performed at the decoder. Since pitch masking is used to preserve the tonal components, the IGF module also takes care not to overwrite the tonal components in the spectrum. The band energy parameters are used to store the energy of the target region, which enables accurate reconstruction of the spectrum.
[0089] This method has some advantages over classical SBR [1] in that the core encoder preserves the harmonic grid of the multi-tone signal, while only the gaps between the sinusoids are filled with the best matching "shaped noise" from the source area. Another advantage of this system over ASR (Exact Spectral Replacement) [2-4] is the absence of signal synthesis segments that create important parts of the signal at the decoder. Instead, this task is taken over by the core encoder, allowing the important components of the spectrum to be preserved. Another advantage of the proposed system is the continuous scalability provided by the features. Using only tileNum[idx_tar] and xcorr_lag=0 for each tile is called coarse-grained matching and can be used for low bit rates when the variable xcorr_lag is used for each tile, allowing a better match between the target spectrum and the source spectrum.
[0090] Additionally, a tile selection stabilization technique is proposed, which removes frequency domain artifacts such as warble noise or musical noise.
[0091] In the case of stereo channel pairs, additional joint stereo processing is applied. This is necessary because for a certain target range, the signal can come from a highly correlated panning sound source. In the case where the source area selected for this specific area is not well correlated, although the energy is matched for the target area, the spatial image may be impaired due to the irrelevant source area. The encoder analyzes each target area energy band, usually performs the cross-correlation of the spectral values, and if a certain threshold is exceeded, a joint flag is set for the energy band. In the decoder, if the joint stereo flag is not set, the left channel energy band and the right channel energy band are processed separately. In the case of setting the joint stereo flag, both energy and patching are performed in the joint stereo domain. The joint stereo information of the IGF area is sent with a signal similar to the joint stereo information of the core coding, including a flag indicating whether the direction of the prediction is from downmix to residual or from residual to downmix in the case of prediction.
[0092] The energy can be calculated based on the transmitted energy in the L / R domain.
[0093] midNrg[k]=leftNrg[k]+rightNrg[k];
[0094] sideNrg[k]=leftNrg[k]-rightNrg[k];
[0095] where k is the frequency index in the transform domain.
[0096] Another solution is to directly compute and send the energy in the joint-stereo domain for the frequency bands where joint-stereo is active, so no additional energy transformation is needed on the decoder side.
[0097] Source tiles are always created from a mid / side matrix:
[0098] midTile[k]=0.5·(leftTile[k]+rightTile[k])
[0099] sideTile[k]=0.5·(leftTile[k]-rightTile[k])
[0100] Energy Adjustment:
[0101] midTile[k]=midTile[k]*midNrg[k];
[0102] sideTile[k]=sideTile[k]*sideNrg[k];
[0103] Joint Stereo->LR Transform:
[0104] If no additional parameters are encoded:
[0105] leftTile[k]=midTile[k]+sideTile[k]
[0106] rightTile[k]=midTile[k]-sideTile[k]
[0107] If additional prediction parameters are encoded and if the signaled direction is from center to side:
[0108] sideTile[k]=sideTile[k]-predictionCoeff·midTile[k]
[0109] leftTile[k]=midTile[k]+sideTile[k]
[0110] rightTile[k]=midTile[k]-sideTile[k]
[0111] If the signaling direction is from side to center:
[0112] midTile1[k]=midTile[k]-predictionCoeff·sideTile[k]
[0113] leftTile[k]=midTile1[k]-sideTile[k]
[0114] rightTile[k]=midTile1[k]+sideTile[k]
[0115] This process ensures that even if the source regions are uncorrelated, the resulting left and right channels still represent correlated and panned sound sources, preserving the stereo image of such regions, based on the tiles used to regenerate highly correlated destination regions and panned destination regions.
[0116] In other words, in the bitstream, a joint stereo flag is transmitted indicating whether L / R or M / S should be used as an example of general joint stereo encoding. In the decoder, first, the core signal is decoded as indicated by the joint stereo flag of the core band. Second, the core signal is stored in both the L / R representation and the M / S representation. For IGF tile filling, the source tile representation is selected to match the target tile representation as indicated by the joint stereo information of the IGF band.
[0117] Temporal noise shaping (TNS) is a standard technique and is part of ACC [11-13]. TNS can be considered as an extension of the basic scheme of the perceptual encoder, with an optional processing step inserted between the filter bank and the quantization stage. The main task of the TNS module is to hide the generated quantization noise in the temporal masked region of the transient-like signal, resulting in a more efficient coding scheme. First, TNS calculates a set of prediction coefficients using "forward prediction" in the transform domain, such as MDCT. These coefficients are then used to flatten the temporal envelope of the signal. As quantization affects the TNS-filtered spectrum, the quantization noise is also flattened in time. By applying inverse TNS filtering on the decoder side, the quantization noise is shaped according to the temporal envelope of the TNS filter, so the quantization noise is temporarily masked.
[0118] IGF is based on MDCT representation. For efficient coding, preferably, long blocks of about 20ms must be used. If the signal within such a long block includes transient signals, audible pre-echoes and post-echoes appear in the IGF spectral band due to tile filling. Figure 7C A typical pre-echo effect before the onset of a transient due to IGF is shown. On the left, the spectrogram of the original signal is shown, on the right, the spectrogram of the bandwidth extended signal without TNS filtering is shown.
[0119] In the context of IGF, this pre-echo effect is reduced by using TNS. Here, TNS is used as a Time Tile Shaping (TTS) tool when spectral regeneration is performed on the TNS residual signal in the decoder. The required TTS prediction coefficients are calculated and applied using the full spectrum on the encoder side as usual. The TNS / TTS start and end frequencies are not affected by the IGF start frequency f of the IGF tool. IGFstartCompared with the traditional TNS, the TTS termination frequency is increased to the termination frequency of the IGF tool, which is higher than f IGFstart At the decoder side, the full spectrum, ie the core spectrum plus the regenerated spectrum, is again added with the pitch component from the pitch map (see Fig. 7E ) Apply TNS / TTS coefficients. The application of TTS is necessary for forming the temporal envelope of the regenerated spectrum to match the envelope of the original signal again. Therefore, the pre-echo shown is reduced. In addition, TNS is still used as usual for the signal below f IGFstart The quantization noise is shaped.
[0120] In conventional decoders, spectral patching of an audio signal destroys spectral correlations at patch boundaries and thus impairs the temporal envelope of the audio signal by introducing dispersion. Therefore, another benefit of performing IGF tile filling on the residual signal is that after applying the shaping filter, the tile boundaries are seamlessly correlated, resulting in a more realistic temporal reproduction of the signal.
[0121] In the encoder of the present invention, the spectrum that has undergone TNS / TTS filtering, tone masking processing and IGF parameter estimation lacks any signal above the IGF starting frequency except the tone component. This sparse spectrum is now encoded by the core encoder using the principles of arithmetic coding and predictive coding. These encoded components form the bit stream of audio together with the signaling bits.
[0122] Figure 2A The corresponding decoder implementation is shown. Figure 2A The bit stream corresponding to the encoded audio signal in is input to the demultiplexer / decoder 200, which is relatively Figure 1B will be connected to blocks 112 and 114. The bitstream demultiplexer separates the incoming audio signal into Figure 1B The first encoding represents 107 and Figure 1B The first coded representation of the first set of first spectral portions is input to the Figure 1B The second coded representation is input to the joint channel decoding block 204 corresponding to the spectral domain decoder 112 of Figure 2A The parameterized decoder 114 shown in FIG. 1 is then input to the Figure 1BThe IGF block 202 corresponding to the frequency regenerator 116 of the frequency regenerator 116. The first set of first spectrum parts required for frequency regeneration is input to the IGF block 202 via line 203. In addition, after the joint channel decoding 204, a specific core decoding is applied in the tone masking block 206 so that the output of the tone masking 206 corresponds to the output of the spectrum domain decoder 112. Then, the combination is performed by the combiner 208, that is, the frame construction in which the output of the combiner 208 now has a full range spectrum but is still in the TNS / TTS filter domain. Then, in block 210, the inverse TNS / TTS operation is performed using the TNS / TTS filter information provided via line 109, that is, the TTS side information is preferably included in the first encoded representation generated by the spectrum domain encoder 106, or can be included in the second encoded representation, the spectrum domain encoder 106 can be, for example, a simple AAC or USAC core encoder. At the output of block 210, the complete spectrum up to the maximum frequency is provided, which is the full range of frequencies defined by the sampling rate of the original input signal. Then, a spectrum / time conversion is performed in the synthesis filter bank 212 to finally obtain an audio output signal.
[0123] Figure 3A A schematic representation of the spectrum is shown. The spectrum is subdivided into scale factor bands SCB, where Figure 3A In the example shown, there are seven scale factor bands SCB1 to SCB7. Figure 3A As schematically shown in , the scale factor bands may be AAC scale factor bands defined in the AAC standard and have an increasing bandwidth to the upper frequency. It is preferred not to perform smart gap filling from the beginning of the spectrum, i.e. at low frequencies, but to start the IGF operation at the IGF start frequency shown at 309. Thus, the core band extends from the lowest frequency to the IGF start frequency. Above the IGF start frequency, spectral analysis is applied to separate the high resolution spectral components 304, 305, 306, 307 (the first group of first spectral portions) from the low resolution components represented by the second group of second spectral portions. Figure 3A The spectrum that is exemplarily input to the spectral domain encoder 106 or the joint channel encoder 228 is shown, i.e., the core encoder operates at the full range, but encodes a large number of zero spectral values, i.e., these zero spectral values are quantized to zero or set to zero before or after quantization. In any case, the core decoder operates at the full range, i.e., as the spectrum would be as shown, i.e., the core decoder does not have to be aware of any intelligent gap filling or encoding of the second set of second spectral portions with lower spectral resolution.
[0124] Preferably, the high resolution is defined by a linear coding of spectral lines, such as MDCT lines, while the second or low resolution is defined by, for example, calculating only a single spectral value per scale factor band, wherein the scale factor band covers several frequency lines. Thus, the second low resolution is much lower with respect to its spectral resolution than the first or high resolution, which is defined by a linear coding typically applied by a core encoder, such as an AAC or USAC core encoder.
[0125] Regarding the scaling factor or energy calculation, Figure 3B Due to the fact that the encoder is a core encoder and due to the fact that in each frequency band there can be, but not necessarily, a first set of spectrum portions, the core encoder operates not only in the core range below the IGF start frequency 309 but also above the IGF start frequency up to a maximum frequency f IGFstop Calculate the scaling factor for each frequency band within the range of IGFstop Less than or equal to half of the sampling frequency, i.e. f s / 2 Thus, the coded tone parts 302, 304, 305, 306, 307 of Fig. 3a and in this embodiment together with the scale factors SCB1 to SCB7 correspond to high resolution spectral data. The low resolution spectral data are calculated starting from the IGF start frequency and correspond to energy information values E1, E2, E3, E4, which are transmitted together with the scale factors SF4 to SF7.
[0126] In particular, when the core encoder is in low bit rate conditions, an additional noise filling operation can be additionally applied in the core band, i.e., in the scale factor bands SCB1 to SCB3, i.e., at frequencies below the IGF start frequency. In noise filling, there are several adjacent spectral lines that are quantized to zero. At the decoder side, these spectral values quantized to zero are resynthesized, and the magnitude of the resynthesized spectral values is calculated using the noise filling energy as Figure 3B 308 in the NF2 is adjusted. The noise filling energy, which can be given in absolute or relative values, corresponds to the energy of the set of spectral values quantized to zero, especially with respect to the scaling factors in the USAC. These noise filled spectral lines can also be regarded as a third set of third spectral portions, which are regenerated by a simple noise filling synthesis without any IGF operation relying on frequency regeneration using frequency tiles from other frequencies, for reconstructing the frequency tiles using spectral values from the source range and the energy information E1, E2, E3, E4.
[0127] Preferably, the frequency band for calculating the energy information coincides with the scale factor frequency band. In other embodiments, the energy information value grouping is applied so that, for example, for scale factor frequency bands 4 and 5 ( Figure 3B312 in ) only sends a single energy information value, but even in this embodiment, the boundaries of the grouped reconstruction bands coincide with the boundaries of the scale factor bands. If a different band separation is applied, some recalculation or synthesis can be applied, and this may be reasonable depending on a certain implementation.
[0128] Preferably, Figure 1A The spectral domain encoder 106 is as follows Figure 4A Typically, as shown in the MPEG2 / 4AAC standard or the MPEG1 / 2, layer 3 standard, the spectral range ( Figure 4A 401 in the MPEG-4 AAC standard) after which the encoded audio signal is forwarded to a scale factor calculator 400. The scale factor calculator is controlled by a psychoacoustic model 402, which additionally receives the audio signal to be quantized or, as in MPEG1 / 2 layer 3 or in accordance with the MPEGAAC standard, receives a complex spectral representation of the audio signal. The psychoacoustic model 402 calculates a scale factor for representing a psychoacoustic threshold for each scale factor band. In addition, the scale factor is then adjusted by the cooperation of known inner and outer iterative loops or by any other suitable encoding process so that certain bit rate conditions are achieved. The spectral values to be quantized on the one hand and the calculated scale factors on the other hand are then input to a quantizer processor 404. In a simple audio encoder operation, the spectral values to be quantized are weighted by the scale factors, and the weighted spectral values are then input to a fixed quantizer that usually has a compression function of an upper amplitude range. Then, at the output of the quantizer processor, there are indeed quantization indices, which are then forwarded to an entropy encoder, which typically has a specific and very efficient encoding for a set of zero quantization indices for adjacent frequency values, or “runs” of zero values, also referred to in the art.
[0129] However, in Figure 1A In an audio encoder of the type quantizer processor 404, the quantizer processor typically receives information about the second spectrum portion from the spectrum analyzer. Thus, the quantizer processor 404 ensures that: in the output of the quantizer processor 404, the second spectrum portion as identified by the spectrum analyzer 102 is zero or has a representation recognized by the encoder or decoder as a zero representation, and the zero representation can be encoded very efficiently, especially when there are "strings" of zero values in the spectrum.
[0130] Figure 4BThe implementation of the quantizer processor is shown. The MDCT spectrum value can be input to the set to zero block 410. Then, before weighting is performed by the scale factor in block 412, the second spectrum part has been set to zero. In another implementation, block 410 is not provided, but the set to zero collaboration is performed in block 418 after the weighting block 412. In another implementation, the set to zero operation can also be performed in the set to zero block 422 after the quantization in the quantizer block 420. In this implementation, blocks 410 and 418 will not appear. Generally, at least one of blocks 410, 418, 422 is provided according to the specific implementation.
[0131] Then, at the output of block 422, the Figure 3A Then, the quantized spectrum is input to an entropy encoder such as Figure 2B 232 in which may be a Huffman encoder or an arithmetic encoder as defined in, for example, the USAC standard.
[0132] The zero blocks 410, 418, 422, which are arranged alternately or in parallel with each other, are controlled by a spectrum analyzer 424. Preferably, the spectrum analyzer comprises any implementation of a known tone detector or comprises any different type of detector operatively used to separate the spectrum into components to be encoded at a high resolution and components to be encoded at a low resolution. Other such algorithms implemented in the spectrum analyzer may be a voice activity detector, a noise detector, a speech detector or any other detector that makes decisions based on spectral information of different spectral parts or associated metadata about the resolution requirements.
[0133] Figure 5A Shows Figure 1AThe preferred implementation of the time spectrum converter 100 of time spectrum converter 100, for example, realizes according to AAC or USAC. Time spectrum converter 100 comprises a window adder 502 controlled by a transient detector 504. When the transient detector 504 detects transient state, the switching from long window to short window is sent to the window adder with a signal. Then, the window adder 502 is for the frame of overlapping block calculation windowing, and wherein the frame of each windowing has 2N value such as 2048 values usually. Then, in block converter 506, conversion is performed, and this block converter usually provides extraction additionally to make the extraction / conversion of execution combination to obtain the spectrum frame with N value such as MDCT spectrum value. Thereby, for long window operation, the frame at the input end of block 506 comprises 2N value such as 2048 values and spectrum frame then has 1024 values. Then, however, when 8 short blocks are performed, and each short block has 1 / 8 of the windowed time domain value compared to the long window and each spectral block has 1 / 8 of the spectral value compared to the long block, switching is performed on the short blocks. Thus, when this decimation is combined with the 50% overlap operation of the window adder, the spectrum is a critically sampled version of the time domain audio signal 99.
[0134] Then, refer to Figure 5B , Figure 5B Shows Figure 1B The frequency regenerator 116 and the spectrum time converter 118 or Figure 2A The specific implementation of the combined operation of blocks 208 and 212. Figure 5B In the example, we consider a specific reconstruction band such as Figure 3A The first spectral portion in the reconstructed band, i.e., the first spectral portion 306 of FIG. 3a, is input to the frame builder / adjuster block 510. In addition, the reconstructed second spectral portion of the scale factor band 6 is also input to the frame builder / adjuster 510. In addition, the energy information of the scale factor band 6 is as follows: Figure 3B E3 is also input to block 510. The reconstructed second spectral portion in the reconstructed frequency band has been generated by frequency tiling using the source range, then the reconstructed frequency band corresponds to the target range. Now, energy adjustment of the frame is performed so as to finally obtain a complete reconstructed frame with N values as in Figure 2AThe MDCT is obtained at the output of the combiner 208. Then, in block 512, an inverse block transform / interpolation is performed to obtain 248 time domain values of, for example, 124 spectral values at the input of block 512. Then, a synthetic windowing operation is performed in block 514, which is again controlled by the long window / short window indication sent as side information in the encoded audio signal. Then, in block 516, an overlap / addition operation with the previous time frame is performed. Preferably, the MDCT applies a 50% overlap so that for each new time frame of 2N values, N time domain values are finally output. A 50% overlap is strongly preferred due to the fact that it provides critical sampling and provides continuous crossing from one frame to the next due to the overlap / addition operation in block 516.
[0135] like Figure 3A As shown in 301, for example, for Figure 3A For the envisaged reconstructed frequency bands overlapping with the scale factor band 6, a noise filling operation may be additionally applied not only below the IGF start frequency but also above the IGF start frequency, and then the noise filled spectrum values may also be input to the frame builder / adjuster 510, and adjustment of the noise filled spectrum values may also be applied in this block, or the noise filling energy may be used to adjust the noise filled spectrum values before being input to the frame builder / adjuster 510.
[0136] Preferably, the IGF operation, i.e., the spectrum tiling operation using spectrum values from other parts, can be applied to the complete spectrum. Thus, the spectrum tiling operation can be applied not only in the high frequency band above the IGF start frequency, but also in the low frequency band. In addition, noise filling without spectrum tiling can also be applied not only below the IGF start frequency, but also above the IGF start frequency. However, it has been found that if Figure 3A As shown in , high quality and efficient audio coding can be achieved when the noise filling operation is limited to the spectrum range below the IGF start frequency and when the frequency tile filling operation is limited to the spectrum range above the IGF start frequency.
[0137] Preferably, the target tile (TT) (having a frequency greater than the IGF start frequency) is subject to the scale factor band boundaries of the full rate encoder. The source tile (ST) from which information is obtained, that is, the source tile with a frequency lower than the IGF start frequency is not subject to the scale factor band boundaries. The size of the ST should correspond to the size of the associated TT. This is illustrated using the following example. TT[0] has a length of 10 MDCT elements. This corresponds exactly to the length of two subsequent SCBs (such as 4+6). Then, all possible STs to be associated with TT[0] also have a length of 10 elements. The second target tile TT[1] adjacent to TT[0] has a length of 15 elements I (SCB with a length of 7+8). Then, its ST has a length of 15 elements instead of the length of 10 elements of TT[0].
[0138] If the situation arises that no TT can be found for an ST with the length of the target tile (when, for example, the length of the TT is larger than the available source range), the correlation is not calculated and the source range is copied multiple times into the TT (one after another so that the frequency line of the lowest frequency of the second copy closely follows in frequency the frequency line of the highest frequency of the first copy) until the target tile TT is completely filled.
[0139] Then, refer to Figure 5C , Figure 5C Shows Figure 1B The frequency regenerator 116 or Figure 2A Block 522 is a frequency tile generator that receives not only the target band ID but also the source band ID. Exemplarily, it is determined on the encoder side: Figure 3A The scale factor of band 3 ( Figure 3B 310 in ) is very well suited for reconstructing scale factor band 7. Thus, the source band ID will be 2 and the target band ID will be 7. Based on this information, the frequency tile regenerator 522 applies a replication or harmonic tile filling operation or any other tile filling operation to generate an original second portion of spectral components 523. The original second portion of spectral components has the same spectral resolution as the frequency resolution included in the first set of first spectral portions.
[0140] Then, the reconstructed frequency band is Figure 3AThe first spectral portion of 307 of is input to the frame builder 524 and the original second portion 523 is also input to the frame builder 524. The adjuster 526 then adjusts the reconstructed frame using the gain factor of the reconstructed band calculated by the gain factor calculator 528. However, importantly, the first spectral portion in the frame is not affected by the adjuster 526, but only the original second portion of the reconstructed frame is affected by the adjuster 526. To this end, the gain factor calculator 528 analyzes the source band or the original second portion 523 and additionally analyzes the first spectral portion in the reconstructed band to finally find the correct gain factor 527 so that the energy of the adjusted frame output by the adjuster 526 has an energy E4 when the scale factor band 7 is assumed.
[0141] In this context, it is very important to evaluate the high frequency reconstruction accuracy of the present invention compared to HE-AAC. Figure 3A 13A . Assume that a prior art encoder, such as the encoder shown in FIG. 13A , detects a portion of the spectrum 307 to be encoded at high resolution as a "missing harmonic". The energy of this spectral component is then sent to the decoder together with the spectral envelope information about the reconstruction band, such as scale factor band 7. The decoder will then reconstruct the missing harmonics. However, the spectral values at which the missing harmonics 307 are reconstructed by the prior art decoder of FIG. 13B will be located in the middle of band 7 at the frequency represented by the reconstruction frequency 390. Thus, the present invention avoids the frequency error 391 that would be introduced by the prior art decoder of FIG. 13B .
[0142] In one implementation, the spectrum analyzer is further implemented to calculate the similarity between the first spectrum portion and the second spectrum portion and determine, for the second spectrum portion within the reconstruction range, the first spectrum portion that matches the second spectrum portion as closely as possible based on the calculated similarity. Then, in this variable source range / destination range implementation, the parameterized encoder additionally introduces matching information indicating the matching source range for each destination range into the second encoded representation. Then, at the decoder side, this information is provided by Figure 5C The spectrum tile generator 522 uses, Figure 5C It is shown that the original second part 523 is generated based on the source band ID and the target band ID.
[0143] In addition, if Figure 3A As shown in , the spectrum analyzer is configured to analyze the spectral representation up to a maximum analysis frequency, which is only slightly lower than half the sampling frequency and preferably at least one quarter of the sampling frequency or typically higher.
[0144] As shown, the encoder operates without downsampling and the decoder operates without upsampling.In other words, the spectral domain audio encoder is configured to generate a spectral representation having a Nyquist frequency defined by the sampling rate of the original input audio signal.
[0145] In addition, if Figure 3A As shown in , the spectrum analyzer is configured to analyze a spectrum representation starting from a gap filling start frequency and ending at a maximum frequency represented by a maximum frequency included in the spectrum representation, wherein a spectrum portion extending from the minimum frequency upward to the gap filling start frequency belongs to a first group of spectrum portions, and wherein additional spectrum portions such as 304, 305, 306, 307 having frequency values higher than the gap filling frequency are also included in the first group of first spectrum portions.
[0146] As outlined, the spectral domain audio decoder 112 is configured such that the maximum frequency represented by the spectral value in the first decoded representation is equal to the maximum frequency included in the time representation having the sampling rate, wherein the spectral value of the maximum frequency in the first set of first spectral portions is zero or non-zero. In any case, for this maximum frequency in the first set of spectral components, there is a scale factor for the scale factor band, regardless of whether as in Figure 3A and Figure 3B The scaling factor is generated and sent regardless of whether all spectral values in the scaling factor band discussed in the context of FIG. 1 are set to zero or not.
[0147] Therefore, the invention is advantageous in that it enables accurate frequency reproduction of tonal components relative to other parameterization techniques for increasing compression efficiency, such as noise substitution and noise filling, which are specifically designed for efficient representation of noise-like local signal content. Currently, there is no prior art that addresses efficient parameterization of arbitrary signal content by spectral gap filling without being restricted to a fixed a priori partition in a low frequency band (LF) and a high frequency band (HF).
[0148] Embodiments of the inventive system improve upon prior art approaches, providing high compression efficiency, no or only a small amount of perceived annoyance, and full audio bandwidth even at low bit rates.
[0149] The general system includes:
[0150] Full-band core coding
[0151] Intelligent gap filling (tile filling or noise filling)
[0152] Sparse tonal parts in the core selected by pitch masking
[0153] Full-band joint stereo pair encoding, including tile padding
[0154] TNS for paving
[0155] Spectral whitening within the IGF range
[0156] The first step towards a more efficient system is to remove the need to transform the spectral data into a second transform domain that is different from the transform domain of the core encoding. As most audio codecs such as AAC use MDCT as the basic transform, it is useful to also perform BWE in the MDCT domain. The second requirement for a BWE system would be the need to preserve the pitch grid, whereby even HF tonal components are preserved, so that the quality of the encoded audio is better than existing systems. In order to carefully handle the two requirements mentioned above for the BWE scheme, a new system called Intelligent Gap Filling (IGF) is proposed. Figure 2B A block diagram of the proposed system showing the encoder side, Figure 2A The system on the decoder side is shown.
[0157] Fig. 6A A device for decoding an encoded audio signal in another implementation of the present invention is shown. The device for decoding includes: a spectral domain audio decoder 602, which is used to generate a first decoded representation of a first group of spectral parts; and a frequency regenerator 604, which is connected downstream of the spectral domain audio decoder 602, and is used to use the first spectral part in the first group of first spectral parts to generate a reconstructed second spectral part. As shown at 603, the spectral values in the first spectral part and the second spectral part are spectral prediction residual values. In order to transform these spectral prediction residual values into a full spectrum representation, a spectral prediction filter 606 is provided. The inverse prediction filter is configured to perform inverse prediction about frequency using the spectral residual values of the first group of first frequencies and the reconstructed second spectral part. The spectral inverse prediction filter 606 is configured by filter information included in the encoded audio signal. Figure 6B Shows Fig. 6A The spectral prediction residual values 603 are input to a frequency tile regenerator 612 which generates the original spectral values about the reconstructed frequency band or some second frequency portion, whereby this original data, now having the same resolution as the high resolution of the first spectral representation, is input to a spectral shaper 614. The spectral shaper now shapes the spectrum using the envelope information transmitted in the bitstream, and the spectral shaped data is then applied to a spectral prediction filter 616 which uses the filter information 607 sent from the encoder to the decoder via the bitstream to finally generate a frame of full spectral values.
[0158] exist Figure 6BIt is assumed in : at the encoder side, the calculation of the filter information sent via the bitstream and used via line 607 is performed after the envelope information is calculated. So, in other words, Figure 6B The decoder matching the encoder will first calculate the spectral residual value, and then use the spectral residual value to calculate the envelope information, such as Fig. 7A However, for some implementations, other implementations are also useful, where at the encoder side (such as Figure 6C Before performing TNS or TTS filtering (as shown in FIG. 6 ) with a frequency tile generator 620, envelope information is calculated. Then, before performing spectrum shaping in block 624, a spectrum prediction filter 622 is applied. Thus, in other words, before applying the spectrum shaping operation 624, (full) spectrum values are generated.
[0159] Preferably, a complex-valued TNS filter or TTS filter is calculated. Fig. 7A The original audio signal is input to the complex MDCT block 702. Then, TTS filter calculation and TTS filtering are performed in the complex domain. Then, in block 706, IGF side information 712 is calculated, and any other operations such as spectral analysis for encoding are also calculated. Then, the first set of first spectral parts generated by block 706 is encoded by the psychoacoustic model driven encoder shown at 708 to obtain Fig. 7A The first group of first spectrum parts represented at X(k) in , and then all these data are forwarded to the bitstream multiplexer 710.
[0160] At the decoder side, the encoded data is input to a demultiplexer 720 to separate the IGF side information 712 on the one hand and the TTS side information and the encoded representation of the first group of first spectral portions on the other hand.
[0161] Then, block 724 is used for calculating complex spectrum according to one or more real value spectrums. Then, both real value spectrum and complex spectrum are input to block 726 to generate the reconstructed frequency value in the second group of second spectrum parts for the reconstruction frequency band. Then, the full-band frame completely obtained and ply-filled is performed inverse TTS operation 728, and at the decoder side, in block 730, final inverse complex MDCT operation is performed. Thereby, the use of complex TNS filter information makes it possible to automatically generate ply boundary processing when not only being applied in core band or independent ply frequency band but also being applied on core / ply boundary or ply / ply boundary, and finally it reintroduces the spectrum correlation between ply. This spectrum correlation on the ply boundary can not only be obtained by generating frequency ply and performing spectrum envelope adjustment to the raw data of frequency ply.
[0162] Figure 7CA comparison of the original signal (left panel) and the extended signal without TTS is shown. It can be seen that there are strong artifacts shown by the expanded portion in the upper frequency range shown at 750. However, when the same portion of the spectrum at 750 is compared to Figure 7C The artifact-related components 750 are compared to Fig. 7E This does not happen in .
[0163] An embodiment or audio coding system of the present invention uses a major share of the available bit rate to waveform encode only the most perceptually relevant structures of the signal in the encoder, and the resulting spectral gaps are filled in the decoder with a signal content that roughly approximates the original spectrum. A very limited bit budget is consumed to control parameter-driven so-called spectrally intelligent gap filling (IGF) through dedicated side information sent from the encoder to the decoder.
[0164] The storage or transmission of audio signals is usually subject to strict bit rate constraints. In the past, encoders were forced to drastically reduce the transmission audio bandwidth only when very low bit rates were available. Modern audio codecs are now able to encode wideband signals by using bandwidth extension (BWE) methods similar to spectral bandwidth replication (SBR) [1]. These algorithms rely on a parameterized representation of the high frequency content (HF), which is generated from the waveform of the decoded signal by encoding the low frequency part (LF) by means of a shift ("patching") to the HF spectral region, and the application of parameter-driven post-processing. In a BWE scheme, the reconstruction of the HF frequency region above a given so-called crossover frequency is usually based on spectral patching. Typically, the HF region consists of a plurality of adjacent patches, and each of these patches originates from a bandpass (BP) region of the LF spectrum below a given crossover frequency. Prior art systems effectively perform patching within a filter bank representation by copying adjacent subband coefficient groups from a source region to a target region.
[0165] If the BWE system is implemented in the filter bank or time-frequency transform domain, there is only a limited possibility to control the temporal shape of the bandwidth extended signal. Usually, the temporal granularity is limited by the hop distance used between adjacent transform windows. This can lead to unwanted pre-echoes and post-echoes within the BWE spectrum.
[0166] From perceptual audio coding, it is known that the shape of the temporal envelope of an audio signal can be restored by using spectral filtering techniques such as temporal envelope shaping (TNS)
[14] . However, TNS filters known from the prior art are real-valued filters on a real-valued spectrum. Such real-valued filters on a real-valued spectrum can be severely affected by aliasing artifacts, especially when the underlying real transform is a modified discrete cosine transform (MDCT).
[0167] Temporal envelope tiling applies a complex filter to the complex-valued spectrum, such as obtained by, for example, a complex modified discrete cosine transform (CMDCT). Thus, aliasing artifacts are avoided.
[0168] Time paving shaping includes:
[0169] Complex filter coefficient estimation and application of a flattening filter to the original signal spectrum at the encoder
[0170] Transmission of filter coefficients in side information
[0171] Apply a shaping filter to the tile-filled reconstructed spectrum at the decoder
[0172] The present invention extends the prior art known from audio transform coding, in particular temporal noise shaping (TNS), by linear prediction in the frequency direction to be used in a modified manner in the context of bandwidth extension.
[0173] In addition, the bandwidth extension algorithm of the present invention is based on intelligent gap filling (IGF), but in contrast to the IGF standard configuration that relies on a real-valued critically sampled MDCT representation of the signal, an oversampled complex-valued transform (CMDCT) is utilized. The CMDCT can be viewed as a combination of the MDCT coefficients of the real part of each complex-valued spectral coefficient and the MDST coefficients of the imaginary part.
[0174] Although the new method is described in the context of IGF, the process of the present invention can be used in conjunction with any BWE method based on a filter bank representation of the audio signal.
[0175] In this novel context, linear prediction along the frequency direction is not used as temporal noise shaping, but as a temporal tile shaping (TTS) technique. This renaming is justified by the fact that, in contrast to quantization noise shaping by TNS in prior art perceptual transform codecs, the tile-filled signal component is temporally shaped by TTS.
[0176] Fig. 7A A block diagram of a BWE encoder using IGF and the new TTS method is shown.
[0177] So the basic encoding scheme works as follows:
[0178] - Calculate the CMDCT of the time domain signal x(n) to obtain the frequency domain signal X(k)
[0179] - Compute complex-valued TTS filters
[0180] - Obtain side information for BWE and remove spectral information that must be copied by the decoder
[0181] -Apply quantization using the Psychoacoustic Module (PAM)
[0182] -Store / send data, only real-valued MDCT coefficients are sent
[0183] Figure 7B The corresponding decoder is shown. It essentially reverses the steps performed in the encoder.
[0184] Here, the basic decoding scheme works as follows:
[0185] - Estimate the MDST coefficients from the MDCT values (this process adds one block decoder delay), and combine the MDCT coefficients and the MDST coefficients into complex-valued CMDCT coefficients
[0186] - Perform tile filling using its post-processing
[0187] - Apply inverse TTS filtering using the transmitted TTS filter coefficients
[0188] - Calculate the inverse CMDCT
[0189] Note that, alternatively, if TTS analysis and IDG parameter estimation are consistently reversed at the encoder, the order of TTS synthesis and IGF post-processing may also be reversed in the decoder.
[0190] For efficient transform coding, preferably so-called "long blocks" of approximately 20 ms have to be used to achieve reasonable transform gain. If the signal within such a long block includes transients, audible pre- and post-echoes appear in the reconstructed spectral band due to tile filling. Figure 7C The typical pre-echo and post-echo effects due to IGF damage transients are shown. Figure 7C On the left panel of FIG. 1 , the spectrum of the original signal is shown, while on the right panel, the spectrum of the tile-filled signal without the TTS filtering of the present invention is shown. In this example, the IGF starting frequency f between the core band and the tile-filled band is IGFstart or Split Selected as f s / 4. Figure 7C In the right panel, distinct pre- and post-echoes are visible around the transient, especially at the upper end of the spectrum in the copied frequency region.
[0191] The main task of the TTS module is to limit these unwanted signal components near transients, thereby hiding them in the temporal region dominated by the temporal masking effects of human perception. Therefore, the required TTS prediction coefficients are calculated and applied in the CMDCT domain using "forward prediction".
[0192] In embodiments where TTS and IGF are combined in a codec, it is important to align certain TTS parameters and IGF parameters so that the IGF tile is either completely filtered or not filtered by a TTS filter (flat or shaping filter). Thus, all TTSstart[..] or TTSstop[..] frequencies should not be included in the IGF tile, but aligned to the corresponding f IGF... frequency. Fig.7D Examples of TTS and IGF operating regions for a set of three TTS filters are shown.
[0193] The TTS stop frequency is adjusted to the stop frequency of the IGF tool, which is higher than f IGFstart If more than one filter is used for TTS, it must be ensured that: the crossover frequency between the two TTS filters must match the IGF cutoff frequency. Otherwise, one TTS sub-filter will exceed f IGFstart , leading to unwanted artifacts such as overshaping.
[0194] exist Fig. 7A and Figure 7B In the implementation variant depicted in , additional care must be taken to correctly adjust the energy in the decoder IGF. This is particularly true in the case where different TTS filters with different prediction gains are applied to the source region (e.g. a flattening filter) and the target spectral region (e.g. a shaping filter, which is not an exact copy of the flattening filter) of one IGF tile during the TTS and IGF processing. In this case, the prediction gain ratio of the two applied TTS filters is no longer equal to 1, and therefore an energy adjustment with this ratio must be applied.
[0195] In an alternative implementation variant, the order of IGF post-processing and TTS is reversed. In the decoder, this means that the energy adjustment by IGF post-processing is calculated after TTS filtering and is therefore the final processing step before the synthesis transform. Therefore, regardless of different TTS filter gains applied to a tile during encoding, the final energy is always correctly adjusted by IGF processing.
[0196] At the decoder side, the TTS filter coefficients are again applied to the full spectrum, i.e. the core spectrum extended by the regenerated spectrum. The application of TTS is necessary to form the temporal envelope of the regenerated spectrum to match the envelope of the original signal again. Therefore, the pre-echo shown is reduced. In addition, like conventional TNS, it still has no effect on the signals below f. IGFstart The quantization noise is shaped in time.
[0197] In conventional encoders, spectral patching (e.g., SBR) of an audio signal destroys spectral correlations at patch boundaries and thus impairs the temporal envelope of the audio signal by introducing diffusion. Therefore, another benefit of performing IGF tile filling on the residual signal is that after applying the TTS shaping filter, the tile boundaries are seamlessly correlated, resulting in a more realistic temporal reproduction of the signal.
[0198] exist Fig. 7E The results of the corresponding processed signal are shown in . Compared with the unfiltered version ( Figure 7C , right panel), the TTS-filtered signal shows a good reduction of unwanted pre- and post-echoes ( Fig. 7E , right panel).
[0199] Furthermore, as discussed, Fig. 7A Shown with Figure 7B The decoder or Fig. 6A The decoder matches the encoder. Basically, the device for encoding an audio signal comprises a time-spectral converter such as 702, which is used to convert the audio signal into a spectral representation. The spectral representation can be a real-valued spectral representation, or a complex-valued spectral representation as shown in block 702. In addition, a prediction filter such as 704 for performing a prediction about frequency is arranged to generate a spectral residual value, wherein the prediction filter 704 is defined by the prediction filter information obtained from the audio signal and forwarded to the bitstream multiplexer 710, such as Fig. 7A In addition, an audio encoder such as a psychoacoustically driven audio encoder 704 is also provided. The audio encoder is configured to encode the first set of first spectral portions of the spectral residual values to obtain an encoded first set of first spectral values. In addition, a parameterized encoder such as Fig. 7A The parametric encoder shown at 706 in is arranged for encoding the second set of second spectral portions. Preferably, the first set of first spectral portions is encoded at a higher spectral resolution than the second set of second spectral portions.
[0200] Finally, if Fig. 7A As shown in FIG. 1 , the output interface is configured to output the second set of second spectral portions encoded parametrically, the encoded first set of first spectral portions and the Fig. 7A The encoded signal output of the filter information is shown at 714 as "TTS side information".
[0201] Preferably, the prediction filter 704 comprises a filter information calculator configured to calculate the filter information using spectral values of the spectral representation. Furthermore, the prediction filter is configured to calculate the spectral residual value using the same spectral values used for calculating the spectral representation of the filter information.
[0202] Preferably, the TTS filter 704 is configured in the same way as in known prior art audio encoders applying the TTS tool according to the AAC standard.
[0203] Later, in FIG. 8A to FIG. 8E Another implementation using dual-channel decoding is discussed in the context of . Figure 2A , Figure 2B Description of the corresponding elements in the context of (joint channel encoding 228 and joint channel decoding 204).
[0204] Fig. 8A An audio decoder for generating a decoded binaural signal is shown. The audio decoder comprises four audio decoders 802 for decoding the encoded binaural signal to obtain a first set of first spectral portions, and further comprises a parameterized decoder 804 for providing parameterized data for a second set of second spectral portions and a binaural identification additionally for identifying a first binaural representation or a different second binaural representation of the second spectral portions. Furthermore, a frequency regenerator 806 is arranged for regenerating the second spectral portion based on the first spectral portion of the first set of first spectral portions and the parameterized data for the second portion and the binaural identification of the second portion. Figure 8B Different combinations of binaural representations in the source range and the destination range are shown. The source range can be represented in a first binaural representation, and the destination range can also be represented in the first binaural representation. Alternatively, the source range can be represented in a first binaural representation, and the destination range can be represented in a second binaural representation. In addition, as Figure 8B As shown in the third column of , the source range may be in accordance with the second two-channel representation and the destination range may be in accordance with the first two-channel representation. Finally, both the source range and the destination range may be in accordance with the second two-channel representation. In one embodiment, the first two-channel representation is a separated two-channel representation in which the two channels of the two-channel signal are represented separately. Then, the second two-channel representation is a joint representation in which the two channels of the two-channel representation are represented jointly, i.e., further processing or representation transformation is required to recalculate the separated two-channel representation as required for output to the corresponding speakers.
[0205] In one implementation, the first two-channel representation may be a left / right (L / R) representation, and the second two-channel representation is a joint stereo representation. However, other two-channel representations besides left / right or M / S or stereo prediction may also be applied and used in the present invention.
[0206] Figure 8C Shown by Fig. 8A Flow chart of the operations performed by the audio decoder of FIG. 802. In step 812, the audio decoder 802 performs source range decoding. Figure 3A, the source range may include scale factor bands SCB1 to SCB3. In addition, there may be a binaural identification of each scale factor band, and scale factor band 1 may be, for example, represented as a first representation (such as L / R), while the third scale factor band may be represented as M / S or predicted downmix / residual according to a second binaural representation. Thus, step 812 may produce different representations of different frequency bands. Then, in step 814, frequency regenerator 806 is configured to select a source range for frequency regeneration. In step 816, frequency regenerator 806 then checks the representation of the source range, and in block 818, frequency regenerator 806 compares the binaural representation of the source range with the binaural representation of the target range. If the two representations are identical (820), frequency regenerator 806 provides separate frequency regeneration for each channel of the binaural signal. However, when the two representations detected in block 818 are different, signal stream 824 is used, and block 822 calculates a further two-channel representation from the source range, and uses the calculated further two-channel representation to reproduce the target range. Fig. 8A The decoder makes it possible to use a source range according to a first two-channel representation to reproduce a destination range represented with a second two-channel identification. Of course, the invention also makes it possible to use a source range with the same two-channel identification to reproduce a destination range. And in addition, the invention makes it possible to: reproduce a destination range with a two-channel identification representing a joint two-channel representation, and then transform this representation into the separated channel representation required for storage or transmission to the corresponding loudspeakers of the two-channel signal.
[0207] It is emphasized that the two channels of the two-channel representation may be two stereo channels such as a left channel and a right channel. However, the signal may also be a multi-channel signal having, for example, five channels and a subwoofer channel or having even more channels. Then, the FIG. 8A to FIG. 8E The paired two-channel processing discussed in the context of FIG. 1 , where the pairs may be, for example, left and right channels, left and right surround channels, and center and LFE (subwoofer) channels. Any other pairings may be used to represent, for example, six input channels by three two-channel processing processes.
[0208] Fig.8D Shown with Fig. 8A The source range or core decoder 830 may correspond to the audio decoder 802. The other blocks 832, 834, 836, 838, 840, 842, 846 may be Fig. 8APart of the frequency regenerator 806 of FIG. 832. In particular, block 832 is a representation transformer for transforming the source range representations in the respective frequency bands so that there is a complete set of source ranges according to the first representation on the one hand and a complete set of source ranges according to the second binaural representation on the other hand at the output of block 832. The two complete source range representations may be stored in a memory 834 for the two representations of the source ranges.
[0209] Then, block 836 applies frequency tile generation, which uses the source range ID as input and additionally uses the target range's binaural ID as input. Based on the target range's binaural ID, the frequency tile generator accesses memory 834 and receives a source range binaural representation that matches the target range's binaural ID input to the frequency tile regenerator at 835. Thus, when the target range's binaural ID indicates joint stereo processing, then the frequency tile generator 836 accesses memory 834 to obtain the joint stereo representation of the source range indicated by the source range ID 833.
[0210] Frequency tile generator 836 performs this operation for each target range, and the output of the frequency tile generator makes each channel represented by the channel indicated by the binaural identifier present. Then, envelope adjustment is performed by envelope adjuster 838. Envelope adjustment is performed in the binaural domain identified by the binaural identifier. For this reason, envelope adjustment parameters are required, and these parameters are sent from the encoder to the decoder with the same binaural representation described. When the binaural identifier of the target range to be processed by the envelope adjuster has a binaural identifier representing a binaural representation different from the envelope data of the target range, then the parameter converter 840 converts the envelope parameters into the required binaural representation. When, for example, the binaural identifier of a frequency band represents joint stereo coding and when the parameters of the target range have been sent as L / R envelope parameters, then the parameter converter calculates the joint stereo envelope parameters according to the described L / R envelope parameters, so that the correct parameterization representation is used for the spectrum envelope adjustment of the target range.
[0211] In another preferred embodiment, when joint stereo is used for the target frequency band, the envelope parameters are already sent as joint stereo parameters.
[0212] When it is assumed that the input to the envelope adjuster 838 is a set of target ranges with different binaural representations, the output of the envelope adjuster 838 is also a set of target ranges in different binaural representations. When the target ranges have a joint representation such as M / S, the target ranges are processed by the representation converter 842 to calculate the separate representations required for storage or transmission to the speakers. However, when the target ranges already have separate representations, the signal stream 844 is used and the representation converter 842 is bypassed. At the output of block 842, a binaural spectral representation as a separated binaural representation is obtained, which can then also be further processed as shown by block 846, where the further processing can be, for example, frequency / time conversion or any other desired processing.
[0213] Preferably, the second spectral portion corresponds to a frequency band, and the binaural identification is arranged to correspond to Figure 8B The parameterized decoder is then configured to check whether the flag is set and to control the frequency regenerator 106 to use the first representation or the second representation of the first spectrum portion according to the flag.
[0214] In one embodiment, only Figure 3A The reconstruction range of the IGF start frequency 309 has a binaural identification of the different reconstruction frequency bands. In yet another embodiment, this also applies to the frequency range below the IGF start frequency 309.
[0215] In yet another embodiment, the source and target band identifiers may be adaptively determined by similarity analysis, however, the binaural processing of the present invention may also be applied when there is a fixed association of source range to target range. The source range may be used to recreate a wider target range by a harmonic spectrum tiling operation or a replicated frequency tiling operation using two or more frequency tiling operations similar to the processing of multiple patches known from efficient AAC processing with respect to frequency.
[0216] Fig. 8E An audio encoder for encoding a two-channel audio signal is shown. The encoder comprises a time-spectrum converter 860, which is used to convert the two-channel audio signal into a spectrum representation. In addition, a spectrum analyzer 866 is also included, which is used to convert the two-channel audio signal into a spectrum representation. In addition, the spectrum analyzer 866 is also configured to perform analysis to determine which spectrum parts will be encoded with high resolution, that is, find out a first group of first spectrum parts and find out a second group of second spectrum parts additionally.
[0217] Furthermore, the binaural analyzer 864 is arranged for analyzing the second set of second spectral portions to determine a binaural identification for identifying the first binaural representation or the second binaural representation.
[0218] Depending on the result of the binaural analyzer, the frequency bands according to the second spectral representation are parameterized using the first binaural representation or the second binaural representation, and this is performed by a parameter encoder 868. The core frequency range, i.e. below Figure 3A The frequency bands of the IGF start frequency 309 are encoded by the core encoder 870. The results of blocks 868 and 870 are input to the output interface 872. As shown, the binaural analyzer provides a binaural identification of each frequency band above the IGF start frequency or the entire spectrum range, and the binaural identification is also forwarded to the output interface 872 so that this data is also included in the encoded signal 873 output by the output interface 872.
[0219] In addition, preferably, the audio encoder comprises a band converter 862. Based on the decision of the two-channel analyzer 864, the output signal of the time-spectrum converter 860 is converted into a representation represented by the two-channel analyzer, specifically by the two-channel ID 835. Thus, the output of the band converter 862 is a set of frequency bands, wherein each frequency band can be represented according to the first two-channel representation or a different second two-channel representation. When the present invention is applied to the full frequency band, that is, when both the source range and the reconstruction range are processed by the band converter, the spectrum analyzer 860 can analyze the representation. However, alternatively, the spectrum analyzer 860 can also analyze the signal output by the time-spectrum converter as represented by the control line 861. Thus, the spectrum analyzer 860 can apply a preferred pitch analysis to the output of the band converter 862 or the output of the time-spectrum converter 860 before being processed by the band converter 862. In addition, the spectrum analyzer can apply the identification of the best matching source range of a certain target range to the result of the band converter 862 or the result of the time-spectrum converter 860.
[0220] Then, refer to 9A to 9D , which is used to show the Figure 3A and Figure 3B The preferred calculation of energy information values has been discussed in the context of .
[0221] Prior art audio encoders apply various techniques to minimize the amount of data representing a given audio signal. Audio encoders such as USAC [1] apply a time-to-frequency transform such as MDCT to obtain a spectral representation of a given audio signal. These MDCT coefficients are quantized using the psychoacoustic characteristics of the human auditory system. If the available bit rate is reduced, the quantization becomes coarse, introducing a large number of zero spectral values, which causes audible artifacts at the decoder side. In order to improve the perceived quality, prior art decoders use random noise to fill these zero spectral portions. The IGF method obtains tiles from the remaining non-zero signal to fill those gaps in the spectrum. It is crucial for the perceived quality of the decoded audio signal to preserve the spectral envelope and the energy distribution of the spectral coefficients. The energy adjustment method presented here uses the transmitted side information to reconstruct the spectral MDCT envelope of the audio signal.
[0222] In eSBR
[15] , the audio signal is downsampled by at least a factor of 2 and the high-frequency part of the spectrum is completely zeroed [1, 17]. This deleted part is replaced by eSBR at the decoder side using parameterization techniques. eSBR implies the use of an additional transform, the QMF transform, which is used to replace the empty high-frequency part and resample the audio signal
[17] . This increases the computational complexity and memory consumption of the audio encoder.
[0223] The USAC coder
[15] offers the possibility to fill spectral holes (zero spectral lines) with random noise, but has the following disadvantages: random noise cannot preserve the temporal fine structure of transient signals, nor can it preserve the harmonic structure of tonal signals.
[0224] The region in which eSBR operates on the decoder side is completely deleted by the encoder [1]. As a result, eSBR tends to delete tonal lines in high-frequency regions or distort the harmonic structure of the original signal. The regeneration of the tonal components of eSBR in the copied frequency range has a low accuracy when the QMF frequency resolution of the eSBR is low and the reinsertion of the sinusoidal components is only possible at the coarse resolution of the underlying filter bank.
[0225] eSBR uses a technique to adjust the energy of the patched region, i.e., spectral envelope adjustment [1]. This technique uses the energy values transmitted on the QMF frequency-time grid to reshape the spectral envelope. Existing techniques do not handle partially deleted spectra and, because of the high temporal resolution, tend to require a relatively large number of bits to transmit appropriate energy values or to apply coarse quantization to the energy values.
[0226] While it uses the conventional MDCT transform computed as described in
[15] , the IGF method does not require additional transforms.
[0227] The energy adjustment method presented here reconstructs the spectral envelope of the audio signal using the side information generated by the encoder. As outlined below, the side information is generated by the encoder:
[0228] a) Applying a windowed MDCT transform to the input audio signal [16, Section 4.6], optionally computing the windowed MDST or estimating the windowed MDST from the computed MDCT
[0229] b) Apply TNS / TTS to the MDCT coefficients [15, Section 7.8]
[0230] c) Calculate the frequency above IGF onset (f IGFstart ) until the IGF termination frequency (f IGFstop ) is the average energy of each MDCT scale factor band
[0231] d) Quantify the average energy value
[0232] f IGFstart and f IGFstop is a parameter given by the user.
[0233] The values calculated according to steps c) and d) are losslessly encoded and sent to the decoder together with the bitstream as side information.
[0234] The decoder receives the transmitted values and uses them to adjust the spectral envelope.
[0235] a) Dequantize the transmitted MDCT value
[0236] b) If signaled, then conventional USAC noise filling is applied
[0237] c) Application of IGF patch filling
[0238] d) Dequantize the energy value sent
[0239] e) Adjust the spectrum envelope scale factor and frequency band
[0240] f) If signaled, use TNS / TTS
[0241] set up is the real-valued spectral representation of the MDCT transform of the windowed audio signal with a window length of 2N. The transform is described in
[16] . Optionally, the encoder Optionally apply TNS.
[0242] In [16, 4.6.2], the scale factor bands are described A scale factor band is a set of indices and is denoted by scb in this paper.
[0243] Each scb k The limits of scb are defined by the array swb_offset[16, 4.6.2], where swb_offset[k] and swb_offset[k+1]-1 define the limits of scb. k The first and last indices of the lowest and highest spectral coefficients included in . The scale factor band is represented as:
[0244] scb k :={swb_offset[k], 1+swb_offset[k], 2+swb_offset[k],..., swb_offset[k+1]-1}
[0245] If the encoder uses the IGF tool, the user defines the IGF start frequency and the IGF stop frequency. These two values are mapped to the optimal filling scale factor band indexes igfStartSfb and igfStopSfb. These two are signaled to the decoder in the bitstream.
[0246]
[16] describes long block transform and short block transform. For long blocks, only one set of spectral coefficients is sent to the decoder along with a set of scale factors. For short blocks, 8 short windows are calculated along with 8 different sets of spectral coefficients. To save bit rate, the scale factors of these 8 short block windows are grouped by the encoder.
[0247] In the case of IGF, the method presented here uses conventional scale factor bands to group the spectral values sent to the decoder:
[0248]
[0249] Among them, k=igfStartSfb, 1+igfStartSfb, 2+igfStartSfb,..., igfEndSfb.
[0250] For quantization, calculation All values is sent to the decoder.
[0251] We assume that the encoder decides to group num_window_group scale factor groups. Let w denote the group decomposition as the set {0,1,2,…,7} of indices of 8 short windows. l represents the l-th subset of w, where l represents the index of the window group and 0≤l<num_window_group.
[0252] For short block calculations, the user defined IGF start / end frequencies are mapped to the appropriate scale factor bands. However, for simplicity, for short blocks: k = igfStartSfb, 1 + igfStartSfb, 2 + igfStartSfb, ..., igfEndSfb.
[0253] IGF energy calculation uses grouping information to calculate the value E k,l To group:
[0254]
[0255] For quantization, calculation All values is sent to the decoder.
[0256] The encoding formula mentioned above uses only real-valued MDCT coefficients To obtain a more stable energy distribution within the IGF range, i.e., to reduce the temporal amplitude fluctuations, an alternative method can be used to calculate the value
[0257] set up is the real-valued spectral representation of the MDCT transform of the windowed audio signal with a window length of 2N, and is the spectral representation of the real-valued MDST transform of the same portion of the audio signal. It can be calculated exactly or according to Estimating MDST spectral representation represents the complex spectrum representation of the windowed audio signal, such that For its actual part, is its imaginary part. Optionally, the encoder and Apply TNS.
[0258] Now, the energy of the original signal in the IGF range can be measured using the following formula:
[0259]
[0260] The real-valued energy and complex-valued energy of the reconstruction frequency band, i.e., the energy of the reconstruction IGF range scb k The tiles that should be used on the decoder side are calculated using the following formula:
[0261]
[0262] Among them, tr k Is a set of indexes - dependent on scb k In the above two formulas, replace the index group scb k , you can use the group (defined later in this article) to reconstruct tr k To achieve a more accurate value E t and E r .calculate
[0263]
[0264] If E tk > 0, otherwise f k =0.
[0265] Take advantage now Calculate E k A more stable version of k The MDCT values are impaired by the fact that they do not obey Parseval's theorem, and therefore, they do not reflect the complete energy information of the spectral values.
[0266] As noted earlier, for short blocks, assume that the encoder decides to group num_window_group scale factor groups. As described above, w l represents the l-th subset of w, where l represents the index of the window group and 0≤l<num_window_group.
[0267] Again, the calculations outlined above for calculating E can be calculated k,l An alternative to a more stable version of . is the MDCT transformed windowed audio signal of length 2N and is an MDST-transformed windowed audio signal of length 2N, and calculates:
[0268]
[0269] Similarly, calculate,
[0270]
[0271] And using the factor f k,l To deal with,
[0272]
[0273] It is used to adjust the previously calculated E rk,l :
[0274]
[0275] Calculate according to the above
[0276] The process of using not only the energy of the reconstructed frequency bands derived from the complex reconstruction frequency bands or from the MDCT values but also the energy information from the source range provides an improved energy reconstruction.
[0277] Specifically, the parameter calculator 1006 is configured to calculate energy information about the reconstruction frequency band using information about the energy of the reconstruction frequency band and additionally using information about the energy of a source range to be used for reconstructing the reconstruction frequency band.
[0278] Furthermore, the parameter calculator 1006 is configured to calculate energy information (E ok ) to calculate additional energy information (E ) about the source range of the real-valued part of the complex spectrum of the original signal to be used for reconstructing the reconstruction band rk ), wherein the parameter calculator is configured to use energy information (E ok ) and additional energy information (E rk ) to calculate the energy information about the reconstructed frequency band.
[0279] Furthermore, the parameter calculator 1006 is configured to determine first energy information (E ) of the reconstructed scale factor bands of the complex spectrum of the original signal. ok ); for determining second energy information (E ) about a source range of the complex spectrum of the original signal to be used for reconstructing the reconstructed scale factor bands tk ); for determining a third energy information (E ) about a source range of a real part of the complex spectrum of the original signal to be used for reconstructing the reconstructed scale factor bands rk ); for determining weighting information based on a relationship between at least two of the first energy information, the second energy information and the third energy information; and for weighting one of the first energy information and the third energy information using the weighting information to obtain weighted energy information and using the weighted energy information as energy information about the reconstructed frequency band.
[0280] An example for the calculation is as follows, but many other examples will occur to those skilled in the art given the general principles above:
[0281] A)
[0282] f_k=E_ok / E_tk;
[0283] E_k = sqrt(f_k*E_rk);
[0284] B)
[0285] f_k=E_tk / E_ok;
[0286] E_k = sqrt((1 / f_k)*E_rk);
[0287] C)
[0288] f_k=E_rk / E_tk;
[0289] E_k=sqrt(f_k*E_ok)
[0290] D)
[0291] f_k=E_tk / E_rk;
[0292] E_k=sqrt((1 / f_k)*E_ok)
[0293] All these examples acknowledge the fact that, although only real MDCT values are processed on the decoder side, the actual calculation is a time domain anti-aliasing process implicitly using complex numbers due to the overlap and addition. However, in particular, the determination 918 of tile energy information for further spectral portions 922, 923 of the reconstructed frequency band 920 having frequency values different from the first spectral portion 921 having frequencies in the reconstructed frequency band 920 depends on the real MDCT values. Therefore, the energy information sent to the decoder will usually be smaller than the energy information E of the reconstructed frequency band about the complex spectrum of the original signal. ok For example, for case C above, this means that the factor f_k (weighted information) will be less than 1.
[0294] On the decoder side, if the IGF tool is signaled to be on, the value sent is obtained from the bitstream and should be used for all k=igfStartSfb, 1+igfStartSfb, 2+igfStartSfb, ..., igfEndSfb To quantify.
[0295] The decoder dequantizes the transmitted MDCT value into And calculate the remaining residual energy:
[0296]
[0297] where k is within the range defined above.
[0298] We indicate This set includes scale factor bands scb that have been quantized to zero by the encoder. k All indexes of .
[0299] The IGF obtains the subband method (not described here) for filling the spectral gaps caused by the coarse quantization of the MDCT spectral values at the encoder side by using the non-zero values of the transmitted MDCT. x will also include values that replace all previously cleared values. The tile energy is calculated by:
[0300]
[0301] where k is within the range defined above.
[0302] The energy lost in the reconstructed band is given by:
[0303] mE k :=|scb k |E k 2 -sE k
[0304] And the gain factor used for adjustment is obtained by the following formula:
[0305]
[0306] Where g′=min(g, 10).
[0307] The spectral envelope adjustment using this gain factor is: for all and k is within the range defined above,
[0308] x i : =g′x i
[0309] This reshapes the spectral envelope of x into the original spectral envelope shape.
[0310] For short window sequences, all the calculations outlined above remain the same in principle, but consider the grouping of scale factor bands. We denote the energy value of the dequantized group obtained from the bitstream as F k,l .calculate
[0311]
[0312] as well as
[0313]
[0314] The index j describes the window index of the short block sequence.
[0315] Calculate mE k,l :=|scb k |E k,l 2 -sE k,l
[0316] as well as
[0317] Where g′=min(g,10)
[0318] For all
[0319] Applicationx j,i : =g′x j,i .
[0320] For low bit rate applications, the value E k Pairwise grouping of is possible without losing too much precision. This method is only applied by long blocks:
[0321]
[0322] Among them, k=igfStartSfb, 2+igfStartSfb, 4+igfStartSfb, ..., igfStartSfb.
[0323] Again, after quantization, all values E k>>1 is sent to the decoder.
[0324] Fig. 9A A device for decoding an encoded audio signal comprising an encoded representation of a first set of first spectral portions and an encoded representation of parametric data representing spectral energy of a second set of second spectral portions is shown. Fig. 9A In the figure, the first group of first spectrum parts is shown at 901a, and at Fig. 9A901b of represents the encoded representation of parameterized data. An audio decoder 900 is provided for decoding the encoded representation 901a of the first group of first spectrum parts to obtain a decoded first group of first spectrum parts 904, and for decoding the encoded representation of the parameterized data to obtain decoded parameterized data 902 of a second group of second spectrum parts representing individual energies of individual reconstruction bands, wherein the second spectrum parts are located in the reconstruction band. In addition, a frequency regenerator 906 is provided for reconstructing the spectrum values of the reconstruction band including the second spectrum parts. The frequency regenerator 906 uses the first spectrum parts of the first group of first spectrum parts and the individual energy information about the reconstruction band, wherein the reconstruction band includes the first spectrum parts and the second spectrum parts. The frequency regenerator 906 includes a calculator 912, which is used to determine the residual energy information of the cumulative energy of the first spectrum parts including the frequencies in the reconstruction band. In addition, the frequency regenerator 906 includes a calculator 918, which is used to determine tile energy information of another spectral portion of the reconstructed frequency band, and the frequency values are different from the first spectral portion, wherein these frequency values have frequencies in the reconstructed frequency band, wherein the other spectral portion is to be generated by frequency regeneration using a first spectral portion different from the first spectral portion in the reconstructed frequency band.
[0325] The frequency regenerator 906 also includes a calculator 914 for determining the missing energy in the reconstructed frequency band, and the calculator 914 operates using the individual energies of the reconstructed frequency band and the residual energy generated by block 912. In addition, the frequency regenerator 906 includes a spectrum envelope adjuster 916 for adjusting additional spectrum portions in the reconstructed frequency band based on the missing energy information and the tile energy information generated by block 918.
[0326] Reference Fig. 9C , Fig. 9C A certain reconstructed frequency band 920 is shown. The reconstructed frequency band includes a first spectral portion in the reconstructed frequency band schematically shown at 921, such as the first spectral portion 306 in Figure 3a. In addition, the remaining spectral values in the reconstructed frequency band 920 are to be generated using, for example, source areas from scale factor bands 1, 2, 3 below the smart gap filling start frequency of Figure 3a. The frequency regenerator 906 is configured to generate original spectral values of second spectral portions 922 and 923. Then, as shown in Figure 9c, a gain factor g is calculated to ultimately adjust the original spectral values in the frequency bands 922, 923 to obtain a reconstructed and adjusted second spectral portion in the reconstructed frequency band 920, which now has the same spectral resolution, i.e. the same line distance, as the first spectral portion 921. It is important to understand that Fig. 9CThe first spectral portion in the reconstructed frequency band shown at 921 in FIG is decoded by the audio decoder 900 without being affected by the envelope adjustment execution block 916 of FIG. 9b . Alternatively, the first spectral portion in the reconstructed frequency band represented at 921 is left as is because the first spectral portion is output by the full bandwidth or full rate audio decoder 900 via line 904.
[0327] Subsequently, a specific example using real numbers is discussed. The remaining residual energy calculated by block 912 is, for example, 5 energy units, and this energy is the energy of the exemplarily represented 4 spectral lines in the first spectral portion 921 .
[0328] In addition, Figure 3B or Figure 3A The energy value E3 of the reconstructed band corresponding to the scale factor band 6 of is equal to 10 units. Importantly, the energy value includes not only the energy of the spectral parts 922, 923, but also the total energy of the reconstructed band 920 calculated on the encoder side, i.e. before performing the spectral analysis using, for example, channel masking. Thus, 10 energy units cover the first spectral part and the second spectral part in the reconstructed band. Then, it is assumed that the energy of the source range data of blocks 922, 923 or the original target range data of blocks 922, 923 is equal to 8 energy units. Thus, 5 units of lost energy are calculated.
[0329] Based on the lost energy divided by the tile energy tEk, a gain factor of 0.79 is calculated. Then, the original spectral lines of the second spectral parts 922, 923 are multiplied by the calculated gain factor. Thus, only the spectral values of the second spectral parts 922, 923 are adjusted, while the spectral lines of the first spectral part 921 are not affected by the envelope adjustment. After multiplying the original spectral values of the second spectral parts 922, 923, a complete reconstructed frequency band is calculated, which includes the first spectral part in the reconstructed frequency band and includes the spectral lines in the second spectral parts 922, 923 in the reconstructed frequency band 920.
[0330] Preferably, the source range used to generate the original spectral data in the frequency bands 922 , 923 is below the IGF start frequency 309 with respect to frequency, while the reconstruction frequency band 920 is above the IGF start frequency 309 .
[0331] Furthermore, preferably, the reconstruction band boundaries coincide with the scale factor band boundaries. Thus, in one embodiment, the reconstruction bands have the size of the corresponding scale factor bands of the core audio decoder, or are sized such that when energy pairing is applied, the energy value of the reconstruction band provides the energy of an integer number of scale factor bands equal to or greater than 2. Thus, when it is assumed that energy accumulation is performed for scale factor band 4, scale factor band 5, and scale factor band 6, then the lower frequency boundary of reconstruction band 920 is equal to the lower boundary of scale factor band 4, and the upper frequency boundary of reconstruction band 920 coincides with the upper boundary of scale factor band 6.
[0332] Then, discuss Fig.9D To show Fig. 9A The audio decoder 900 receives dequantized spectral values corresponding to a first spectral portion of a first set of spectral portions and, in addition, converts the dequantized spectral values into Figure 3B The scale factors of the scale factor bands shown in FIG. 1 are provided to the inverse scaling block 940. The inverse scaling block 940 provides Figure 3A All first groups of first spectrum parts of the IGF start frequency 309 and additionally provide first spectrum parts above the IGF start frequency, i.e., are located at Fig.9D The reconstructed frequency band shown at 941 in Figure 3A Furthermore, the first spectral portion in the source band for frequency tiling filling in the reconstruction band is provided to the envelope adjuster / calculator 942, and the block additionally receives the first spectral portion provided as Fig.9D The energy information of the reconstructed frequency band of the parameterized side information of the encoded audio signal shown at 943 in FIG. Then, the envelope adjuster / calculator 942 provides Fig. 9B and Fig. 9C The function of , and finally outputs the adjusted spectrum value of the second spectrum part in the reconstructed frequency band. These adjusted spectrum values 922, 923 and Fig.9D The first spectral portion 921 in the reconstructed frequency band represented by the line 941 in jointly represents the complete spectral representation of the reconstructed frequency band.
[0333] Then, refer to Fig. 10A and Fig. 10B , Fig. 10A and Fig. 10BA preferred embodiment of an audio encoder for encoding an audio signal to provide or generate an encoded audio signal is described. The encoder comprises a time / spectral converter 1002, which feeds a spectrum analyzer 1004, and the spectrum analyzer 1004 is connected to a parameter calculator 1006 on the one hand and to an audio encoder 1008 on the other hand. The audio encoder 1008 provides an encoded representation of a first group of first spectral portions and does not cover a second group of second spectral portions. On the other hand, the parameter calculator 1006 provides energy information of reconstructed frequency bands covering the first spectral portions and the second spectral portions. In addition, the audio encoder 1008 is configured to generate a first encoded representation of a first group of first spectral portions with a first spectral resolution, wherein the audio encoder 1008 provides scale factors for all frequency bands of the spectral representation generated by block 1002. In addition, as Figure 3B As shown in , the encoder provides a Figure 3A Energy information of at least the reconstructed frequency band above the shown IGF start frequency 309. Thus, for a reconstructed frequency band preferably coinciding with a scale factor frequency band or with a group of scale factor frequency bands, two values are given, namely the corresponding scale factor from the audio encoder 1008 and the energy information additionally output by the parameter calculator 1006.
[0334] Preferably, the audio encoder has scale factor bands with different frequency bandwidths, i.e. with different numbers of spectral values. Therefore, the parameterization calculator comprises a normalizer 1012 for normalizing the energies of the different bandwidths with respect to the bandwidth of a particular reconstruction band. To this end, the normalizer 1012 receives as input the energy in the band and a number of spectral values in the band, and then the normalizer 1012 outputs a normalized energy per reconstruction / scale factor band.
[0335] also, Fig. 10A The parameterization calculator 1006a includes an energy value calculator, which is used to Fig. 10A 1007 in FIG. 1004 receives control information from a core or audio encoder 1008. The control information may include information about the long / short blocks used by the audio encoder and / or grouping information. Thus, while the information about the long / short blocks and the grouping information about the short windows are related to "time" grouping, the grouping information may also refer to spectral grouping, i.e., grouping the two scale factor bands into a single reconstructed band. Thus, when only the spectral portion has been grouped, the energy value calculator 1014 outputs a single energy value for each grouped band covering the first spectral portion and the second spectral portion.
[0336] Fig. 10DYet another embodiment for implementing spectrum grouping is shown. To this end, block 1016 is configured to calculate energy values for two adjacent frequency bands. Then, in block 1018, the energy values of the two adjacent frequency bands are compared, and when the energy values are not significantly different or differ to a lesser extent than, for example, those defined by a threshold, a single (normalized) value for the two frequency bands is generated as shown in block 1020. As shown by line 1019, block 1018 can be bypassed. In addition, the generation of a single value for two or more frequency bands performed by block 1020 can be controlled by an encoder bit rate controller 1024. Thus, when the bit rate is to be reduced, the encoding bit rate controller 1024 controls block 1020 to generate a single normalized value for two or more frequency bands, even if the comparison in block 1018 is not allowed to group the energy information values.
[0337] In case the audio encoder performs grouping of two or more short windows, this grouping is also applied for the energy information. When the core encoder performs grouping of two or more short blocks, then for these two or more blocks only a single set of scale factors is calculated and sent. Then, at the decoder side, the audio decoder applies the same set of scale factors for both grouped windows.
[0338] Regarding the energy information calculation, the spectrum values in the reconstruction band are accumulated for two or more short windows. In other words, this means that the spectrum values in a certain reconstruction band of a short block and a subsequent short block are accumulated together, and only a single energy information is sent for this reconstruction band covering two short blocks. Then, at the decoder side, the envelope adjustment discussed with respect to Figures 9a to 9d is not performed separately for each short block, but for a group of grouped short windows, the envelope adjustment discussed with respect to Figures 9a to 9d is performed together.
[0339] Then, the corresponding normalization is applied again so that even if any grouping in frequency or grouping in time has been performed, the normalization easily enables: for the calculation of the energy value information on the decoder side, on the one hand only the energy value information and the amount of spectral lines in the reconstructed frequency band or in a group of grouped reconstructed frequency bands have to be known.
[0340] In prior art BWE schemes, the reconstruction of the HF spectral region above a given so-called crossover frequency is usually based on spectral patching. Typically, the HF region consists of a number of adjacent patches, and each of these patches originates from a bandpass (BP) region of the LF spectrum below a given crossover frequency. Within a filter bank representation of the signal, such a system copies a set of adjacent subband coefficients in the LF spectrum into the target region. The boundaries of the selected groups are usually system-dependent, not signal-dependent. For some signal contents, this static patching selection can lead to unpleasant timbre and coloration of the reconstructed signal.
[0341] Other approaches convert LF signals to HF by signal adaptive single sideband (SSB) modulation. Such approaches have high computational complexity compared to [1], since they operate on time domain samples at high sampling rates. Furthermore, the patching can become unstable, especially for non-tonal signals (e.g., non-speech speech), so that prior art signal adaptive patching introduces impairments to the signal.
[0342] The method of the present invention is called intelligent gap filling (IGF), and according to its preferred configuration, it is applied to BWE systems based on time-frequency conversion, such as modified discrete cosine transform (MDCT). However, the teachings of the present invention are generally applicable, for example, in systems based on quadrature mirror filter banks (QMF).
[0343] The advantage of the MDCT-based IGF configuration is seamless integration into MDCT-based audio codecs such as MPEG Advanced Audio Coding (AAC).Sharing the same transform for waveform audio coding and BWE significantly reduces the overall computational complexity of the audio codec.
[0344] Furthermore, the present invention provides solutions to inherent stability issues found in prior art adaptive patching schemes.
[0345] The proposed system is based on the observation that for some signals, unguided patching selection can lead to timbre changes and signal coloration. If a signal is tonal in the spectral source region (SSR) and noise-like in the spectral target region (STR), patching the noise-like STR by the tonal SSR can lead to unnatural timbre. The timbre of the signal can also change because the tonal structure of the signal can become misaligned or even destroyed by the patching process.
[0346] The proposed IGF system performs intelligent tile selection using cross-correlation as a measure of similarity between a particular SSR and a particular STR. The cross-correlation of two signals provides a measure of the similarity of those signals and in addition the lag of the maximum correlation and its sign. Thus, the method of correlation-based tile selection can also be used to precisely adjust the spectral offset of the replicated spectrum to become as close as possible to the original spectral structure.
[0347] The basic contribution of the proposed system is the selection of an appropriate similarity measure and furthermore a technique for stabilizing the tile selection process. The proposed technique provides an optimal balance between instantaneous signal adaptation and simultaneous temporal stability. Providing temporal stability is particularly important for signals with very small similarities of SSR and STR and thus exhibiting low cross-correlation values or when an ambiguous similarity measure is employed. In this case, stabilization prevents pseudo-random behavior of the adaptive tile selection.
[0348] For example, one class of signals that often cause problems with prior art BWE is characterized by different energy concentrations to arbitrary spectral regions, such as Fig. 12A (left). Although there are available methods to adjust the spectral envelope and pitch of the reconstructed spectrum in the target region, for some signals these methods are not able to Fig. 12A (right) shows good preservation of timbre. Fig. 12A In the example shown in FIG. 1 , the target region of the original signal is above the so-called crossover frequency f xover ( Fig. 12A , left side) decreases approximately linearly. In contrast, in the reconstructed spectrum ( Fig. 12A , right), there are different groups of valleys that are perceived as timbre coloration artifacts.
[0349] An important step of the new method is to define a set of tiles from which subsequent similarity-based selection can be performed. First, the tile boundaries between the source and target regions must be defined relative to each other. Thus, the IGF starting frequency f of the core encoder is IGFstart With the highest available frequency f IGFstop The target region between is divided into tiles of an arbitrary integer number nTar, each of which has a predetermined size. Then, for each target tile tar[idx_tar], a set of source tiles src[idx_src] of equal size is generated. By this, the basic degrees of freedom of the IGF system are determined. The total number of source tiles nSrc is determined by the bandwidth of the source region,
[0350] bw src =(f IGFstart -f IGFmin )
[0351] Among them, f IGFmin is the lowest available frequency for tile selection such that an integer number of nSrc of source tiles fits bw src The minimum number of source tiles is 0.
[0352] To further increase the freedom of selection and adjustment, source tiles may be defined to overlap each other with an overlap factor between 0 and 1, where 0 means no overlap and 1 means 100% overlap. 100% overlap may imply that only one or no source tiles are available.
[0353] Fig. 12B An example of a tile boundary for a set of tiles is shown. In this case, all target tiles are associated with each source tile. In this example, the source tiles overlap by 50%.
[0354] For the target tile, the cross-correlation is calculated using the various source tiles at a lag of xcorr_maxLag bins upwards. For a given target tile idx_tar and source tile idx_src, xcorr_val[idx_tar][idx_src] gives the maximum value of the absolute cross-correlation between the tiles, whereas xcorr_lag[idx_tar][idx_src] gives the lag at which the maximum occurs, and xcorr_sign[idx_tar][idx_src] gives the sign of the cross-correlation at xcorr_lag[idx_tar][idx_src].
[0355] The parameter xcorr_lag is used to control the closeness of the match between the source and target tiles. This parameter results in reduced artifacts and helps better preserve the timbre and timbre of the signal.
[0356] In some scenarios, it may happen that the size of a particular target tile is larger than the size of the available source tiles. In this case, the available source tiles are repeated whenever necessary to completely fill the target tile. Still, a cross-correlation between the large target tile and the smaller source tiles may be performed in order to obtain the best position of the source tile in the target tile according to the cross-correlation lag xcorr_lag and sign xcorr_sign.
[0357] The cross-correlation of the original spectrum tile and the original signal may not be the most suitable similarity measure to be applied to an audio spectrum with a strong formant structure. Whitening of the spectrum removes coarse envelope information, thereby emphasizing the fine structure of the spectrum, which is most beneficial for evaluating tile similarity. For areas processed by IGF, whitening also helps to easily perform envelope shaping of STR at the decoder. Therefore, optionally, before calculating the cross-correlation, the tile and the source signal are whitened.
[0358] In other configurations, only the tiles are whitened using a predetermined process.The "whiten" flag sent indicates to the decoder that the same predetermined whitening process should be applied to the tiles within the IGF.
[0359] To whiten a signal, first a spectral envelope estimate is calculated. Then, the MDCT spectrum is divided by the spectral envelope. The spectral envelope estimate can be estimated based on the MDCT spectrum, the MDCT spectrum energy, the MDCT-based complex power spectrum, or the power spectrum estimate. From now on, the signal on which the envelope is estimated will be referred to as the base signal.
[0360] Computing the envelope of the base signal on the MDCT-based complex power spectrum or power spectrum estimate has the advantage that there are no temporal fluctuations on the tonal components.
[0361] If the base signal is in the energy domain, the MDCT spectrum must be divided by the square root of the envelope to properly whiten the signal.
[0362] There are different ways to calculate the envelope:
[0363] Transform the base signal using a discrete cosine transform (DCT), keep only the lower DCT coefficients (set the highest to zero), and then compute the inverse DCT
[0364] Compute the spectral envelope of a set of linear prediction coefficients (LPC) computed over a time-domain audio frame
[0365] Filter the base signal using a low-pass filter
[0366] Preferably, the last method is chosen. For applications requiring low computational complexity, the whitening of the MDCT spectrum can be simplified somewhat: First, the envelope is calculated by means of a moving average. This requires only two processor cycles per MDCT bin. Then, to avoid division and square root calculations, the spectral envelope is approximated by 2 n , where n is the integer logarithm of the envelope. In this domain, the square root operation simply becomes a shift operation, and furthermore the division by the envelope can be performed by another shift operation.
[0367] After calculating the correlation of each source tile with each target tile, for all nTar target tiles, the source tile with the highest correlation is selected to replace it. In order to best match the original spectral structure, the copied spectrum is modulated by an integer number of transform bins using a lag of the correlation. In the case of an odd lag, the tile is additionally modulated by multiplying by an alternating time sequence of -1 / 1 to compensate for the frequency-inverted representation of each other band within the MDCT.
[0368] Fig. 12C An example of the correlation between the source tile and the target tile is shown. In this example, the lag of the correlation is 5, so the source tile must be modulated by 5 bins towards the higher bins in the copy segment of the BWE algorithm. In addition, when the maximum correlation value is negative, the sign of the tile must be flipped, and the additional modulation as described above results in an odd lag.
[0369] Therefore, the total amount of side information sent from the encoder to the decoder may include the following data:
[0370] tileNum[nTar]: The index of the selected source tile per target tile
[0371] tileSign[nTar]: The symbol of the target tile
[0372] tileNum[nTar]: Lag of correlation per target tile
[0373] Tile trimming and stabilization are important steps in IGF. Using an example to illustrate the need and advantage, assume a static tonal audio signal like, for example, a steady pitch pipe note. If the source tiles for a given target region are always selected from the same source region across multiple frames, logic dictates that minimal artifacts are introduced. Even if the signal is assumed to be static, this condition does not hold well in each frame because similarity measures (e.g., correlation) for other equally similar source regions will dominate the similarity results (e.g., cross-correlation). This results in tileNum[nTar] between adjacent frames oscillating between two or three very similar choices. This can be a source of annoying musical noise such as artifacts.
[0374] To eliminate this type of artifacts, a set of source tiles should be pruned so that the remaining members of the source set are maximally different. This is for a set of source tiles S = {s1, s2, ... s n} is implemented as follows. For any source tile S i , and relate it to all other source tiles to find S i With S j The best correlation between them is stored in the matrix S x Here, S x [i][j] includes S i With S j The maximum absolute cross-correlation value between . Add the matrix S along the columns x , given the source tile S i The sum of the cross-correlations with all other source tiles T.
[0375] T[i]=S x [i][1]+S x [i][2]...+S x [i][n]
[0376] Here, T represents a measure of how similar a source is to other source tiles. If for any source tile i, T>threshold, then source tile i can be removed from a set of potential sources because it is highly correlated with other sources. The tile in the set of tiles with the smallest correlation that satisfies the condition in equation 1 is selected as the representative tile of the subgroup. In this way, it is ensured that the source tiles are maximally different from each other.
[0377] The tile pruning method also includes a memory 1148 of the pruned tile sets used in the previous frame. Tiles that were active in the previous frame are also retained in the next frame if there are alternative candidate tiles for pruning.
[0378] Assume that tiles S3, S4 and S5 are valid among tiles {s1, s2 ..., s5} in frame K, then in frame k+1, even though tiles S1, S3 and S2 compete to be pruned using S3 which is most correlated with other tiles, S3 is retained because it is a useful source tile in the previous frame and thus retaining it in the set of source tiles is beneficial for enforcing temporal continuity in tile selection. If denoted as T x This method is preferably applied if the mutual correlation between source i and target j of [i][j] is high.
[0379] Another approach for tile stabilization is to preserve the tile order from the previous frame k-1 if none of the source tiles in the current frame k correlates well with the target tile. x This may happen if the cross-correlation between source i and target j of [i][j] is low for all i, j.
[0380] For example, if T x [i][j]<0.6
[0381] The experimental threshold used now is, then for all nTar of this frame k,
[0382] tileNum[nTar] k =tileNum[nTar] k-1
[0383] The above two techniques greatly reduce the artifacts that occur due to the rapid change of the number of group tiles across multiple frames. Another additional advantage of the tile pruning and stabilization is that no extra information needs to be sent to the decoder and no changes to the decoder architecture are required. The proposed tile pruning is an excellent way to reduce potential musical noise such as artifacts or excessive noise in spectral areas.
[0384] Fig.11A An audio decoder for decoding an encoded audio signal is shown.The audio decoder comprises an audio (core) decoder 1102 for generating a first decoded representation of a first set of first spectral portions, the decoded representation having a first spectral resolution.
[0385] In addition, the audio decoder includes a parameterized decoder 1104 for generating a second decoded representation of a second group of second spectral portions, the second decoded representation having a second spectral resolution, the second spectral resolution being less than the first spectral resolution. In addition, a frequency regenerator 1106 is provided for receiving the decoded first spectral portion as a first input 1101 and receiving parameterized information as a second input at 1103, the parameterized information including source range information for each target frequency or target reconstruction band. Then, the frequency regenerator 1106 applies frequency regeneration by using the spectral values from the source range identified by the matching information to generate spectral data of the target range. Then, the outputs of the first spectral portion 1101 and the frequency regenerator 1107 are both input to a spectrum-time converter 1108 to ultimately generate a decoded audio signal.
[0386] Preferably, the audio decoder 1102 is a spectral domain audio decoder, although the audio decoder may also be implemented as any other audio decoder such as a time domain or a parametric audio decoder.
[0387] As in Fig. 11B As shown, the frequency regenerator 1106 may include the following functions: a block 1120 showing a source range selector tile modulator for odd lags, a whitening filter 1122 when a whitening flag 1125 is provided, and additionally includes a spectrum envelope adjustment function shown in block 1128 using the raw spectrum data generated by block 1120 or block 1122 or the cooperation of both blocks. In any case, the frequency regenerator 1106 may include a switch 1124 responsive to the received whitening flag 1125. When the whitening flag is set, the output of the source range selector / tile modulator for odd lags is input to the whitening filter 1122. However, then, if the whitening flag 1125 is not set for a certain reconstruction band, the bypass line 1126 is activated so that the output of block 1120 is provided to the spectrum envelope adjustment block 1128 without any whitening.
[0388] There may be more than one level of whitening (1123) signaled in the bitstream, and these levels may be signaled per tile. In the case where there are three levels signaled per tile, they should be coded in the following way:
[0389]
[0390]
[0391]
[0392] MID_WHITENING and STRONG_WHITENING refer to different whitening filters (1122), which may differ in the way they calculate the envelope (as described above).
[0393] When only a coarse spectrum tiling selection scheme is applied, the decoder side frequency regenerator may be controlled by the source range ID 1121. However, when a fine tuned spectrum tiling selection scheme is applied, then a source range hysteresis 1119 is additionally provided. Furthermore, assuming that the correlation calculation provides a negative result, then, additionally, the sign of the correlation may be applied to block 1120 such that the page data spectrum lines are each multiplied by "-1" to account for the negative sign.
[0394] Thus, if Fig.11A , Fig. 11B As discussed, the present invention ensures that the best audio quality is obtained due to the fact that the best matching source range for some destination or target range is calculated at the encoder side and applied at the decoder side.
[0395] Fig. 11C is an audio encoder for encoding an audio signal, which comprises a time-spectrum converter 1130, followed by a spectrum analyzer 1132, and additionally a parameter calculator 1134 and a core encoder 1136. The core encoder 1136 outputs the encoded source range, and the parameter calculator 1134 outputs matching information of the target range.
[0396] The encoded source range is sent to the decoder along with matching information about the target range, so that Fig.11A The decoder shown in is capable of performing frequency regeneration.
[0397] The parameter calculator 1134 is configured to calculate the similarity between the first spectrum portion and the second spectrum portion and to determine a matching first spectrum portion that matches the second spectrum portion for the second spectrum portion based on the calculated similarity. Fig. 12A , Fig. 12B The matching result of the different source ranges and target ranges shown in the figure determines that the selected matching pair includes the second spectral portion, and the parameter calculator is configured to provide the matching information identifying the matching pair into the encoded audio signal. Preferably, the parameter calculator 1134 is configured to use a predetermined target area in the second set of second spectral portions or a predetermined source area in the first set of first spectral portions, such as Fig. 12B Preferably, the predetermined target regions are non-overlapping, or the predetermined source regions are overlapping. When the predetermined source region is below Figure 3AWhen a subset of the first group of first spectrum portions is selected from the gap filling start frequency 309, and preferably, a predetermined target area covering a lower spectrum area coincides with its lower frequency boundary with the gap filling start frequency, so that any target range is located above the gap filling start frequency and the source range is located below the gap filling start frequency.
[0398] As discussed, fine granularity is obtained by comparing a target region to a source region without any hysteresis between the target region and the source region but with a specific hysteresis for the same target region. Fig.11D These lags are applied in the cross-correlation calculator 1140 , and the matching pair selection is ultimately performed by the tile selector 1144 .
[0399] In addition, it is preferred to perform source range and / or target range whitening as shown at block 1142. This block 1142 then provides a whitening flag to the bitstream, which is used to control Fig. 11B The decoder side switch 1124 of the cross-correlation calculator 1140 is also provided. In addition, if the cross-correlation calculator 1140 provides a negative result, the negative result is also signaled to the decoder. Thus, in a preferred embodiment, the tile selector outputs the source range ID, hysteresis, sign of the target range, and block 1142 additionally provides a whitening flag.
[0400] In addition, the parameter calculator 1134 is configured to perform source tile pruning 1146 by reducing the number of potential source ranges, namely, removing source patches from a group of potential source tiles based on a similarity threshold. Thus, when the similarity between two source tiles is greater than or equal to the similarity threshold, one of the two source tiles is removed from the group of potential sources, and the removed source tile is no longer used for further processing, and specifically, may not be selected by the tile selector 1144 or may not be used for the cross-correlation calculation between different source ranges and the target range performed in block 1140.
[0401] Different implementations have been described with reference to different figures. Figures 1A to 5C Involves full-rate or full-bandwidth encoder / decoder schemes. 6A to 7E Involves encoder / decoder schemes using TNS or TTS processing. FIG. 8A to FIG. 8E Concerning encoder / decoder schemes with specific binaural processing. 9A to 10D Involving specific energy information calculations and applications, and FIG. 11A to FIG. 12C Involves specific methods for paving selection.
[0402] All of these different aspects may be independent uses of the present invention, but additionally, they may also be essentially the same. Figure 2A and Figure 2B, are applied together. However, specific binaural processing can also be applied to the encoder / decoder scheme as shown in Figure 13, and is equally established for TNS / TTS processing, envelope energy information calculation and application or adaptive source range identification and the corresponding application of decoder side in reconstruction frequency band. On the other hand, when using or not using TNS / TTS processing, using or not using binaural processing, using or not using adaptive source range identification or using other types of energy calculations represented by spectrum envelope, full rate aspect is applicable. Thereby, it is clear that the feature of one of these various aspects can also be applicable in other aspects.
[0403] Although some aspects are described in the context of a device for encoding or decoding, it is clear that these aspects also provide a description of the corresponding method, wherein a block or device corresponds to a method step or the feature of a method step. Similarly, the aspects described in the context of the method step also provide the description of the corresponding block or project or the feature of the corresponding device. Some or all of the method steps can be performed by (or using) hardware devices such as microprocessors, codable computers or electronic circuits. In some embodiments, some one or more of the most important method steps can be performed by such equipment.
[0404] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be implemented using non-transitory storage media such as digital storage media, such as floppy disks, hard disk drives (HDD), DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs or flash memories, on which are stored electrically readable control signals that cooperate (or are capable of cooperating) with a programmable computer system so that the corresponding method is performed. Thus, the digital storage medium may be computer readable.
[0405] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
[0406] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may be stored, for example, on a machine readable carrier.
[0407] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
[0408] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
[0409] A further embodiment of the method of the present invention is therefore a data carrier (or a digital storage medium or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.
[0410] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may be configured to be transmitted, for example, via a data communication connection, for example via the Internet.
[0411] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.
[0412] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
[0413] According to another embodiment of the present invention, a device or system is configured to transmit (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The device or system may include, for example, a file server for transmitting the computer program to the receiver.
[0414] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, preferably, these methods are performed by any hardware device.
[0415] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein are obvious to other persons skilled in the art. Therefore, the invention is not limited to the specific details given by the description of the embodiments herein.
[0416] Citation list
[0417] [1] Dietz, L. Liljeryd, K. and O. Kunz, “Spectral Band Replication, a novel approach in audio coding”, 112th AES (Audio Engineering Society) Conference, Munich, May 2002
[0418] [2] Ferreira, D. Sinha, “Accurate Spectral Replacement,” Audio Engineering Society Conference, Barcelona, Spain, 2005
[0419] [3] D. Sinha, A. Ferreira1, and E. Harinarayanan, “A Novel Integrated Audio Bandwidth Extension Toolkit (ABET)”, Audio Engineering Society Conference, Paris, France, 2006
[0420] [4] R. Annadana, E. Harinarayanan, A. Ferreira, and D. Sinha, “New Results in Low Bit Rate Speech Coding and Bandwidth Extension,” Audio Engineering Society Conference, San Francisco, USA, 2006
[0421] [5]T. M.Bartkowiak, “Audio bandwidth extension by frequenciescaling of sinusoidal partials”, Audio Engineering Society Conference, San Francisco, USA, 2008
[0422] [6] J. Herre, D. Schulz, Extending the MPEG-4AAC Codec by Perceptual NoiseSubstitution, 104th AES (Audio Engineering Society) Conference, Amsterdam, 1998, preprint 4720
[0423] [7] M. Neuendorf, M. Multrus, N. Rettelbach et al., MPEG Unified Speech and Audio Coding - The ISO / MPEG Standard for High-Efficiency Audio Coding of all Content Types, 132nd AES (Audio Engineering Society) Conference, Budapest, Hungary, April 2012
[0424] [8] McAulay, Robert J., Quatieri, and Thomas F., “Speech Analysis / Synthesis Based on a Sinusoidal Representation,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 34(4), August 1986
[0425] [9] Smith, J. O. and Serra, X., “PARSHL: An analysis / synthesis program for non-harmonic sounds based on a sinusoidal representation”, Proceedings of the International Conference on Computer Music, 1987
[0426]
[10] Purnhagen, H.; Meine, Nikolaus, “HILN - the MPEG-4 parametric audiocoding tools”, Circuits and Systems, Proceedings of the 2000 ISCAS (International Symposium on Circuits and Systems), Geneva, 2000, 2000 IEEE International Symposium on Circuits and Systems, Vol. 3, pp. 201, 204, Vol. 3, 2000
[0427]
[11] International Standard ISO / IEC 13818-3, “Generic Coding of Moving Pictures and Associated Audio: Audio”, Geneva, 1998
[0428]
[12] M. Bosi, K. Brandenburg, S. Quackenbush, L. Fielder, K. Akagiri, H. Fuchs, M. Dietz, J. Herre, G. Davidson, Oikawa, “MPEG-2 Advanced Audio Coding”, 101st AES (Audio Engineering Society) Conference, Los Angeles, 1996
[0429]
[13] J. Herre, “Temporal Noise Shaping, Quantization and Coding methods in Perceptual Audio Coding: A Tutorial introduction,” 17th AES (Audio Engineering Society) International Conference on High-Quality Audio Coding, August 1999
[0430]
[14] J. Herre, “Temporal Noise Shaping, Quantization and Coding methods in Perceptual Audio Coding: A Tutorial introduction”, 17th AES (Audio Engineering Society) International Conference on High-Quality Audio Coding, August 1999
[0431]
[15] International Standard ISO / IEC 23001-3:2010, Unified speech and audio coding Audio, Geneva, 2010
[0432]
[16] International Standard ISO / IEC 14496-3:2005, Information technology-Coding of audio-visual objects-Part 3: Audio, Geneva, 2005
[0433]
[17] P. Ekstrand, “Bandwidth Extension of Audio Signals by Spectral Band Replication,” in Proceedings of the 1st IEEE Benelux Workshop on MPCA, Leuven, November 2002
[0434]
[18] F. Nagel, S. Disch, S. Wilde, A continuous modulated single sideband bandwidth extension, ICASSP International Conference on Acoustics, Speech and Signal Processing, Dallas, Texas (USA), April 2010
[0435] Invention Concept
[0436] The present invention provides the following inventive concepts:
[0437] 1. A device for decoding a coded signal, comprising:
[0438] an audio decoder (1102) for decoding the encoded representation of the first group of first spectral portions to obtain a decoded first group of first spectral portions (1101);
[0439] a parameterized decoder (1104) for decoding the encoded parameterized representation of the second set of second spectral portions to obtain a decoded representation (1103) of the parameterized representation, wherein the parameterized information comprises, for each target frequency tile, a source region identification as matching information; and
[0440] A frequency regenerator (1106) for regenerating a target frequency tile using a source region from the first set of first frequency spectrum portions (1101) identified by the matching information.
[0441] 2. The device according to inventive concept 1,
[0442] wherein the audio decoder (1102) is a spectral domain audio decoder, and
[0443] The device further comprises a spectrum-time converter (1108), and the spectrum-time converter (1108) is used to convert the spectrum representation of the first spectrum part and the reconstructed second spectrum part into a time representation.
[0444] 3. The device according to one of the inventive concepts 1 or 2,
[0445] The frequency regenerator (1106) includes a controllable whitening filter (1122, 1123),
[0446] wherein the parameterization information (1103) includes whitening information (1123), and
[0447] Wherein, the frequency regenerator (1106) is configured to: when the whitening information of the source region selected according to the matching information indicates that the source region is to be whitened, apply the whitening filter (1122) to the source region before performing the spectrum envelope adjustment (1128).
[0448] 4. The device according to inventive concept 3,
[0449] wherein the whitening information comprises, for a tile or a group of tiles, whitening level information indicating a whitening level to be applied to a source frequency tile when regenerating the target frequency tile, and
[0450] Therein, the frequency regenerator (1106) is configured to: apply a whitening filter selected from a group of different whitening filters in response to the whitening information.
[0451] 5. The device according to one of the preceding inventive concepts,
[0452] Wherein, the frequency regenerator (1106) comprises a source region modifier,
[0453] The parameterized information includes not only the source area identifier but also symbol information, and
[0454] The source region modifier is configured to: apply an operation of obtaining a phase shift of a source region spectrum value according to the symbol information.
[0455] 6. The device according to one of the preceding inventive concepts,
[0456] The frequency regenerator (1106) includes a tile modulator (1120),
[0457] The parameterized information (1103) includes, in addition to the source region identifier (1121), a correlation lag, and
[0458] Wherein the tile modulator is configured to apply tile modulation according to the correlation lag associated with the source region identifier (1121).
[0459] 7. The device according to one of the preceding inventive concepts,
[0460] The frequency regenerator (1106) includes a tile modulator (1120),
[0461] The parameterized information (1103) includes, in addition to the source region identifier (1121), a correlation lag, and
[0462] Wherein, the tile modulator (1120) is configured to: when the correlation lag is an odd number, apply a tile modulation using an alternating time sequence of -1 / 1.
[0463] 8. A device for encoding an audio signal, comprising:
[0464] A time-to-spectral converter (1130) for converting an audio signal into a spectral representation;
[0465] a spectrum analyzer for analyzing the spectral representation to determine a first set of first spectral portions to be encoded at a first spectral resolution and a second set of second spectral portions to be encoded at a second spectral resolution, wherein the second spectral resolution is lower than the first spectral resolution;
[0466] a parameter calculator (1134), the parameter calculator (1134) being used to calculate a similarity between a predetermined source region and a target region, the source region comprising a frequency spectrum portion, and the target region comprising a second frequency spectrum portion,
[0467] wherein the parameter calculator (1134) is configured to: compare matching results of different pairs of the first spectrum portion and the second spectrum portion to determine a selected matching pair, and provide matching information identifying the matching pair; and
[0468] A core encoder (1136), the core encoder (1136) is used to encode the first group of first spectrum parts, wherein the first group of first spectrum parts includes the predetermined source area and spectrum parts different from the predetermined source area.
[0469] 9. The device according to inventive concept 8,
[0470] The parameter calculator (1134) is configured to use a predetermined target region in the second group of second spectrum parts or a predetermined source region in the first group of first spectrum parts.
[0471] 10. The device according to inventive concept 8,
[0472] wherein the parameter calculator (1134) is configured such that: the predetermined target area is non-overlapping, or the predetermined source area is overlapping, or wherein the predetermined source area is a subset of the first group of first spectrum portions below a gap filling start frequency (309), or wherein the predetermined target area covers a lowest spectrum area coinciding with the gap filling start frequency (309).
[0473] 11. The device according to one of inventive concepts 8, 9 or 10,
[0474] Wherein, the parameter calculator (1134) is configured to compare pairs of target regions and source regions and pairs of the target regions and the same source region, wherein the same source region is shifted by a correlation lag to provide information about the correlation lag of the selected pair as additional matching information.
[0475] 12. The audio encoder according to any one of inventive concepts 8 to 11,
[0476] wherein the parameter calculator (1134) is configured to: perform correlation processing to obtain a matching result for a pair of the first spectrum portion and the second spectrum portion, the matching result having a negative sign, and
[0477] Wherein, the parameter calculator (1134) is configured to provide information about the negative sign as additional matching information.
[0478] 13. The audio encoder according to any one of inventive concepts 8 to 12,
[0479] wherein the parameter calculator (1134) is configured to: determine an integer number of target tiles and determine a plurality of source tiles of equal size for each target tile, or
[0480] The parameter calculator is configured to: when the target tile is larger than the source tile, copy the source tile to the target tile multiple times until the target tile is filled with multiple copies of the source tile.
[0481] 14. The audio encoder according to any one of inventive concepts 8 to 13,
[0482] The parameter calculator (1134) is configured to spectrally whiten (1142) the first spectral portion or the second spectral portion of the pair before performing correlation processing to obtain a matching signature.
[0483] 15. The device according to one of the inventive concepts 8 to 14,
[0484] The parameter calculator (1134) is configured to calculate the spectrum envelope of the first spectrum portion or the second spectrum portion using at least one of the following processes:
[0485] transforming the spectrum using a discrete cosine transform (DCT), preserving lower frequency DCT coefficients by setting higher DCT coefficients to zero, and computing an inverse DCT;
[0486] Compute the spectral envelope of a set of linear prediction coefficients computed over a time-domain audio frame; or
[0487] The modified discrete cosine transform (power) spectrum is filtered using a low-pass filter.
[0488] 16. The apparatus according to one of inventive concepts 8 to 15, further comprising: a source tile pruning operation (1146) and a memory (1148) for storing source tile information of an earlier frame prior to the current frame.
[0489] 17. The device according to one of the inventive concepts 8 to 16,
[0490] The source pruning operation (1146) includes analyzing the plurality of source tiles with respect to their similarity and removing source tiles having a similarity with a different source tile greater than a predetermined threshold from a set of potential tiles used for cross-correlation calculation (1140).
[0491] 18. The device according to one of the inventive concepts 12 to 17,
[0492] The parameter calculator is configured to retain a set of matching information for each target region from the previous frame when none of the source regions in the current frame is related to the target region of the current frame and is better than a predetermined threshold relative to the previous frame.
[0493] 19. A method for decoding an encoded signal, comprising:
[0494] decoding (1102) the encoded representation of the first set of first spectral portions to obtain a decoded first set of first spectral portions (1101),
[0495] decoding (1104) the encoded parameterized representation of the second set of second spectral portions to obtain a decoded representation (1103) of the parameterized representation, wherein the parameterized information comprises, for each target frequency tile, a source region identification as matching information; and
[0496] A target frequency tile is regenerated (1106) using a source region from the first set of first frequency spectrum portions (1101) identified by the matching information.
[0497] 20. A method for encoding an audio signal, comprising:
[0498] converting (1130) the audio signal into a spectral representation;
[0499] analyzing (1132) the spectral representation to determine a first set of first spectral portions to be encoded at a first spectral resolution and a second set of second spectral portions to be encoded at a second spectral resolution, wherein the second spectral resolution is lower than the first spectral resolution;
[0500] calculating (1134) a similarity between a predetermined source region and a target region, the source region including the spectral portion and the target region including the second spectral portion,
[0501] wherein the calculating (1134) comprises: comparing matching results of different pairs of the first spectral portion and the second spectral portion to determine a selected matching pair, and providing matching information identifying the matching pair; and
[0502] The first set of first spectral portions is encoded (1136), wherein the first set of first spectral portions includes the predetermined source region and spectral portions different from the predetermined source region.
[0503] 21. A computer program for executing any of the methods according to inventive concept 19 or 20 when running on a computer or a processor.
Claims
1. A device for decoding a coded signal, comprising: an audio decoder (1102) for decoding an encoded representation of a first set of first spectral portions of the encoded signal to obtain a decoded first set of first spectral portions (1101); a parameterized decoder (1104) for decoding the encoded parameterized representation of the second set of second spectral portions of the encoded signal to obtain a decoded representation of the encoded parameterized representation, wherein the decoded representation of the encoded parameterized representation comprises, for each target frequency tile, a source region identification (1121) as matching information; and a frequency regenerator (1106) for regenerating a target frequency tile using a source region from the first set of first frequency spectrum portions (1101) identified by the matching information, wherein the frequency regenerator (1106) is configured to apply a whitening filter (1122) to the source region, wherein the frequency regenerator (1106) is configured to: when applying the whitening filter (1122), calculate a spectral envelope estimate of the source region and divide the spectrum of the source region by the spectral envelope indicated by the spectral envelope estimate.
2. The device according to claim 1, wherein: The audio decoder (1102) is a spectral domain audio decoder, and The device further comprises a spectrum-time converter (1108), and the spectrum-time converter (1108) is used to convert the spectrum representation of the first spectrum part and the reconstructed second spectrum part into a time representation.
3. The device according to claim 1, wherein: A first spectral portion (306) of the first set of first spectral portions is surrounded by two second spectral portions (307A, 307B) of the second set of second spectral portions.
4. The device according to claim 1, in, The frequency regenerator (1106) comprises a source region modifier, wherein the decoded representation of the encoded parameterized representation includes, in addition to the source region identifier (1121), symbol information, and The source region modifier is configured to: apply an operation of obtaining a phase shift of a source region spectrum value according to the symbol information.
5. The device according to claim 1, in, The frequency regenerator (1106) includes a tile modulator (1120), wherein the decoded representation of the encoded parameterized representation includes, in addition to the source region identifier (1121), a correlation lag, and Wherein the tile modulator is configured to apply tile modulation according to the correlation lag associated with the source region identifier (1121).
6. The device according to claim 1, in, The frequency regenerator (1106) includes a tile modulator (1120), wherein the decoded representation of the encoded parameterized representation includes, in addition to the source region identifier (1121), a correlation lag, and Wherein, the tile modulator (1120) is configured to: when the correlation lag is an odd number, apply a tile modulation using an alternating time sequence of -1 / 1.
7. A device for encoding an audio signal, comprising: a time-to-spectrum converter (1130), the time-to-spectrum converter (1130) being used to convert the audio signal into a spectral representation; a spectrum analyzer for analyzing the spectral representation to determine a first group of first spectral portions to be encoded at a first spectral resolution and a second group of second spectral portions to be encoded at a second spectral resolution, wherein the second spectral resolution is lower than the first spectral resolution, and wherein a first spectral portion (306) of the first group of first spectral portions is surrounded by two second spectral portions (307A, 307B) of the second group of second spectral portions; a parameter calculator (1134), the parameter calculator (1134) being used to calculate a similarity between a predetermined source region and a target region, the source region comprising a frequency spectrum portion, and the target region comprising a second frequency spectrum portion, wherein the parameter calculator (1134) is configured to: compare matching results of different pairs of the first spectral portion and the second spectral portion to determine a selected matching pair, and provide matching information identifying the matching pair, wherein the parameter calculator (1134) is configured to: spectrally whiten (1142) the first spectral portion or the second spectral portion in the different pairs before performing correlation processing to obtain the matching information, wherein the spectrally whitening (1142) comprises: calculating a spectral envelope estimate of the first spectral portion or the second spectral portion and dividing the spectrum of the first spectral portion or the second spectral portion, respectively, by a spectral envelope indicated by the spectral envelope estimate; and A core encoder (1136), the core encoder (1136) is used to encode the first group of first spectrum parts, wherein the first group of first spectrum parts includes the predetermined source area and spectrum parts different from the predetermined source area.
8. The device according to claim 7, wherein: The parameter calculator (1134) is configured to use a predetermined target region in the second set of second spectral portions or a predetermined source region in the first set of first spectral portions.
9. The device according to claim 8, wherein: The parameter calculator (1134) is configured such that: the predetermined target area is non-overlapping, or the predetermined source area is overlapping, or wherein the predetermined source area is a subset of the first group of first spectrum parts below the gap filling start frequency (309), or wherein the predetermined target area covers the lowest spectrum area coinciding with the gap filling start frequency (309).
10. The device according to claim 7, in, The parameter calculator (1134) is configured to compare pairs of a target region and a source region and pairs of the target region and the same source region, wherein the same source region is shifted by a correlation lag to provide information about the correlation lag of the selected pair as additional matching information.
11. The device according to claim 7, in, The parameter calculator (1134) is configured to: perform a correlation process to obtain a matching result for a pair of the first spectrum portion and the second spectrum portion, the matching result having a negative sign, and Wherein, the parameter calculator (1134) is configured to provide information about the negative sign as additional matching information.
12. The device according to claim 7, in, The parameter calculator (1134) is configured to determine an integer number of target tiles and to determine a plurality of equally sized source tiles for each target tile.
13. The device according to claim 7, in, The parameter calculator (1134) is configured to calculate the spectral envelope of the first spectral portion or the second spectral portion using at least one of the following processes: transforming the spectrum of the first spectral portion or the second spectral portion using a discrete cosine transform (DCT), preserving lower frequency DCT coefficients by setting higher DCT coefficients to zero, and computing an inverse DCT; Calculating the spectral envelope of a set of linear prediction coefficients calculated over a time domain audio frame; or The modified discrete cosine transform or power spectrum is filtered using a low pass filter.
14. The apparatus according to claim 7, further comprising: Source tiling trimming operations (1146) and memory (1148) for storing source tiling information for earlier frames preceding the current frame.
15. The device according to claim 7, in, The source pruning operation (1146) includes analyzing a plurality of source tiles with respect to their similarity and removing source tiles from a set of potential tiles used for cross-correlation calculation (1140) that have a similarity to a different source tile greater than a predetermined pruning threshold.
16. The device according to claim 11, in, The parameter calculator (1134) is configured to retain a set of matching information for each target region from a previous frame when none of the source regions in the current frame correlates with the target region of the current frame better than a predetermined threshold relative to a previous frame.
17. A method for decoding an encoded signal, comprising: decoding (1102) the encoded representation of the first set of first spectral portions of the encoded signal to obtain a decoded first set of first spectral portions (1101), decoding (1104) the encoded parameterized representation of the second set of second spectral portions to obtain a decoded representation of the encoded parameterized representation, wherein the decoded representation of the encoded parameterized representation comprises, for each target frequency tile, a source region identification as matching information; and regenerating (1106) a target frequency tile using a source region from said first set of first frequency spectrum portions (1101) identified by said matching information, wherein the regeneration (1106) comprises applying a whitening filter (1122) to the source region, wherein the regeneration (1106) is configured to: when applying the whitening filter (1122), calculate a spectral envelope estimate of the source region and divide the spectrum of the source region by the spectral envelope indicated by the spectral envelope estimate.
18. A method for encoding an audio signal, comprising: converting (1130) the audio signal into a spectral representation; analyzing (1132) the spectral representation to determine a first group of first spectral portions to be encoded at a first spectral resolution and a second group of second spectral portions to be encoded at a second spectral resolution, wherein the second spectral resolution is lower than the first spectral resolution, and wherein a first spectral portion (306) of the first group of first spectral portions is surrounded by two second spectral portions (307A, 307B) of the second group of second spectral portions; calculating (1134) a similarity between a predetermined source region and a target region, the source region including the spectral portion and the target region including the second spectral portion, wherein the calculation (1134) comprises: comparing the matching results of different pairs of the first spectral portion and the second spectral portion to determine the selected matching pair, and providing matching information identifying the selected matching pair, wherein the calculation (1134) comprises: spectrally whitening (1142) the first spectral portion or the second spectral portion in the different pairs before performing correlation processing to obtain the matching information, wherein the spectrally whitening (1142) comprises: calculating a spectral envelope estimate of the first spectral portion or the second spectral portion and dividing the spectrum of the first spectral portion or the second spectral portion, respectively, by the spectral envelope indicated by the spectral envelope estimate; and The first set of first spectral portions is encoded (1136), wherein the first set of first spectral portions includes the predetermined source region and spectral portions different from the predetermined source region.
19. A storage medium having stored thereon a computer program for performing any of the methods according to claim 17 or 18 when run on a computer or a processor.
