Method and apparatus for unified time / frequency domain coding of sound signals

Abstract

A unified time / frequency domain coding method and apparatus for encoding an input sound signal includes a classifier that classifies the input sound signal into one of a plurality of sound signal categories, including an unclear signal type category that represents an unclear nature of the input sound signal. One of a plurality of encoding sub-modes for encoding the input sound signal in the event that the input sound signal is classified into the unclear signal type category is selected. A hybrid time / frequency domain encoder encodes the input sound signal using the selected encoding sub-mode. The hybrid time / frequency domain encoder includes a frequency band selector and a bit allocator for selecting frequency bands to be quantized and for allocating a bit budget available for quantization among the selected frequency bands. A corresponding sound signal decoder and decoding method are also provided.

Description

Technical Field

[0001] This disclosure relates to a unified time-domain / frequency-domain coding apparatus and method for encoding input audio signals using a hybrid time-domain and frequency-domain coding mode, as well as a corresponding decoder apparatus and decoding method.

[0002] In this disclosure and the appended claims:

[0003] The term "sound" can refer to speech, general audio signals such as music and reverberant speech, and any other sound. Background Technology

[0004] Existing session codecs can represent clean speech signals at bit rates of approximately 8 kbps with very good quality and are nearly transparent at bit rates of 16 kbps. However, at bit rates below 16 kbps, low-processing-latency session codecs (which most often encode input speech signals in the time domain) are unsuitable for general audio signals such as music and reverberant speech. To overcome this shortcoming, switching codecs have been introduced, essentially using a time-domain approach to encode speech-dominant input sound signals and a frequency-domain approach to encode general audio signals. However, this switching solution typically requires longer processing latency due to speech-music classification and the computation of the frequency-domain transform.

[0005] To overcome the aforementioned drawbacks associated with longer processing latency, a more unified time-domain and frequency-domain coding model has been proposed in U.S. Patent No. 9,015,038 (see Reference [1], the entire contents of which are incorporated herein by reference). This unified time-domain and frequency-domain coding model is part of the EVS (Enhanced Voice Services) sound codec standardized by 3GPP (3rd Generation Partnership Project) as described in Reference [2], the entire contents of which are incorporated herein by reference. In recent years, 3GPP has begun working on developing a 3D (three-dimensional) sound codec based on the EVS codec for immersive services known as IVAS (Immersive Voice and Audio Services) (see Reference [3], the entire contents of which are incorporated herein by reference).

[0006] To make the coding model even more effective for certain types of signals, coding schemes have been added to efficiently allocate available bits between the time and frequency domains and between low and high frequencies. The additional coding schemes are triggered by a new speech / music classifier whose output allows for ambiguous categories of signals that cannot be clearly classified as music or speech (see reference [4], the entire contents of which are incorporated herein by reference). Summary of the Invention

[0007] This disclosure relates to a unified time-domain / frequency-domain coding method for encoding an input audio signal. The method includes: classifying the input audio signal into one of a plurality of audio signal categories, wherein the audio signal categories include an ambiguous signal type category indicating that the nature of the input audio signal is unclear; selecting one of a plurality of coding sub-modes for encoding the input audio signal when it is classified into the ambiguous signal type category; and performing hybrid time-domain / frequency-domain coding on the input audio signal using the selected coding sub-mode.

[0008] This disclosure also relates to a unified time-domain / frequency-domain coding method for encoding an input audio signal, comprising: classifying the input audio signal into one of a plurality of audio signal categories, wherein the audio signal categories include an ambiguous signal type category indicating that the nature of the input audio signal is unclear; and performing hybrid time-domain / frequency-domain coding on the input audio signal in response to the input audio signal being classified into the ambiguous signal type category. Hybrid time-domain / frequency-domain coding of the input audio signal includes band selection and bit allocation, for selecting a frequency band to be quantized and for allocating a bit budget available for quantization among the selected frequency bands.

[0009] According to this disclosure, a unified time-domain / frequency-domain coding apparatus for encoding an input sound signal is also provided, comprising: a classifier for classifying the input sound signal into one of a plurality of sound signal categories, wherein the sound signal categories include an ambiguous signal type category indicating that the nature of the input sound signal is unclear; a selector for encoding one of a plurality of coding sub-modes when the input sound signal is classified into the ambiguous signal type category; and a hybrid time-domain / frequency-domain encoder for encoding the input sound signal using the selected coding sub-mode.

[0010] This disclosure also relates to a unified time-domain / frequency-domain coding apparatus for encoding an input audio signal, comprising: a classifier for classifying the input audio signal into one of a plurality of audio signal categories, wherein the audio signal categories include an ambiguous signal type category indicating that the nature of the input audio signal is unclear; and a hybrid time-domain / frequency-domain encoder for encoding the input audio signal in response to the input audio signal being classified into an ambiguous signal type category. The hybrid time-domain / frequency-domain encoder includes a frequency band selector and a bit allocator for selecting a frequency band to be quantized and for allocating a bit budget available for quantization among the selected frequency bands.

[0011] This invention provides a method for decoding a sound signal, comprising: receiving a bitstream conveying information for reconstructing a mixed time-domain / frequency-domain excitation, the mixed time-domain / frequency-domain excitation representing a sound signal classified as an unclear signal type category, the unclear signal type category indicating that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the sound signal classified as an unclear signal type category; reconstructing the mixed time-domain / frequency-domain excitation in response to the information including the coding sub-patterns for encoding the input sound signal transmitted in the bitstream; converting the mixed time-domain / frequency-domain excitation to the time domain; and filtering the converted mixed time-domain / frequency-domain excitation to the time domain using a synthesis filter to generate a synthesized version of the sound signal.

[0012] This disclosure proposes a method for decoding an audio signal, comprising: receiving a bitstream conveying information available for reconstructing a hybrid time-domain / frequency-domain excitation, the hybrid time-domain / frequency-domain excitation representing an audio signal, the audio signal being (a) classified into an ambiguous signal type category, the ambiguous signal type category indicating that the nature of the audio signal is ambiguous, and (b) encoding it using (i) a frequency band selected for quantization and (ii) a bit budget available for quantization allocated between the frequency bands; reconstructing the hybrid time-domain / frequency-domain excitation in response to the information transmitted in the bitstream, wherein reconstructing the hybrid time-domain / frequency-domain excitation includes selecting the frequency band for quantization and allocating the bit budget available for quantization between the frequency bands; converting the hybrid time-domain / frequency-domain excitation to the time domain; and filtering the converted hybrid time-domain / frequency-domain excitation to the time domain using a synthesis filter to produce a synthesized version of the audio signal.

[0013] According to this disclosure, a sound signal decoder is provided, comprising: a receiver of a bitstream transmitting information for reconstructing a mixed time-domain / frequency-domain excitation, the mixed time-domain / frequency-domain excitation representing a sound signal classified as an unclear signal type category, the unclear signal type category indicating that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the sound signal classified as an unclear signal type category; a reconstructor responsive to the information in the bitstream including the coding sub-patterns for encoding an input sound signal; a converter for converting the mixed time-domain / frequency-domain excitation to the time domain; and a synthesis filter for filtering the converted mixed time-domain / frequency-domain excitation to the time domain to produce a synthesized version of the sound signal.

[0014] This disclosure also relates to an audio signal decoder, comprising: a receiver of a bitstream transmitting information for reconstructing a mixed time-domain / frequency-domain excitation, the mixed time-domain / frequency-domain excitation representing an audio signal, the audio signal being (a) classified into an ambiguous signal type category, the ambiguous signal type category indicating that the nature of the audio signal is ambiguous, and (b) encoded using (i) a frequency band selected for quantization and (ii) a bit budget allocated between the frequency bands for quantization; a reconstructor of the mixed time-domain / frequency-domain excitation in response to the information transmitted in the bitstream, wherein the reconstructor selects the frequency band for quantization and the allocation of the bit budget available for quantization between the frequency bands; a converter for converting the mixed time-domain / frequency-domain excitation to the time domain; and a synthesis filter for filtering the converted mixed time-domain / frequency-domain excitation to the time domain to produce a synthesized version of the audio signal.

[0015] The foregoing and other features will become more apparent from the following non-limiting description of illustrative embodiments of a unified time-domain / frequency-domain coding method, a unified time-domain / frequency-domain coding apparatus, a decoding method, and a decoder apparatus, given only as examples with reference to the accompanying drawings. Attached Figure Description

[0016] In the attached diagram:

[0017] Figure 1 It is a schematic block diagram that simultaneously illustrates the unified time-domain / frequency-domain CELP (code-excited linear prediction) coding method and the corresponding unified time-domain / frequency-domain CELP coding device (e.g., ACELP (algebraic code-excited linear prediction) coding method and device);

[0018] Figure 2 yes Figure 1 A schematic block diagram of a more detailed structure of a unified time-domain / frequency-domain coding method and device, wherein a preprocessor performs first-level analysis to classify the input audio signal;

[0019] Figure 3 It is a schematic block diagram that simultaneously shows the calculator for the cut-off frequency of the time-domain excitation contribution and an overview of the corresponding operation for estimating the cut-off frequency;

[0020] Figure 4 It is shown Figure 3 A schematic block diagram showing the calculator for the cutoff frequency and the corresponding operation for estimating the cutoff frequency in more detail;

[0021] Figure 5 It is a schematic block diagram that simultaneously shows an overview of the frequency quantizer and the corresponding frequency quantization operation;

[0022] Figure 6 yes Figure 5A schematic block diagram of the frequency quantizer and frequency quantization operation in more detail;

[0023] Figure 7 This is a schematic block diagram that simultaneously illustrates the unified time-domain / frequency-domain CELP coding method and the corresponding alternative implementations of the unified time-domain / frequency-domain CELP coding device;

[0024] Figure 8 It is a schematic block diagram that simultaneously shows the operation of selecting an encoding sub-mode and the corresponding sub-mode selector;

[0025] Figure 9 This is a schematic block diagram illustrating both a band selector and a bit allocator, as well as the corresponding operations of band selection and bit allocation, used in... Figure 7 and Figure 8 In an alternative implementation, the available bit budget is allocated to the frequency domain coding mode when the input sound signal is neither classified as speech nor music.

[0026] Figure 10 It is a simplified block diagram of an example configuration of hardware components for forming a unified time-domain / frequency-domain coding device and method for encoding input audio signals;

[0027] Figure 11 It simultaneously shows the use for processing data from... Figure 7 A schematic block diagram of a decoder device 1100 and a corresponding decoding method 1150 for decoding bitstreams of a unified time-domain / frequency-domain coding device and a corresponding unified time-domain / frequency-domain coding method; and

[0028] Figure 12 This is a schematic block diagram showing both a sound signal decoder and a corresponding sound signal decoding method. The sound signal decoder and the corresponding sound signal decoding method are used to decode bitstreams from a unified time-domain / frequency-domain coding device and a corresponding unified time-domain / frequency-domain coding method when the sound signal is classified as having an unclear signal type category. Detailed Implementation

[0029] This disclosure proposes a unified time-domain and frequency-domain coding model that improves the synthesis quality of general audio signals (e.g., music and / or reverberant speech) without increasing processing latency and bit rate. The unified time-domain and frequency-domain coding model includes:

[0030] - A time-domain coding scheme operating in the linear prediction (LP) residual domain, where available bits are dynamically allocated among an adaptive codebook, one or more fixed codebooks (e.g., algebraic codebooks, Gaussian codebooks, etc.), and a variable-length fixed codebook; and

[0031] -Frequency domain coding mode

[0032] It depends on the characteristics of the input sound signal.

[0033] To achieve low processing latency and low bit rate session sound codecs (which improve the synthesis quality of general audio signals such as music and / or reverberant speech), the frequency domain coding scheme is integrated as closely as possible to the CELP (Code-Excited Linear Prediction) time domain coding scheme. For this purpose, the frequency domain coding scheme uses a frequency transformation performed in the LP (Linear Prediction) residual domain. This allows switching from one frame (e.g., a 20ms frame) to another with almost no artifacts. As is well known in the field of sound codecs, the input audio signal is sampled at a given sampling rate and processed by groups of these samples called “frames,” which are typically divided into multiple “subframes.” Here, the integrals of the two (2) time-domain and frequency-domain coding schemes are close enough to allow the bit budget to be dynamically reallocated to another coding scheme if it is determined that the current coding scheme is not efficient enough.

[0034] A feature of the proposed unified time-domain and frequency-domain coding model is the variable-time support for the time-domain components, which varies from quarter-frames (subframes) to full frames on a frame-by-frame basis. As a non-limiting illustrative example, a frame can represent a 20ms input audio signal. If the audio codec's internal sampling rate is 16kHz, such a frame corresponds to 320 samples of the input audio signal, or if the codec's internal sampling rate is 12.8kHz, such a frame corresponds to 256 samples per frame. Subframes (in this example, a quarter of a frame) then represent either 80 or 64 samples, depending on the audio codec's internal sampling rate. In this non-limiting illustrative embodiment, the audio codec's internal sampling rate is 12.8kHz, which gives an input audio signal with a frame length of 256 samples and a subframe length of 64 samples.

[0035] Variable timing support allows for the capture of major temporal events at a minimum bit rate to create a basic temporal excitation contribution. At very low bit rates, the timing support is typically the entire frame. In this case, the temporal contribution of the excitation consists only of an adaptive codebook; then, the corresponding adaptive codebook (pitch) information and gain are transmitted once per frame. When more bit rate is available, more temporal events can be captured by shortening the timing support and increasing the bit rate allocated to the temporal coding mode. Finally, when the timing support is sufficiently short (less than a quarter of a frame (subframe)) and the available bit rate is sufficiently high, for each subframe, the temporal contribution of the excitation can include an adaptive codebook contribution with a corresponding adaptive codebook gain, a fixed codebook contribution with a corresponding fixed codebook gain, or both an adaptive codebook contribution with a corresponding gain and a fixed codebook contribution. Optionally, for each half of a frame (subframe), an adaptive codebook contribution with a corresponding adaptive codebook gain and a fixed codebook contribution with a corresponding fixed codebook gain can also be transmitted; this has the advantage of not consuming too much bit rate while still being able to encode temporal events. Parameters describing the codebook index and gain are then transmitted for each subframe.

[0036] At low bit rates, conversational sound codecs cannot adequately encode higher frequencies. This leads to a significant reduction in synthesis quality when the input audio signal includes music and / or reverberant speech. To address this issue, a feature is added to calculate the efficiency of the temporal excitation contribution. In some cases, the temporal excitation contribution is not valuable regardless of the input bit rate and time frame support. In these cases, all bits are reallocated to the next step of frequency domain encoding. However, most of the time, the temporal excitation contribution is only valuable at a certain frequency (hereinafter referred to as the "cutoff frequency"). In these cases, the temporal excitation contribution is filtered above the cutoff frequency. The filtering operation allows valuable information encoded with the temporal excitation contribution to be preserved, while worthless information above the cutoff frequency is removed. In a non-limiting illustrative embodiment, filtering is performed in the frequency domain by setting the frequency bin above a certain frequency (cutoff frequency) to zero.

[0037] The combination of variable time support and variable cutoff frequency makes bit allocation within the unified time-domain and frequency-domain coding model highly dynamic. The bit rate after quantization of the LP filter can be allocated entirely to the time domain, entirely to the frequency domain, or somewhere in between. The bit rate allocation between the time and frequency domains is a function of the number of subframes contributed to the time-domain excitation, the available bit budget, and the calculated cutoff frequency. To make the unified time-domain and frequency-domain coding model even more effective for certain types of input audio signals, specific coding sub-patterns are added to efficiently allocate available bits between the time and frequency domains, and between low and high frequencies. These added specific coding sub-patterns are determined using a novel speech / music audio classifier that produces an output that allows for signals of unclear category (signals that cannot be clearly classified as music or speech).

[0038] To create a total excitation that more effectively matches the input LP residuals, a frequency-domain coding mode is applied. One feature is performing frequency-domain coding on a vector containing the difference between the frequency representation (frequency transform) of the input LP residuals and the frequency representation (frequency transform) of the filtered temporal excitation contribution up to the cutoff frequency, and also containing the frequency representation (frequency transform) of the input LP residuals themselves above that cutoff frequency. A smooth spectral transition is inserted only between the two segments above the cutoff frequency. In other words, the high-frequency portion of the frequency representation of the temporal excitation contribution first zeros out above the cutoff frequency. A transition region between the unchanged portion of the spectrum of the temporal excitation contribution and the zero-out portion of the spectrum is inserted just above the cutoff frequency to ensure a smooth transition between the two portions of the spectrum. This modified spectrum of the temporal excitation contribution is then subtracted from the frequency representation of the input LP residuals. Therefore, the resulting spectrum corresponds to the difference between the two spectra below the cutoff frequency and to the frequency representation of the LP residuals above the cutoff frequency, with some transition region. As mentioned above, the cutoff frequency can vary from one frame to another.

[0039] Regardless of the frequency quantization method (frequency domain coding mode) chosen, there is always the possibility of pre-echo, especially in the case of long windows. In the technique disclosed herein, the window used is a square window, such that the additional window length compared to the encoded input audio signal is zero (0), i.e., no overlap addition is used. While this corresponds to the optimal window for reducing any potential pre-echo, some pre-echo can still be audible under time attacks. Many techniques exist to address such pre-echo problems, but this disclosure proposes a simple feature for eliminating this pre-echo problem. This feature is based on a memoryless time domain coding mode derived from the “Transition Mode” of ITU-T Recommendation G.718; see reference [5], sections 6.8.1.4 and 6.8.4.2, the entire contents of which are incorporated herein by reference. The idea behind this feature is to take advantage of the fact that the proposed unified time and frequency domain coding model is integrated into the LP residual domain, which allows for switching with almost no artifacts at any time. When the input audio signal is considered general audio (music and / or reverberant speech) and a temporal attack is detected in the frame, the frame is encoded using only a memoryless temporal coding scheme. This memoryless temporal coding scheme takes into account temporal attacks, thus avoiding pre-echoes that could be introduced when using frequency-domain coding of the frame.

[0040] Non-limiting illustrative examples

[0041] In the proposed unified time-domain and frequency-domain coding model, the aforementioned adaptive codebook, one or more fixed codebooks (e.g., algebraic codebooks, Gaussian codebooks, etc.) (i.e., the so-called time-domain codebook), and frequency-domain quantization (frequency-domain coding mode) can be considered as a codebook library, and bits can be allocated among all available codebooks or subsets thereof. This means that, for example, if the input audio signal is clean speech, all bits will be allocated to the time-domain coding mode, thus essentially reducing the coding to the traditional CELP scheme. On the other hand, for some musical segments, all bits allocated to encoding the input LP residual are sometimes best spent in the frequency domain, such as in the transform domain. Furthermore, specific cases can be added where (a) the time domain uses a larger portion of the total available bit rate to encode more time-domain events, while still reserving bits for encoding some frequency information, or (b) low-frequency content takes precedence over high-frequency content, and vice versa.

[0042] As indicated in the preceding description, the time support for time-domain and frequency-domain coding modes does not need to be the same. While the bits spent on different time-domain coding operations (adaptive and algebraic book search) are typically allocated based on subframes (usually a quarter of a frame, or 5ms of time support), the bits allocated to the frequency-domain coding mode are allocated based on frames (usually 20ms of time support) to improve frequency resolution.

[0043] The bit budget allocated to the time-domain CELP coding mode can also be dynamically controlled based on the input audio signal. In some cases, the bit budget allocated to the time-domain CELP coding mode can be zero, effectively meaning that the entire bit budget is attributed to the frequency-domain coding mode. The choice between the time-domain and frequency-domain coding modes operating in the LP residual domain has two (2) main benefits. First, it is compatible with the time-domain CELP coding mode, proving effective in speech signal coding. Therefore, no artifacts are introduced due to the switching between the two types of coding modes (time-domain and frequency-domain coding modes). Second, the lower dynamics of the LP residual relative to the original input audio signal and its relative flatness make it easier to use a square window for frequency transformation, thus allowing the use of a non-overlapping window.

[0044] In a non-restrictive example where the codec uses a rate of 12.8 kHz (meaning 256 samples per frame), similar to ITU-T Recommendation G.718 (Reference [5]), the length of the subframes used in the time-domain CELP coding mode can vary from a typical 1 / 4 (5 ms) of the frame length to half a frame (10 ms) or a full frame length (20 ms). The subframe length is determined based on the available bit rate and analysis of the input audio signal, particularly the spectral dynamics of the input audio signal. The subframe length can be determined in a closed-loop manner. To save complexity, the subframe length can also be determined in an open-loop manner. The subframe length can also be controlled by the properties of the input audio signal detected by a signal classifier (e.g., a speech / music classifier). The subframe length can be changed frame by frame.

[0045] Once the subframe length is selected in the current frame, standard closed-loop tone analysis is performed, and a first contribution to the excitation signal is selected from the adaptive codebook. Then, depending on the available bit budget and the characteristics of the input audio signal (e.g., in the case of an input speech signal), a second contribution from one or more fixed codebooks can be added before the transformation in the transform domain. The resulting excitation contribution is the time-domain excitation contribution. On the other hand, at very low bit rates and in the case of general audio signals, it is generally better to skip the fixed codebook stage and use all remaining bits for transform-domain coding. The transform-domain coding can be, for example, a frequency-domain coding mode. As mentioned above, the subframe length can be one-quarter, half, or one full frame. The fixed codebook contribution is used only when the subframe length is equal to one-quarter of the frame length. When the subframe length is determined to be half or the full frame length, only the adaptive codebook contribution is used to represent the time-domain excitation contribution, and all remaining bits are allocated to the frequency-domain coding mode. Alternatively, an additional coding mode will be described where the fixed codebook can be used when the subframe length is equal to half the frame length. This addition is made to improve the quality of specific types of input audio signals that contain time events, while maintaining an acceptable bit budget to encode the frequency domain excitation contribution.

[0046] Once the time-domain excitation contribution has been calculated, its efficiency needs to be evaluated and quantified. If the gain of coding in the time domain is very low, it is more efficient to completely remove the time-domain excitation contribution and use all bits for the frequency-domain coding mode. On the other hand, for example, in the case of a clean input speech signal, a frequency-domain coding mode is not needed, and all bits are allocated to the time-domain coding mode. However, coding in the time domain is typically only effective at a certain frequency. This frequency corresponds to the aforementioned cutoff frequency of the time-domain excitation contribution. Determining this cutoff frequency ensures that the entire time-domain coding contributes to a better final synthesis, rather than the opposite of frequency-domain coding.

[0047] The cutoff frequency can be estimated in the frequency domain. To calculate the cutoff frequency, the spectrum of the LP residual and the temporal excitation contribution is first divided into a predetermined number of frequency bands, with multiple frequency segments defined within each band. The number of frequency bands and the number of frequency segments covered by each band can vary from one implementation to another. For each frequency band, a normalized correlation is calculated between the frequency representation of the temporal excitation contribution and the frequency representation of the LP residual, and the correlation between adjacent frequency bands is smoothed. As a non-limiting example, the lower bound for the correlation per band is 0.5 and normalized between 0 and 1, and then the average correlation is calculated as the average of the correlations across all frequency bands. To estimate the cutoff frequency initially, the average correlation is then scaled between 0 and half of the internal sampling rate (half of the internal sampling rate corresponds to a normalized correlation value of 1). At very low bit rates or for additional coded sub-modes as described below, the average correlation is doubled before finding the cutoff frequency. This is done when the temporal excitation contribution is known to be needed even if the correlation is not very high due to the use of a low bit rate or because the type of input audio signal does not allow for high correlation. The first estimate of the cutoff frequency is then found as the upper bound of the band that is closest to the value of the scaled average correlation. In the implemented example, sixteen (16) bands at an internal sampling rate of 12.8 kHz are defined for correlation calculation.

[0048] Leveraging the psychoacoustic properties of the human ear, the reliability of the cutoff frequency estimation can be improved by comparing the estimated position of the eighth harmonic frequency of the pitch with the cutoff frequency estimated through correlation calculations. If the position is higher than the cutoff frequency estimated through correlation calculations, the cutoff frequency is modified to correspond to the position of the eighth harmonic frequency of the pitch. If one of the additional coding sub-modes is used, the cutoff frequency has a minimum value higher than or equal to, for example, 2775 Hz (7th band). The final value of the cutoff frequency is then quantized and sent to a remote decoder. In the implemented example, 3 or 4 bits are used for this quantization, thus giving 8 or 16 possible cutoff frequencies depending on the bit rate.

[0049] Once the cutoff frequency is known, frequency quantization of the frequency-domain excitation contribution is performed. First, the difference between the frequency representation (frequency transform) of the input LP residual and the frequency representation (frequency transform) of the time-domain excitation contribution is determined. Then, a new vector is created that includes this difference up to the cutoff frequency, as well as a smooth transition to the frequency representation of the input LP residual to the remaining spectrum. Frequency quantization is then applied to the entire new vector. In the implemented example, quantization involves encoding the sign and position of the dominant (most energetic) spectral pulses. The number of pulses to be quantized per frequency band is related to the bit rate available for the frequency-domain coding mode. If the available bits are insufficient to cover all frequency bands, the remaining bands are simply filled with noise.

[0050] Frequency quantization of a band using the quantization methods described in the preceding paragraphs does not guarantee that all frequency bands within that band will be quantized. This is especially true at low bit rates where the number of quantized spectral pulses per band is relatively low. To prevent audible artifacts caused by these unquantized frequency bands, some noise is added to fill these gaps. Since at low bit rates, the quantized spectral pulses should dominate the spectrum rather than the inserted noise, the noise spectral amplitude corresponds only to a portion of the pulse amplitude. When the available bit budget is low (allowing for more noise), the amplitude of the noise added to the spectrum is higher, and when the available bit budget is high, the amplitude of the noise added to the spectrum is lower.

[0051] In frequency-domain coding mode, a gain is calculated for each frequency band to match the energy of the unquantized signal with that of the quantized signal. The gain is vector-quantized and applied band-by-band to the quantized signal. For example, when a unified time-domain and frequency-domain coding model changes bit allocation from a time-only coding mode to a hybrid time-domain / frequency-domain coding mode, the per-band excitation spectral energy in the time-only coding mode does not match the per-band excitation spectral energy in the hybrid time-domain / frequency-domain coding mode. This energy mismatch can produce some switching artifacts, especially at low bit rates. To reduce any audible degradation caused by this bit reallocation, a long-term gain can be calculated for each frequency band, and this long-term gain can be applied to correct the energy of each frequency band for several frames after switching from a time-only coding mode to a hybrid time-domain / frequency-domain coding mode.

[0052] After completing the frequency domain coding pattern, the total excitation is found by adding the frequency representations (frequency transformations) of the frequency domain excitation contribution and the time domain excitation contribution. Then, the sum of these two (2) excitation contributions is transformed back to the time domain to form the total excitation. Finally, the total excitation is filtered by an LP synthesis filter to calculate the synthesized signal.

[0053] In one embodiment, although only temporal excitation contributions are used to update the CELP-coded memory on a subframe basis, the total excitation is used to update those memories at frame boundaries.

[0054] In another possible implementation, the CELP-encoded memory is updated at frame boundaries on a subframe basis, using only temporal excitation contributions. This results in an embedded architecture where the frequency-domain coded signal constitutes an upquantization layer independent of the core CELP layer. In this particular case, a fixed codebook is always used to update the adaptive codebook content. However, the frequency-domain coding mode can be applied to the entire frame. This embedded approach is suitable for bit rates of approximately 12 kbps and higher.

[0055] 1) Classification of sound signal types

[0056] Figure 1 This is a schematic block diagram illustrating an overview of both the unified time-domain / frequency-domain CELP coding method 150 and the corresponding unified time-domain / frequency-domain CELP coding device 100 (e.g., the ACELP method and device). Of course, other types of CELP coding methods and devices can be implemented using the same concepts.

[0057] Figure 2 yes Figure 1 A schematic block diagram of the more detailed structure of the unified time-domain / frequency-domain CELP coding method 150 and device 100.

[0058] The unified time-domain / frequency-domain CELP coding device 100 includes a preprocessor 102 ( Figure 1 ), used to perform analysis of input sound signal 101 ( Figure 1 and Figure 2 The operation of the parameters is described in section 152. (Refer to...) Figure 2 The preprocessor 102 includes: an LP analyzer 201 for performing LP analysis of the input audio signal 101, operation 251; a spectrum analyzer 202 for performing spectrum analysis, operation 252; an open-loop tone analyzer 203 for performing open-loop tone analysis, operation 253; and a signal classifier 204 for performing classification of the input audio signal, operation 254. Analyzers 201 and 202, along with associated operations 251 and 252, perform LP and spectrum analysis typically performed in CELP coding, as described, for example, in ITU-T Recommendation G.718, reference [5], sections 6.4 and 6.1.4, and therefore will not be described further in this disclosure.

[0059] The preprocessor 102 performs a first-level analysis to classify the input sound signal 101 between speech and non-speech (general audio (music or reverberant speech)), for example in a manner similar to that described in reference [6], the entire contents of which are incorporated herein by reference, or by utilizing any other reliable speech / non-speech differentiation method.

[0060] Following this first-level analysis, the preprocessor 102 performs a second-level analysis of the input signal parameters to allow time-domain CELP coding (no-frequency-domain coding) to be used for some audio signals that have strong non-speech characteristics but are still better coded using time-domain methods. When a significant energy change occurs, this second-level analysis allows the unified time-domain / frequency-domain CELP coding device 100 to switch to a memoryless time-domain coding mode, commonly referred to as the switching mode in reference [7], the entire contents of which are incorporated herein by reference.

[0061] During this second-level analysis, the signal classifier 204 calculates and uses the smoothed version C of the open-loop pitch correlation from the open-loop pitch analyzer 203. st Changes in σ C Current total frame energy E tot (The total energy of the input audio signal in the current frame) and the difference E between the current total frame energy and the previous total frame energy. diff First, the signal classifier 204 uses, for example, the following relationship to calculate the smooth open-loop pitch-related changes:

[0062]

[0063] in:

[0064] -C st It is a smooth open-loop pitch correlation as defined below: C st =0.9·C cl +0.1·C st ;

[0065] -C ol The open-loop tone correlation is calculated by the analyzer 203 using methods known to those skilled in the art of CELP coding, for example, as described in ITU-T Recommendation G.718, Reference [5], Section 6.6;

[0066] - It is a smooth open-loop pitch-related C st The average value over the last 10 frames i;

[0067] -σ C It is a smooth, open-loop pitch-related variation.

[0068] When the signal classifier 204 classifies a frame as non-speech during the first-level analysis, it performs the following verification to determine in the second-level analysis whether using the hybrid time-domain / frequency-domain coding mode is indeed safe. However, sometimes it is better to encode the current frame using only the time-domain coding mode, using one of the time-domain methods estimated by the preprocessing function of the time-domain coding mode. Specifically, it may be better to use a memoryless time-domain symbol mode to minimize any possible pre-echoes that can be introduced by the hybrid time-domain / frequency-domain coding mode.

[0069] As a non-limiting implementation for the first verification of whether a hybrid time-domain / frequency-domain coding mode should be used, the signal classifier 204 calculates the difference between the current total frame energy and the previous frame total energy. When the current total frame energy E... tot The difference E between the total energy of the previous frame and the previous frame diff At levels above, for example, 6 dB, this corresponds to a so-called "time attack" in the input audio signal 101. In this case, the speech / non-speech determination and the selected coding mode are rewritten, and a memoryless time-domain coding mode is enforced. More specifically, the Unified Time / Frequency CELP Coding Device 100 includes a time / time-frequency coding selector 103 for performing an operation 153 to select between time-domain-only coding and mixed time / frequency coding. Figure 1 Therefore, the time / time-frequency encoding selector 103 includes: a voice / general audio selector 205. Figure 2 ), used to perform the operation 255 of selecting between speech and general audio to classify the input sound signal 101; time attack detector 208 ( Figure 2 ), for performing operation 258 to detect time attacks in input audio signal 101; and selector 206 ( Figure 2 ), used to perform the operation 256 of selecting the memoryless time-domain coding mode. In other words:

[0070] - In response to the selection of the voice signal by selector 205, a closed-loop CELP encoder 207 is used ( Figure 2 ) to perform CELP encoding of the speech signal 257.

[0071] - In response to selector 205's determination of non-speech signals (general audio) and detector 208's detection of time attacks in the input audio signal 101, selector 206 forces closed-loop CELP encoder 207 ( Figure 2 The input audio signal is encoded using a memoryless time-domain coding scheme.

[0072] Closed-loop CELP encoder 207 formation Figure 1This is only a part of the time-domain encoder 104. Closed-loop CELP encoders are well known to those skilled in the art and will not be described further in this specification.

[0073] As a non-limiting implementation for a second verification of whether a hybrid time-domain / frequency-domain coding mode should be used, when the current total frame energy E tot The difference E between the total energy of the previous frame and the previous frame diff When it is 6dB or less, but:

[0074] - Smooth open-loop pitch correlation C st Higher than 0.96; or

[0075] - Smooth open-loop pitch correlation C st Above 0.85, and the current total frame energy E tot The difference E between the total energy of the previous frame and the previous frame diff Below 0.3dB; or

[0076] - Smooth open-loop pitch correlation σ C The change is less than 0.1, and the current total frame energy E tot The difference E between the total energy of the last previous frame and the total energy of the previous frame diff Below 0.6 dB; or

[0077] - Current total frame energy E tot Less than 20 dB;

[0078] And this is at least the second consecutive frame (cnt≥2), where the decision of the first-level analysis is changed, and then the speech / general audio selector 205 determines that the closed-loop CELP encoder 207 will be used. Figure 2 Encode the current frame using only time-domain coding mode.

[0079] Otherwise, the time / time-frequency coding selector 103 selects a hybrid time-domain / frequency-domain coding mode, as disclosed in the following description.

[0080] For example, when the non-voice input sound signal is music, the following pseudocode can be used to summarize the second verification:

[0081]

[0082] Where E tot This is the current total frame energy, expressed as:

[0083]

[0084] Where x(i) represents the sample of the input audio signal in the current frame, N is the number of samples of the input audio signal per frame, and E diff It is the current total frame energy Etot The difference between the total energy of the last previous frame and the total energy of the previous frame.

[0085] Figure 7 This is a schematic block diagram showing both the unified time-domain / frequency-domain CELP encoding method 750 and the corresponding unified time-domain / frequency-domain CELP encoding device 700, wherein the preprocessor 702 also performs a first-level analysis to classify the input audio signal 101.

[0086] Specifically, the unified time-domain / frequency-domain CELP coding method 750 includes an operation 752 as described in reference [4], preprocessing the input audio signal 101 to obtain parameters required for classifying the input audio signal. To perform operation 752, the hybrid time-domain / frequency-domain CELP coding device 700 includes a preprocessor 702.

[0087] The Unified Time / Frequency CELP Coding Method 750 includes an operation 751 that classifies the input audio signal 101 into speech, music, and unclear signal type categories using parameters from the preprocessor 702 in a manner similar to that described in reference [4], or using any other reliable speech / music and unclear signal type differentiation method. An unclear signal type category indicates that the nature of the input audio signal 101 is unclear, and specifically, the input audio signal 101 is neither classified as speech nor as music. To perform operation 751, the Unified Time / Frequency CELP Coding Apparatus 700 includes an audio signal classifier 701.

[0088] If the sound signal classifier 701 classifies the input sound signal 101 into a music category, the frequency domain encoder 703 performs an operation 753 to encode the input sound signal 101 using frequency domain coding as described, for example, in reference [2]. The frequency domain-coded music signal can then be synthesized in a music synthesis operation 754 performed by the synthesizer 704 to recover the music signal.

[0089] Similarly, if the sound signal classifier 701 classifies the input sound signal 101 into a speech category, the temporal encoder 705 performs an operation 755 to encode the input sound signal 101 using temporal coding as described, for example, in reference [2]. The temporally encoded speech signal can then be synthesized in a synthesis filtering operation 756 performed by the synthesizer 706 to recover the speech signal.

[0090] Therefore, the unified time-domain / frequency-domain coding apparatus 700 and method 750 maximize the performance of time-domain-only coding and frequency-domain-only coding by limiting their use to input audio signals with clear speech characteristics and input audio signals with clear musical characteristics, respectively. This improves the overall quality of all types of input audio signals from low to medium bit rates.

[0091] The coding sub-modes have been designed as part of the unified time-domain and frequency-domain coding model to efficiently encode input sound signals that are not classified as speech or music (the signal type category is unclear). Two (2) bits are used to transmit three (3) coding sub-modes identified by the corresponding sub-mode flags. The fourth sub-mode allows backward interoperability with the traditional unified time-domain and frequency-domain coding model (EVS).

[0092] like Figure 8 As shown, the operation 751 of classifying the input audio signal 101 includes an operation 850 of selecting one of the encoding sub-modes in response to the bit rate available for encoding the input audio signal 101 and the characteristics of the input audio signal that are classified as an unclear signal type category. To perform operation 850, the audio signal classifier 701 includes a sub-mode selector 800.

[0093] The encoded subpattern is determined by the subpattern flag F tfsm To identify. In Figure 8 In the unrestricted implementation, the sub-pattern selector 800 selects the encoded sub-pattern as follows:

[0094] - If (a) the bit rate available for encoding the input audio signal 101 is no higher than 9.2 kbps and (b) the input audio signal 101 is neither classified as speech nor music, then the sub-mode selector 800 selects the aforementioned backward encoding sub-mode (see 803). Then the sub-mode flag F... tfsm Set to "0" (see 802). The selection of the back-encoding mode results in the use of... Figure 1 and Figure 2 (EV) is a traditional unified time-domain and frequency-domain coding model.

[0095] - If (a) the input audio signal 101 is not classified as speech or music by classifier 701, and the available bit rate is high enough to allow for adaptive and fixed codebook encoding and gain, typically meaning a bit rate higher than 9.2 kbps (see 803), (b) the probability that the input audio signal 101 is music (a weighted speech / music decision tending towards music, wdlp(n)) is not greater than “0” (see 804), and (c) there is no possibility of a time attack being detected in the current frame of the input audio signal (the transition counter is not greater than “0”, as described in ITU-T Recommendation G.718, Reference [5], Sections 6.8.1.4 and 6.8.4.2) (see 806), then the submode selector 800 selects the first coded submode. Then the submode flag F tfsmSet to "1" (see 801). Although the input sound signal 101 is neither classified as speech nor music by the classifier 701, the selector 800 detects speech-like characteristics in the input sound signal 101 and selects the first encoded sub-mode (sub-mode flag F). tfsm =1), because CELP is not optimal for encoding this type of audio signal.

[0096] - If (a) the input audio signal 101 is neither classified as speech nor as music by classifier 701, and the available bit rate is high enough to allow for encoding and gain for both adaptive and fixed codebooks, typically meaning a bit rate greater than 9.2 kbps (see 803), (b) the probability that the input audio signal 101 is music (a weighted speech / music decision tending towards music, wdlp(n)) is not greater than “0” (see 804), and (c) the probability of detecting a time attack in the current frame of the input audio signal (a transition counter greater than “0”, as described in ITU-T Recommendation G.718, Reference [5], Sections 6.8.1.4 and 6.8.4.2) (see 806), then the sub-mode selector 800 selects the second coding sub-mode. Then the sub-mode flag F tfsm Set to "2" (see 807). As will be explained in the following description, the second encoded sub-pattern (sub-pattern flag F) tfsm =2) Allocate more bits to the lower part of the spectrum.

[0097] - If (a) the input audio signal 101 is neither classified as speech nor as music by classifier 701, and the available bit rate is high enough to allow at least adaptive codebook encoding and gain, and still has a large number of bits for frequency encoding, typically meaning a bit rate higher than 9.2 kbps (see 803), and (b) the probability that the input audio signal 101 is music (a weighted speech / music decision tending towards music, wdlp(n)) is greater than "0" (see 804), then submode selector 800 selects the third encoded submode. Then the submode flag F... tfsm Set to "3" (see 808). Although the input sound signal 101 is neither classified as speech nor as music by the classifier 701, the selector 800 detects "musical" characteristics in the input sound signal 101 and selects the third encoding sub-mode (sub-mode flag F). tfsm =3). Such sound signal segments are still considered non-musical, but the sub-mode flag F tfsm It is set to "3" (selecting the third encoding sub-mode), which indicates that the sample includes high-frequency or tonal content.

[0098] Reference [4] describes the probability that the input sound signal 101 is speech, music, or something in between. When the classification of speech or music is unclear, if the probability wdlp(n) is greater than 0, the signal is considered to have some musical characteristics. The following shows the threshold where the probability will be high enough to be considered as music or speech.

[0099] Table 1: Probability thresholds for categories that are unclear

[0100]

[0101] Select the encoding sub-pattern (e.g., sub-pattern flag F) tfsm The signal is sent to the remote decoder in the bitstream. The path selected within the decoder depends on the signaling bits included in the bitstream. Once the decoder detects the presence of a frame encoded using hybrid time-domain / frequency-domain coding, it decodes the submode flag F from the bitstream. tfsm If the detected sub-pattern flag F tfsm If the value is "0", then the EVS backward interoperable traditional unified time-domain and frequency-domain coding model will be used to decode the remainder of the bitstream. On the other hand, if the submode flag F... tfsm Unlike "0", submode decoding is followed. The decoder will replicate the process followed by the encoder, particularly the bit allocation between the time and frequency domains and the bit allocation in different frequency bands, as described later in Section 6.2.

[0102] 2) Determination of subframe length

[0103] In a typical CELP, input audio signal samples are processed in frames of 10-30ms, and these frames are divided into subframes for adaptive and fixed codebook analysis. For example, a 20ms frame (256 samples at an internal sampling rate of 12.8kHz) can be used and divided into four 5ms subframes. Variable subframe length is a feature used to integrate the time and frequency domains into a single coding pattern. The subframe length can vary from a typical quarter of the frame length to half or the full frame length. Of course, it is possible to use an additional number of subframes (subframe length).

[0104] like Figure 2 As shown, the parameter analysis operation 152 of the unified time-domain / frequency-domain CELP coding method 150 includes operation 259 for determining the high-spectral dynamics of the input audio signal 101, and operation 260 for calculating the number of subframes frame by frame. To perform operations 259 and 260, the preprocessor 102 of the unified time-domain / frequency-domain CELP coding device 100 includes a high-spectral dynamics analyzer 209 and a subframe count calculator 210, respectively.

[0105] The determination of subframe length (number of subframes) or time support is made by calculator 210 based on the available bit rate and analysis of the input audio signal, particularly the high-spectral dynamics of the input audio signal 101 from analyzer 209 and the smooth open-loop pitch correlation C from analyzer 203. st Open-loop tone analysis. The high-spectral dynamics analyzer 209 responds to information from the spectrum analyzer 202 to determine the high-spectral dynamics of the input audio signal 101. For example, as described in ITU-T Recommendation G.718, Reference [5], Section 6.7.2.2, the high-spectral dynamics are calculated as the input spectrum without a noise floor, thus giving a representation of the input spectral dynamics. When the average spectral dynamics of the input audio signal 101 in the band between 4.4 kHz and 6.4 kHz, as determined by the analyzer 209, is less than, for example, 9.6 dB and the last frame is considered to have high spectral dynamics, the input audio signal 101 is no longer considered to have high spectral dynamics. In this case, more bits can be allocated to frequencies below, for example, 4 kHz by adding more subframes to the time-domain coding mode or by forcing more pulses in the lower frequency portion of the frequency-domain coding mode.

[0106] On the other hand, if the average spectral dynamics of the input audio signal 101 increases by, for example, 4.5 dB relative to the average spectral dynamics of the last frame, which is not considered to have high spectral dynamics as determined by the analyzer 209, then the input audio signal 101 is considered to have high spectral dynamics content above, for example, 4 kHz. In this case, depending on the available bit rate, some additional bits are used to encode the high frequencies of the input audio signal 101 to allow one or more frequency pulses to be encoded.

[0107] For example, from calculator 210 ( Figure 2 The determined subframe length also depends on the bit budget available for encoding the input audio signal 101. At very low bit rates, such as below 9 kbps, only one subframe is available for time-domain coding, otherwise the number of available bits will be insufficient for frequency-domain coding. At medium bit rates, such as between 9 kbps and 16 kbps, one subframe is used for cases where high-frequency content is included, and two subframes are used if no such content is included. For medium-high bit rates, such as around 16 kbps and higher, a four (4) subframe case also becomes available if the smooth open-loop tone correlation Cst defined above is above, for example, 0.8.

[0108] While the case with one or two subframes limits temporal coding to only adaptive codebook contributions (with encoded tone lag and tone gain), i.e., no fixed codebook is used in this case, the case with four (4) subframes allows for both adaptive and fixed codebook contributions if the available bit budget is sufficient. The case with four (4) subframes is allowed at bit rates starting from approximately 16 kbps upwards. Due to bit budget constraints, the temporal excitation contribution consists only of the adaptive codebook contribution at the lower bit rates. The fixed codebook contribution can be added at higher bit rates, for example, starting at 24 kbps. For all cases, the temporal coding efficiency will then be evaluated to determine at which frequency (the aforementioned cutoff frequency) such temporal coding is valuable.

[0109] When the input audio signal 101 is classified as an unclear signal type by classifier 701 and the sub-mode flag F tfsm When greater than zero "0", Figure 7 and Figure 8 Alternative implementations use the first, second, or third encoding sub-patterns defined above.

[0110] The audio signal classifier 701 determines that the number of subframes is four (4), unless the sub-mode flag F tfsm Setting it to "1" or "2" (selecting the first or second encoding sub-mode) means that the content of the input audio signal 101 is closer to speech (the possibility of detecting speech-like features or time-based attacks in the input audio signal 101) and the available bit rate is less than 15 kbps. Specifically:

[0111] -In the first or second encoded sub-pattern (sub-pattern flag F) tfsm In the configuration of "1" or "2", the audio signal classifier 701 determines the number of four (4) subframes unless the available bit rate for encoding the input audio signal 101 is less than 15 kbps; then, an encoding mode using two (2) subframes is selected. In both cases, the corresponding number of fixed codebooks is used, i.e., two (2) or four (4) fixed codebooks; and

[0112] - In the third encoding mode (submode flag F) tfsm Setting it to 3 means that the content of the input audio signal 101 is closer to music (a “musical”-like feature is detected in the input audio signal 101). The audio signal classifier 701 determines the number of subframes to be four (4), but there is no fixed codebook contribution to keep more bits available for frequency domain excitation contribution unless the available bit rate for encoding the input audio signal 101 is greater than or equal to 22.6 kbps.

[0113] 3) Closed-loop pitch analysis

[0114] In the unified time / frequency domain CELP coding device 100 and method 150 ( Figure 1 In the process, when selector 205 selects general audio as the classification of input sound signal 101 and no time attack is detected in detector 208, hybrid time-domain / frequency-domain coding method 170 and corresponding hybrid time-domain / frequency-domain encoder 120 are used. Alternatively, in unified time-domain / frequency-domain CELP coding device 700 and method 750 ( Figure 7 In the process, when the sound signal classifier 701 classifies the input sound signal 101 into the "unclear signal type" category and selects one of the first, second, and third encoding sub-modes defined above (sub-mode flag F), tfsm When set to "1", "2" or "3"), the hybrid time-domain / frequency-domain coding method 770 and the corresponding hybrid time-domain / frequency-domain encoder 720 are used.

[0115] When using a hybrid time / frequency coding mode, closed-loop tone analysis is performed, followed by a fixed algebra book search if necessary. For this purpose, the hybrid time / frequency coding method 170 / 770 includes an operation 155 for calculating the time-domain excitation contribution. To perform operation 155, the hybrid time / frequency encoder 120 / 720 includes a calculator 105 for the time-domain excitation contribution. Calculator 105 itself includes an analyzer 211 (…). Figure 2 The analyzer 211, in response to the open-loop pitch analysis performed in the open-loop pitch analyzer 203 (or preprocessor 702) and the subframe length (or number of subframes in a frame) determined in the calculator 210 or the sound signal classifier 701, performs a closed-loop pitch analysis operation 261. Closed-loop pitch analysis is well known to those skilled in the art and an example of its implementation is described, for example, in ITU-T G.718 Recommendation, Reference [5]; Section 6.8.4.1.4.1. Closed-loop pitch analysis results in the computation of pitch parameters, also known as adaptive codebook parameters, which mainly consist of pitch lag (adaptive codebook index T) and pitch gain (adaptive codebook gain b). The adaptive codebook contribution is typically a past excitation at delay T or its interpolated version. The adaptive codebook index T is encoded and sent to a remote decoder. The pitch gain b is also quantized and sent to a remote decoder.

[0116] When closed-loop pitch analysis has been completed in operation 261 and fixed codebook contributions have been used, the calculator 105 for time-domain excitation contributions includes searching a fixed algebraic codebook 212 during operation 262 to find optimal fixed codebook parameters, typically including a fixed codebook index and a fixed codebook gain. The fixed codebook index and gain form the fixed codebook contribution. The fixed codebook index is encoded and sent to a remote decoder. The fixed codebook gain is also quantized and sent to a remote decoder. Fixed algebraic codebooks and their search are considered well known to those skilled in the art of CELP coding and will therefore not be described further in this disclosure.

[0117] Adaptive codebook index and gain, as well as fixed codebook index and gain (if used), form the temporal CELP excitation contribution.

[0118] 4) Frequency conversion

[0119] During frequency domain encoding in a hybrid time / frequency domain decoding mode, two signals are represented in the transform domain (e.g., in the frequency domain). In one embodiment, a 256-point Type II (or Type IV) DCT (Discrete Cosine Transform) can be used to implement the time-to-frequency transformation (this DCT provides a resolution of 25 Hz with an internal sampling rate of 12.8 kHz), but any other suitable transform can be used. When using another transform, it may be necessary to modify the frequency resolution (defined above), the number of frequency bands, and the number of frequency segments per band (further defined below) accordingly.

[0120] As described above, in the unified time-domain / frequency-domain CELP coding device 100 and method 150 ( Figure 1 and 2 In the process, when selector 205 selects general audio as the classification of input sound signal 101 and no time attack is detected in detector 208, a hybrid time-domain / frequency-domain coding mode is used. Alternatively, in the unified time-domain / frequency-domain CELP coding device 700 and method 750 ( Figure 7 In the audio signal classifier 701, when the input audio signal 101 is classified as an "unclear signal type" category, a hybrid time-domain / frequency-domain coding mode is used. The hybrid time-domain / frequency-domain encoder 120 / 720 includes a calculator 107 for frequency-domain excitation contributions. Figure 1 and Figure 7 ), which performs an operation 157 to calculate the frequency domain excitation contribution in response to the input LP residual res(n) (reference [5]) generated by the operation 251 of LP analysis of the input sound signal 101 performed by the analyzer 201 (and preprocessor 702). Figure 2As shown, calculator 107 can calculate DCT 213, such as type IIDCT with input LP residual res(n). The hybrid time-domain / frequency-domain encoder 120 / 720 also includes calculator 106 ( Figure 1 and Figure 7 ), used to perform the frequency transformation operation 156 for calculating the time-domain excitation contribution. For example... Figure 2 As shown, calculator 106 can calculate DCT 214, such as the type IIDCT of the temporal excitation contribution. The input LP residual f can be calculated using, for example, the following expression. res and time-domain CELP incentive contribution f exc Frequency transformation:

[0121] as well as:

[0122]

[0123] Where r es (n) is the input LP residual, c td (n) represents the temporal excitation contribution, and N is the frame length. In a possible implementation, the frame length is 256 samples for the corresponding internal sampling rate of 12.8 kHz. The temporal excitation contribution is given by the following relationship:

[0124] e td (n) = bv(n) + gc(n)

[0125] Where v(n) is the adaptive codebook contribution, b is the adaptive codebook gain, c(n) is the fixed codebook contribution, and g is the fixed codebook gain. It should be noted that the temporal excitation contribution may consist solely of the adaptive codebook contribution as described above.

[0126] 5) Cutoff frequency of time-domain contribution

[0127] When sound signal samples are classified as general audio ( Figure 1 Or the sound signal sample is classified as "unclear signal type" ( Figure 7 In the case of frequency domain coding, the contribution of time-domain excitation is not always significant for coding improvement. Typically, it does improve coding in the lower part of the spectrum, while the improvement in coding in the higher part of the spectrum is minimal. The hybrid time-domain / frequency-domain encoder 120 / 720 includes a cutoff frequency lookup unit and a filter 108. Figure 1 and Figure 7 This is used to perform the operation 158 of determining a cutoff frequency, above which the coding improvement provided by the temporal stimulus contribution becomes too low to be valuable. For example... Figure 2 As shown, the cutoff frequency finder and filter 108 includes a cutoff frequency calculator 215 and a filter 216.

[0128] The operation 265, which estimates the cutoff frequency of the time-domain excitation contribution, is first performed by calculator 215. Figure 2 Use calculator 303 ( Figure 3 and Figure 4 Upon completion, calculator 303 performs the frequency transformation of the input LP residual 301 from calculator 107 and the frequency transformation of the time-domain excitation contribution 302 from calculator 106 (specified as f respectively). res and f exc The normalized cross-correlation operation 353 between each frequency band (defined in the preceding section 4). Includes, for example, the last frequency L in each of sixteen (16) frequency bands. f Defined in Hz:

[0129]

[0130] For this illustrative example, for a 20ms frame with an internal sampling rate of 12.8kHz, per frequency band B b The number of frequency bins j per frequency band C Bb The cumulative frequency band and the normalized cross-correlation C of each frequency band i c (i) For example, the following definition:

[0131]

[0132]

[0133]

[0134] in

[0135]

[0136] as well as

[0137]

[0138] Among them B b It is B for each frequency band b The number of frequency bands j, C Bb It is the cumulative frequency range per frequency band, C c (i) is the normalized cross-correlation of each frequency band i. It is the excitation energy of the frequency band, and similarly, It is the residual energy per frequency band.

[0139] The cutoff frequency calculator 215 includes a smoother 304 that performs cross-correlation via frequency bands. Figure 3 and Figure 4This process performs several operations 354 to smooth the cross-correlation vectors between different frequency bands. More specifically, the cross-correlation smoother 304 uses, for example, the following relationship to calculate new cross-correlation vectors.

[0140]

[0141] In the illustrative embodiments,

[0142] α = 0.95; δ = (1-α); N b =13; β=δ / 2

[0143] The cutoff frequency calculator 215 also includes calculator 305 ( Figure 3 and Figure 4 It performs the calculation of a new cross-correlation vector. In the first N b Frequency bands (e.g., N) b =13, indicating the operation of the average value at 5575Hz (355).

[0144] The cutoff frequency calculator 215 also includes a cutoff frequency module 306. Figure 3 ),like Figure 4 As shown, the cutoff frequency module 306 includes a cross-correlation limiter 406, a cross-correlation normalizer 407, and a finder 408 for the frequency band with the lowest cross-correlation. More specifically, the limiter 406 performs the function of reducing the cross-correlation vector... The average value is limited to a minimum of 0.5 in operation 456, and the normalizer 407 performs the operation of cross-correlation vectors. The finite average normalization operation is performed between 0 and 1 (457). The finder (408) performs the operation by searching for the last frequency L of frequency band i. f Operation 458, which obtains a first estimate of the cutoff frequency, minimizes the last frequency L of frequency band i. f cross-correlation vector Normalized mean The difference between multiplying by half the internal sampling rate (Fs / 2) of the input audio signal 101:

[0145] and in

[0146] F5 = 12800Hz and

[0147] In the above relationship, This represents the first estimate of the cutoff frequency.

[0148] At low bit rates, the normalized average value Never very high (in) Figure 1 In the case of the unified time-domain / frequency-domain coding device 100 and method 150, or when the sub-mode flag F tfsm A value greater than "0" means that the input audio signal is classified as an "unclear signal type" (in...). Figure 7 In the case of the unified time-domain / frequency-domain coding device 700 and method 750, or in order to artificially increase To give more weight to the time-domain stimulus, the normalized mean can be amplified using normalizer 407 with a fixed scaling factor. The value of . As a non-limiting example, the first estimate of the cutoff frequency at bit rates below 8 kbps. Multiply by 2.

[0149] The accuracy of the cutoff frequency can be improved by adding the following components to the calculation. For this purpose, the cutoff frequency module 306 includes an extrapolator 410 in corresponding operation 460 that calculates the eighth harmonic using, for example, the minimum or lowest pitch lag value of the temporal excitation contribution from the subframe of the frame using the following relationship. Figure 4 ):

[0150]

[0151] Where F s =12800Hz is the internal sampling rate or frequency, N sub T is the number of subframes in the frame, and T(i) is the adaptive codebook index or tone lag of subframe i.

[0152] Cutoff frequency module 306 includes the eighth harmonic Finder 409 of the frequency band ( Figure 4 More specifically, for subframe i <N sub The finder 409 performs the operation 459 of searching for the highest frequency band, for which, for example, the following inequality is still verified:

[0153]

[0154] The index of this frequency band will be called It also indicates the frequency band where the 8th harmonic might be located.

[0155] Cutoff frequency module 306 ultimately includes the final cutoff frequency f. tc Selector 411 ( Figure 4 More specifically, selector 411 performs a first estimate f that preserves the cutoff frequency from finder 408 using the following relationship. tc1 With the eighth harmonic from finder 409 Operation at higher frequencies between the last frequency of the frequency band 461:

[0156] f tc =max(L f (i 8th ), f tc1 )

[0157] When using encoding sub-patterns, in Figure 7 In the case of the unified time-domain / frequency-domain coding device 700 and method 750, the cutoff frequency f is further defined using, for example, the following relationship. tc Thresholding:

[0158] f tc =max(max(L) f (i 8th ),2775),f tc1 )

[0159] like Figure 3 and Figure 4 As shown:

[0160] The cutoff frequency calculator 215 also includes: a decision maker 307. Figure 3 ), used to perform the operation 357 to determine the number of frequency bands in the frequency band to be zeroed;

[0161] - The decision 307 itself includes: analyzer 415 ( Figure 4 ), for performing parameter analysis operation 465; and selector 416 ( Figure 4 ), used to perform the operation of selecting the frequency band to be zeroed 466; and

[0162] -Filter 216( Figure 2 It operates in the frequency domain and includes a zero-reset unit 308 for performing filtering operation 266. Figure 3 The corresponding operation 358 will determine the frequency range to be zeroed in the determiner 307. The zeroer 308 can zero (a) all frequency ranges ( Figure 4 (a) The zeroing unit 417 and the corresponding zeroing operation 467) or (b) supplemented with a smooth transition region located at the cutoff frequency f tc

[0163] The above higher frequency range ( Figure 4 (Filter 418 and the corresponding filtering operation 468 in the diagram). The transition region is located at the cutoff frequency f. tc Above and below the return-to-zero frequency band, and it allows at the cutoff frequency f tc The following is a smooth spectral transition between the unchanged spectrum and the zero-return band at higher frequencies.

[0164] As a non-limiting illustrative example, when the cutoff frequency f from selector 411 tcAt 775Hz or lower, analyzer 415 considers the cost of the time-domain excitation contribution too high. Then, selector 416 selects all frequency segments representing the frequency of the time-domain excitation contribution to be zeroed, and zeroer 417 forces all frequency segments to zero, and also forces the cutoff frequency f to zero. tc The value is zero. Then, all bits allocated to the time-domain excitation contribution are reassigned to the frequency-domain coding mode. Otherwise, analyzer 415 forces selector 416 to select a frequency higher than the cutoff frequency f. tc The high-frequency band is so that it can be zeroed by the filter (zeroer) 418.

[0165] Finally, the cutoff frequency calculator 215 includes a quantizer 309 ( Figure 3 and Figure 4 ), used to perform the operation of setting the cutoff frequency f tc Quantized to the quantized version f of the cutoff frequency tcQ Operation 359 is used to transmit the data to a remote decoder. For example, if three (3) bits are associated with a cutoff frequency parameter, a set of possible output values ​​(in Hz) can be defined as follows:

[0166] f tcQ ={0,1175,1575,1975,2375,2775,3175,3575}

[0167] Selector 411 can use many mechanisms to stabilize the final cutoff frequency f. tc The choice is to prevent the quantized version f tcQ Switching between 0 and 1175 in inappropriate signal segments. To achieve this, as a non-limiting example, analyzer 415 responds to input from closed-loop tone analyzer 211 ( Figure 2 The long-term average pitch gain G lt 412. Open-loop pitch correlation C from open-loop pitch analyzer 203 ol 413 and smooth open-loop pitch are related to C st 414. To prevent switching solely to frequency domain coding, the analyzer 415 disallows such frequency domain coding only if, for example, the following condition is met: f tcQ Cannot be set to 0:

[0168] f tc >2375Hz or

[0169] f tc >1175Hz and C ol >0.7 and G lt >0.6

[0170] or

[0171] f tc≥1175Hz and C st >0.8 and G lt 0 ≥ 0.4

[0172] or

[0173] f tcQ (t-1)! = 0 and C ol >0.5 and C st >0.5 and G lt ≥0.6

[0174] Where c ol It is open-loop pitch-related 413, and C st Corresponding to, for example, C st =0.9·C ol +0.1·C st The defined open-loop pitch-related smooth version of 414. Additionally, G lt ( Figure 4 Term 412) corresponds to the long-term average of the tone gain obtained by the closed-loop tone analyzer 211 within the time-domain excitation contribution. The long-term average of the tone gain 412 is defined as... in This is the average tone gain on the current frame. To further reduce the switching rate between frequency-only coding and mixed time / frequency coding, a delay can be added.

[0175] 6) Frequency domain coding

[0176] 6.1) Create the difference vector

[0177] Once the cutoff frequency f of the time-domain excitation contribution is determined... tc Frequency domain coding is then performed. To perform this frequency domain coding, the hybrid time-domain / frequency-domain coding method 170 / 770 includes a subtraction operation 159, a frequency quantization operation 160, and an addition operation 161. The hybrid time-domain / frequency-domain encoder 120 / 720 includes a subtractor or calculator 109, a frequency quantizer 110, and an adder 111 to perform operations 159, 160, and 161, respectively.

[0178] Figure 5 This is a schematic block diagram showing an overview of both the frequency quantizer 110 and the corresponding frequency quantization operation 160. Furthermore, Figure 6 This is a schematic block diagram of the more detailed structure of the frequency quantizer 110 and the corresponding frequency quantization operation 160.

[0179] Subtractor or calculator 109 ( Figure 1 , 2 (5 and 6) utilize data from DCT 213 ( Figure 2 The frequency transformation f of the input LP residual res502 ( Figure 5 and 6 (or other frequency representations) and from DCT 214 ( Figure 2 Frequency transformation f of the time-domain excitation contribution exc 501 ( Figure 5 and 6 The difference between (or other frequency representations) forms the first part of the difference vector f, which extends from zero to the cutoff frequency f of the time-domain excitation contribution. tc In frequency transformation f res Before the corresponding spectral portion of 502 is subtracted, the reduction factor can be 603 ( Figure 6 The application (see multiplier 604 and the corresponding multiplication operation 654) to the frequency transform f exc 501 for use in f trans The next transition region is 2kHz (80 frequency bands in this implementation example). The result of the subtraction forms the difference vector f. d The second part, the difference vector f d Indicates from the cutoff frequency f tc to f tc +f trans The frequency range. Frequency transformation f of the input LP residual. res 502 is used for the difference vector f d The remaining third part.

[0180] The difference vector f generated by applying a reduction factor of 603 d The reduction portion can be performed using any type of fade-out function, and it can be shortened to only a few frequency bands, but only when the available bit budget is determined to be sufficient to prevent the cutoff frequency f. tc When energy oscillation artifacts change, they can also be omitted. For example, at 25Hz resolution, this corresponds to one frequency band f in a 256-point DCT at an internal sampling rate of 12.8kHz. bin =25Hz, the difference vector can be constructed as:

[0181] f d (k)=f res (k)-f exc (k)

[0182] where 0≤k≤f tc / f bin

[0183]

[0184] where f tc / f bin <k≤(ftc +f trans ) / f bin

[0185] f d (k)=f res (k), otherwise

[0186] Where f res ,f exc and f tc It has already been defined in the preceding description.

[0187] 6.2) Frequency domain bit allocation for coded sub-modes

[0188] 6.2.1) Allocate a portion of the available bits to lower frequencies.

[0189] In such Figure 7 In the unified time-domain / frequency-domain CELP coding method 750 shown, the hybrid time-domain / frequency-domain encoder 720 includes a frequency band selector and a bit allocator 707, and the hybrid time-domain / frequency-domain coding method 770 includes corresponding operations of frequency band selection and bit allocation detection 757.

[0190] Figure 9 It is shown at the same time Figure 7 A schematic block diagram of a band selector and bit allocator 707 and a corresponding operation 757 for band selection and bit allocation, which is used in... Figure 7 and Figure 8 In an alternative implementation of the unified time-domain / frequency-domain CELP coding method 150 / 750, when the input audio signal 101 is neither classified as speech nor music, the available bit budget is allocated to the difference vector f. d Frequency quantization.

[0191] Specifically, Figure 9 This demonstrates an innovative way in which the band selector and bit allocator 707 can allocate available bits to frequency quantization when the input audio signal 101 is not classified as speech or music, but rather as an "unclear signal type" based on a previously selected encoding sub-pattern. Figure 9 Frequency quantization is performed per band. For simplicity, in the current illustrative example, the bands have the same number of frequency segments, which is sixteen (16) frequency segments, with an internal sampling rate of 12.8 kHz. Band "0" represents the lower part of the spectrum, while band "15" represents the higher part of the spectrum.

[0192] To best utilize the bits available for frequency quantization, the band selection and bit allocation operation 757 includes a first operation 951 that pre-fixes a portion of the available bit budget (see 900) for use according to the quantized cutoff frequency f from the cutoff frequency finder and filter 108. tcQ The function to quantize the difference vector f d The lower frequency. In order to perform operation 951, estimator 901 uses, for example, the following relationship:

[0193]

[0194] P Btf =max(min(P) Btf ,0.75),0.5)

[0195] Where P Blf It is assigned to the difference vector f d The lower frequency is a portion of the available bits quantized by the frequency. In this example, the lower frequency refers to the first five (5) frequency bands, or the first two (2) kHz. Term L f (f tcQ ) refers to the frequency up to the quantization cutoff frequency f. tcQ The number of frequency bands.

[0196] Then, estimator 901 is based on the encoded sub-pattern flag F tfsm Adjusting this portion of P, which contains the available bits for frequency quantization at lower frequencies. Blf If the encoding sub-pattern flag F tfsm Set to "2" ( Figure 8 This means that if a temporal attack is detected in the current frame of the input audio signal 101, the portion of the frequency quantization bits allocated to the lower frequency is increased by 10% of the available bits. If a "musical" feature is detected in the content of the current frame, the sub-mode encoding flag F is set to "3". tfsm The indication is that the portion of P allocated to the frequency quantized bits of the lower frequency is... Blf Reduce available ratio

[0197] 10% of the special amount.

[0198] 6.2.2) Estimate the number of frequency bands to be quantized

[0199] The effect can be used on the difference vector f d Another parameter for the total number of bits per frequency band used in frequency quantization is the difference vector f to be quantized. d The maximum number N of frequency band estimates Bmx In the illustrative example described here, at an internal sampling rate of 12.8 kHz, the maximum total number of frequency bands Ntt It is sixteen (16).

[0200] When using a coding sub-mode, the band selection and bit allocation operation 757 includes estimating the difference vector f to be quantized. d The maximum number of frequency bands N Bmx Operation 952. To perform operation 952, if the encoded sub-pattern flag F... tfsm If set to "1" (selecting the first coding sub-mode), then the estimator 902 will determine the maximum number of frequency bands N. Bmx Set to "10". If the encoding sub-mode flag F tfsm If set to "2" (selecting the second coding sub-mode), then the estimator 902 will determine the maximum number of frequency bands N. Bmx Set to "9". If the encoding sub-mode flag F tfsm If set to "3" (selecting the third coding sub-mode), then the estimator 902 will determine the maximum number of frequency bands N. Bmx Set to "13". Then, estimator 902 uses, for example, the following relationship according to the difference vector f. d The frequency quantization bit budget function is used to readjust the maximum number N of frequency bands to be quantized. Bmx :

[0201]

[0202] N Bmx =max(min(trunc(N) Bmx ·N Badj +0.5), N tt ), 5),

[0203] Among them, B F This indicates that it can be used for the difference vector f d The number of bits for frequency quantization (see 900), B T It is the total bit rate available for encoding the channel being processed (see 900), F tfsm It is a sub-pattern flag (see 900), and N tt It is the maximum total number of frequency bands.

[0204] Estimator 902 can be used with the difference vector f. d The number of quantization bits in the mid-band and higher-band frequencies is related to further reducing the difference vector f to be quantized. d The maximum number of frequency bands. For the purpose of this constraint, it is assumed that the last lower frequency band and the first frequency band thereafter have similar bit counts m. b Or approximately the number of bits P quantized for the lower frequency. Blf17%. For the last frequency band to be quantized, use a minimum of 4.5 bits, m. p To quantize at least one (1) frequency pulse. If a bit rate B is available T If the bit depth is greater than or equal to 15 kbps, then the minimum number of bits is m. p It will be nine (9) to allow more pulses to be quantized per frequency band. However, if the total available bit rate BT is less than 15 kbps, but the submode flag F tfsm Setting it to "3" means the content is similar to music, so the number of bits m in the final frequency band to be quantized will be 6.75 to allow for more precise quantization. Then, the estimator 902 uses, for example, the following relationship to calculate the maximum number of corrected frequency bands N′. Bmx :

[0205] N′ Bmx =min(N) Bmx,5 +(B F -P Blf ·B F ) / 0.5.(m p +m b ))

[0206] Where N′ Bmx N corresponds to the maximum number of corrections for the frequency band to be quantized. Bmx This represents the maximum estimated number of bandwidths, while the number "5" indicates the minimum number of bandwidths. (B) F This indicates that it can be used for the difference vector f d The number of bits for frequency quantization, P Blf It is a portion of the quantized bits allocated to five (5) lower frequency bands, m p It is the minimum number of bits allocated for frequency quantization of the frequency band, m b It is the number of bits allocated to quantize the first frequency band after the five (5) lower frequency bands.

[0207] After calculating the maximum number of frequency bands, the estimator 902 can perform additional verification to ensure that m p Keep below or equal to m b While this additional verification step is optional, it is helpful at low bit rates in terms of difference vector f. d This allows for more efficient allocation of bits between frequency bands.

[0208] 6.2.3) Modify the number of bits allocated to lower frequencies

[0209] The bandwidth selection and bit allocation operation 757 includes operation 953 for calculating low-frequency bits. A calculator 903 is provided to perform operation 953. If the maximum number of bandwidths N′... BmxIf the calculation results in a smaller number of frequency bands to be quantized, then calculator 903 reallocates the bit portions previously allocated to the higher frequency bands using, for example, the following relationship, so that they are no longer related to the quantization of the lower frequency bands:

[0210] B LF =P Blf ·B F +(0.5·(m p +m b )·(N Bmx -N′ Bmx )),

[0211] Among them B LF Corresponding to the bits allocated to the five (5) lower frequency bands, B F Corresponding to the difference vector f d The number of bits used for frequency quantization at lower frequencies, P Blf It is a portion of the aforementioned bits from the estimator 901, allocated to, for example, five (5) lower frequency bands for frequency quantization, m p It is the minimum number of bits allocated to the quantization band, and m b It is the number of bits allocated to the first frequency band after quantizing five (5) lower frequency bands.

[0212] 6.2.4) Dual sorting of frequency bands

[0213] The band selection and bit allocation operation 757 includes a band characterization operation 954. To perform operation 954, the band selector and bit allocator 707 includes a band characterizer 904 that, once the bit rate is allocated between the lower bands and the remaining bands, performs a dual sorting of the bands to determine the importance of each band. The first sorting involves finding one or more bands that have lower energy compared to their neighboring bands. When this occurs, the characterizer 904 marks these bands such that even if the available bit budget is high, only a predetermined minimum number of bits m is allocated. p Frequency quantization can be assigned to these low-energy bands. The second sorting involves performing positional sorting of the medium-energy and higher-energy bands, for example, in descending energy order. These first and second sortings (dual sorting) are not performed on the lower bands, but rather up to a maximum number of bands N′. Bmx The operations for frequency band characterization can be summarized as follows:

[0214]

[0215]

[0216]

[0217] Among them, Ppb (i) For the minimum number of bits m will be used only p The frequency band was set to "1". The positions of the mid-energy and high-energy frequency bands are included in the order of decreasing energy, and E(i) corresponds to the energy of each frequency band. Bb and B b Defined in Section 5 above. Difference vector f d It is defined in Section 6.1.

[0218] exist Figure 7 and Figure 9 The difference vector f is calculated in calculator 708 and corresponding operation 758. d The energy E(i) for each frequency band. Calculator 708 and operator 758 also calculate the gain per frequency band, as referenced. Figure 6 The calculator 615 and operation 665 are described. Difference vector f d The energy E(i) and gain of each frequency band are quantized, for example, as shown regarding Figure 6 The quantizer 616 and operation 666 are described, and both are sent to a remote decoder. This is used in the unified time-domain / frequency-domain coding device 700 and method 750. Figure 7 In the implementation scenario, calculator 708 and operation 758 replace calculator 615 and operation 665, as well as quantizer 616 and operation 666.

[0219] 6.2.5) Allocate bits to the selected frequency band

[0220] The band selection and bit allocation operation 757 includes a final bit allocation operation 955 per band. To perform operation 955, the band selector and bit allocator 707 includes a final bit allocator 905 per band.

[0221] Once the frequency bands have been characterized, the allocator 905 allocates the difference vector f that can be used between the selected frequency bands. d Bit rate or number of bits for frequency quantization (B) F .

[0222] In a non-restrictive example, for the first five (5) lower frequency bands, allocator 905 linearly allocates the bits B allocated for frequency quantization of the lower frequencies. LF The first lowest frequency band received bit B LF 23% and the fifth (5th) lower frequency band received bit B LF The last 17%. In this way, the difference vector f can be quantized with sufficient precision. d The lower frequencies of the spectrum are synthesized to restore better quality of the input audio signal 101.

[0223] Distributor 905 assigns the data to the difference vector f according to a linear function. d Remaining bits B for frequency quantization F The bits are allocated to another, mid-frequency band, and higher-frequency band, but again considering the previous band energy characterization (operation 954), so that more bits can be allocated to higher-energy bands and fewer bits to lower-energy bands compared to their adjacent bands, thereby achieving higher precision quantization of the difference vector f. d The more important part of the spectrum is used to make more relevant use of the available bits. As a non-limiting example, the following relationship illustrates how bit allocation (operation 955) can be performed:

[0224]

[0225]

[0226] Among them B p (i) represents the number of bits allocated to each frequency band i, B F This represents the frequency quantization difference vector f. d The number of bits, B LF Corresponding to the bit rate or bits allocated to the five (5) lower frequency bands, m p P is the minimum number of bits for a frequency pulse in the quantization band. pb (i) contains the minimum number of bits to be used m p The position, and N′ Bmx It is the maximum number of frequency bands to be quantized.

[0227] If there are some unallocated bits after operation 955, allocator 905 allocates them to lower frequency bands. As a non-limiting example, allocator 905 will allocate one remaining bit per band, starting from the fifth (5th) frequency band and going back to the first frequency band, and repeat the process if all remaining bits need to be allocated.

[0228] Later, the allocator 905 may have to round, truncate, or round the number of bits per frequency band based on the algorithm used to perform the quantization of the frequency pulses and the potential fixed-point implementation.

[0229] 6.3) Search frequency pulse

[0230] The hybrid time-domain / frequency-domain CELP coding method 170 / 770 includes the difference vector f d Perform frequency quantization 160 ( Figure 1 , 2 And 7) operation. In order to perform operation 160, the hybrid time-domain / frequency-domain CELP encoder 120 / 720 includes a frequency quantizer 110 ( Figure 2(219 in the middle).

[0231] Several methods can be used to quantize the difference vector f. d In each case, the frequency pulse must be searched and quantized. In one possible implementation, the frequency quantizer 110 searches the difference vector f across the spectrum. d The highest energy pulse. The method for searching for pulses can be as simple as dividing the spectrum into bands and allowing a certain number of pulses per band. The number of pulses per band depends on the available bit budget and the band's position within the spectrum. Typically, more pulses are allocated to lower frequencies.

[0232] 6.4) Quantized difference vector

[0233] Depending on the available bit rate, the quantization of the frequency pulses can be performed by the frequency quantizer 110 using different techniques. In one embodiment, at bit rates below 12 kbps, a simple search and quantization scheme can be used to encode the position and symbol of the pulse. This scheme is described below as a non-limiting example.

[0234] For frequencies below 3175 Hz, a simple search and quantization scheme is used using a factorial pulse coding (FPC) method, which is described in the literature, for example in reference [8], the entire contents of which are incorporated herein by reference.

[0235] More specifically, see reference Figure 5 and Figure 6 The frequency quantizer 110 includes a selector 504 to perform an operation 554 that determines whether to use FPC quantization for all spectrum. Figure 5 As shown, if selector 504 determines that not all spectrum is quantized using FPC, then FPC encoding and pulse position and symbol encoding operations 556 are performed in encoder 506.

[0236] like Figure 6 As shown, the FPC encoding and pulse position and sign encoding operations 556 include a frequency pulse search operation 659, an FPC encoding operation 660, an operation 661 for finding the highest energy pulse, and an operation 662 for quantizing the position and sign of the frequency pulse. To perform operations 659-662, the encoder 506 includes a frequency pulse searcher 609, an FPC encoder 610, a highest energy pulse finder 611, and a frequency pulse position and sign quantizer 612, respectively.

[0237] Searcher 609 searches for frequency pulses below 3175 Hz across all frequency bands. FPC encoder 610 then processes the frequency pulses. Searcher 611 identifies the highest energy pulse for frequencies equal to and greater than 3175 Hz, and quantizer 612 encodes the position and sign of the found highest energy pulse. If more than one (1) pulse is allowed within the frequency band, the amplitude of the previously found pulse is divided by 2, and the search is repeated across the entire frequency band. Each time a pulse is found, its position and sign are stored for use in the quantization and bit packing stages. As a non-limiting example, the following pseudocode illustrates this simple search and quantization scheme:

[0238]

[0239] Where N BD It is the number of frequency bands (N in the illustrative example) BD =16), N p B is the number of pulses i to be encoded in frequency band k. b It is the number of frequency bands in each frequency band, C Bb It is the cumulative frequency segment of each frequency band as previously defined in Section 5, p p p represents a vector containing the found pulse location. s p represents a vector containing the symbol of the found pulse. max This indicates the energy of the pulse found.

[0240] At bit rates above 12 kbps, selector 504 determines whether to use FPC quantization for the entire spectrum. Figure 5 and Figure 6 ).like Figure 5 As shown, FPC encoding operation 555 is then performed in FPC encoder 505. (Refer to...) Figure 6 The encoder 505 includes a frequency pulse searcher 607, and operation 555 includes a corresponding operation 667 for searching for frequency pulses. The search for frequency pulses is performed across the entire frequency band. Operation 555 includes an operation 668 for encoding the found frequency pulses, and the encoder 505 includes an FPC processor 608 for performing operation 668.

[0241] Then, the FPC processor 608 or the quantizer 612 for pulse position and symbol will quantize by having pulse symbol p s The number of pulses nb_pulses and the position p found for each pulse p Add them together to obtain the quantized difference vector f dQ For each frequency band, the quantized difference vector f can be written using, for example, the following pseudocode. dQ :

[0242] for j=0,...,j <nb_pulses

[0243] f dQ (p p (j))+=p s (j)

[0244] 6.5) Noise Filling

[0245] The frequency band is quantized with more or less precision; the quantization method described in the previous section does not guarantee that all frequency segments within the band are quantized. This is especially true at low bit rates where the number of pulses quantized per band is relatively low. To prevent audible artifacts caused by these unquantized frequency segments, the frequency quantizer 110 includes a noise filler 507. Figure 5 This is used to perform the corresponding operation 557 of adding some noise to the unquantized frequency bands to fill these gaps. For example, this noise addition can be performed across the entire spectrum at bit rates below 12 kbps, but for higher bit rates, it can be performed only at the cutoff frequency f contributed by the time-domain excitation. tc The above applies this noise addition. For simplicity, the noise intensity varies only with the available bit rate. The noise level is low at high bit rates, but high at low bit rates.

[0246] Noise filler 507 includes adder 613 ( Figure 6 Adder 613 performs the action of adding noise to the quantized difference vector f after the strength or energy level of such added noise has been determined. dQ Operation 663. For this purpose, frequency quantization operation 160 includes operation 664 estimating the intensity or energy level of the added noise, and to perform operation 664, frequency quantizer 110 includes a corresponding estimator 614 for the noise energy level. Operation 664, estimating the intensity or energy level of the added noise, is performed by estimator 614 and precedes operation 665, which determines the per-band gain in per-band gain calculator 615 of frequency quantizer 110.

[0247] In the illustrative embodiment, in estimator 614, the noise level is directly related to the coding bit rate. For example, at 6.60 kbps, estimator 614 estimates the noise level N′. L The amplitude is set to 0.4 times the amplitude of the frequency pulse encoded in a specific frequency band, and gradually decreased to 0.2 times the amplitude of the frequency pulse encoded in a 24 kbps frequency band. Adder 613 injects noise only into portions of the spectrum where a certain number of consecutive frequency bands have very low energy, for example, when the cumulative energy of half the frequency band is below 0.5. For a specific frequency band i, noise is injected as follows:

[0248]

[0249] Where, for frequency band i, C Bb It is the cumulative number of frequency bands in each frequency band, B b N′ is the number of frequency bands in a specific frequency band i. L It is the level of added noise, and r and It is a random number generator that is restricted to the range of -1 to 1.

[0250] 6.6) Per-band gain quantization

[0251] refer to Figure 5 and Figure 6 The frequency quantization operation 160 of the unified time-domain / frequency-domain coding device 100 and method 150 includes an operation 665 for determining the gain per frequency band, followed by an operation 666 for quantizing the gain per frequency band. To perform operations 665 and 666, the frequency quantizer 110 includes a gain per frequency band calculator 615 and a gain per frequency band quantizer 616.

[0252] Once the quantized difference vector f is found dQ (If noise fill is required), calculator 615 calculates the per-band gain for each frequency band. Specific frequency band G b The gain per band of (l) is defined as the unquantized difference vector f in the logarithmic domain. dQ The difference vector f between the energy and the quantization dQ The ratio between energies can be expressed using, for example, the following relationship:

[0253]

[0254]

[0255] Where C Bb and B b Defined in Part 5 above.

[0256] The per-band gain quantizer 616 vector-quantizes the per-band frequency gain. Prior to vector quantization, at low bit rates, the last gain (corresponding to the last band) is quantized individually, and the remaining fifteen (15) per-band gains (e.g., when using 16 bands) are divided by the quantized last gain. The quantizer 616 then performs vector quantization on the normalized fifteen (15) remaining gains. At higher bit rates, the average per-band gain is first quantized, and then that average is removed from all per-band gains across, for example, sixteen (16) bands before vector quantization of these per-band gains. The vector quantization used can be a standard minimization in the logarithmic domain of the distance between the vector containing the per-band gain and entries in a particular codebook.

[0257] In the frequency domain coding mode, the gain for each frequency band is calculated in calculator 615 to convert the unquantized vector f d Energy and quantized vector f dQ Matching. The gain is vector-quantized in quantizer 616 and then multiplied by multiplier 509. Figure 5 and Figure 6 The vector f applied to the quantization in each frequency band (operation 559) dQ .

[0258] Alternatively, the entire spectrum can be encoded at a rate below 12 kbps by selecting only some frequency bands to be quantized. Before performing the frequency band selection, the unquantized difference vector f is quantized using quantizer 616. d The energy of the frequency band E d Energy can be calculated using, for example, the following relationship:

[0259] E d (i) = log 10 (S d (i))

[0260]

[0261] Where C Bb and B b Defined in Part 5 above.

[0262] In order to execute the frequency band energy E d The quantization of the first 12 of the sixteen frequency bands used is first quantized, and the average energy of the first 12 of the sixteen frequency bands used is subtracted from the total energy of all sixteen (16) frequency bands. Then, all frequency bands are vectors quantized in groups of 3 or 4 frequency bands. The vector quantization used can be a standard minimization in the logarithmic domain of the distance between the vector containing the gain of each frequency band and the entries of a particular codebook. If not enough bits are available, only the first 12 frequency bands can be quantized, and the average of the first three (3) frequency bands can be used, or the last four (4) frequency bands can be extrapolated by any other method.

[0263] Once the energy of the unquantized difference vector's frequency bands has been quantized, the energy can be sorted in descending order, making it reproducible on the decoder side. During sorting, all energy bands below 2kHz are always kept, and only the highest energy bands are then passed to the FPC scheme for encoding frequency pulse amplitudes and symbols. Using this method, the FPC scheme encodes smaller vectors but covers a wider frequency range. In other words, it requires fewer bits to cover important energy events across the entire spectrum.

[0264] In implementation Figure 7In certain cases of the unified time-domain / frequency-domain coding apparatus 700 and method 750, frequency band selection and bit allocation are alternatively performed as described above. Figure 7 and Figure 9 The per-band energy and per-band gain calculator 708 and calculation operation 758, as well as the band selector and bit allocator 707 and band selection and bit allocation operation 757, are performed as determined.

[0265] After pulse quantization, noise filling, similar to that described previously, is performed. Then, the gain adjustment factor G is calculated for each frequency band. a To quantize the difference vector f dQ Energy E dQ The difference vector f with the unquantized d Quantified energy E d The gain is matched. Then, this per-band gain adjustment factor is applied to the quantized difference vector f. dQ This can be expressed as follows:

[0266]

[0267] in,

[0268] as well as

[0269] E′ d It is the unquantized difference vector f as defined earlier. d The quantification of energy per frequency band.

[0270] After completing the frequency domain coding stage, the total time-domain / frequency-domain excitation is found. For this purpose, the hybrid time-domain / frequency-domain CELP coding method 170 / 770 includes an adder 111 using a hybrid time-domain / frequency-domain CELP encoder 120 / 720. Figure 1 , 2 (5 and 6) The difference vector f from the frequency quantizer 110 is obtained by quantizing the frequency. dQ The time-domain excitation contribution f of the filtered frequency transform excF The addition operation 161. When the unified time-domain / frequency-domain coding device 100 / 700 changes its bit allocation from a time-domain-only coding mode to a hybrid time-domain / frequency-domain coding mode, the per-band excitation spectrum energy of the time-domain-only coding mode does not match the per-band excitation spectrum energy of the hybrid time-domain / frequency-domain coding mode. This energy mismatch may produce switching artifacts that are more audible at low bit rates. To reduce any audible degradation caused by this bit reallocation, a long-term gain can be calculated for each band, and the long-term gain can be applied to the summed excitation to correct the energy of each band for several frames after the reallocation. Then, the hybrid time-domain / frequency-domain CELP coding method 170 / 770 includes the use of, for example, IDCT (inverse DCT) 220 ( Figure 2 The difference vector f quantized by frequency dQ The time-domain excitation contribution f after frequency transformation and filtering excF Operations to transform to the time domain 162 ( Figure 1 , 5 and 6).

[0271] The unified time-domain / frequency-domain coding method 150 / 750 includes LP synthesis filters 113 / 706 through coding devices 100 / 700. Figure 1 , 2 Operations 163 / 756, which filter the total time-domain / frequency-domain excitation from the IDCT 220 to generate a synthesized signal, and 7) .

[0272] Forming the quantized difference vector f dQ The quantized position and symbol of the frequency pulse are sent to a remote decoder (not shown).

[0273] In one non-limiting embodiment, while the CELP-coded memory is updated on a subframe-by-subframe basis using only temporal excitation contributions, total temporal / frequency excitation is used to update those memories at frame boundaries. In another possible implementation, the CELP-coded memory is updated on a subframe-by-subframe basis and also only using temporal excitation contributions at frame boundaries. This results in an embedded architecture where the frequency-quantized signal constitutes an upquantization layer independent of the core CELP layer. This presents advantages in certain applications. In this particular case, a fixed codebook is always used to maintain good perceptual quality, and the number of subframes is always four (4) for the same reason. However, frequency domain analysis can be applied to the entire frame. This embedded approach is suitable for approximately 12 kbps and higher bit rates.

[0274] 7) Decoder devices and methods

[0275] Figure 11 This is a schematic block diagram showing a decoder device 1100 and a corresponding decoding method 1150 for decoding the bitstream 1101 from the aforementioned unified time-domain / frequency-domain coding device 700 and the corresponding unified time-domain / frequency-domain coding method 750.

[0276] The decoder device 1100 includes a receiver (not shown) for receiving bit stream 1101 from the unified time-domain / frequency-domain coding device 700.

[0277] If the audio signal encoded by the unified time-domain / frequency-domain encoding device 700 has been classified as "music", this is indicated by the corresponding signaling bits in the bitstream 1101 and detected by the decoder device 1100 (see 1102). The received bitstream 1101 is then decoded by the "music" decoder 1103 (e.g., a frequency-domain decoder).

[0278] If the audio signal encoded by the unified time-domain / frequency-domain coding device 700 has been classified as “speech,” this is indicated by the corresponding signaling bits in the bitstream 1101 and detected by the decoder device 1100 (see 1104). The received bitstream 1101 is then decoded by the “speech” decoder 1105, for example, using a time-domain decoder employing ACELP (Algebraic Code-Excited Linear Prediction) or, more generally, CELP (Code-Excited Linear Prediction).

[0279] If the audio signal encoded by the unified time-domain / frequency-domain coding device 700 has not yet been classified as "music" or "speech" (see 1102 and 1104), and the bit rate available for encoding the audio signal is equal to or less than 9.2 kbps (see 1106), this is indicated in the bitstream by a sub-mode flag F set to "0". tfsm Instructions. Then use the backward encoding mode (i.e., as shown in 1107). Figure 1 and Figure 2 The traditional unified time-domain and frequency-domain coding model (EVS) is used to decode the received bit stream 1101.

[0280] Finally, if the audio signal encoded by the unified time-domain / frequency-domain coding device 700 has not yet been classified as "music" or "speech" (see 1102 and 1104), and the bit rate available for encoding the audio signal is higher than 9.2 kbps (see 1106), this is indicated in bitstream 1101 by a sub-mode flag F set to "1", "2", or "3". tfsm Instructions. Then use Figure 12 The audio signal decoder 1200 and the corresponding audio signal decoding method 1250 decode the received bit stream 1101.

[0281] 7.1) Audio signal decoder and decoding method

[0282] Figure 12 This is a schematic block diagram showing both a sound signal decoder 1200 and a corresponding sound signal decoding method 1250. The sound signal decoder 1200 and the corresponding sound signal decoding method 1250 are used to decode bitstreams from the aforementioned unified time-domain / frequency-domain encoding device 700 and the corresponding unified time-domain / frequency-domain encoding method 750 when the sound signal is classified as having an unclear signal type category.

[0283] As mentioned in the preceding description, the adaptive codebook index T and the adaptive codebook gain b are quantized and transmitted, and thus received in the bitstream by a receiver (not shown). Similarly, when used, the fixed codebook index and fixed codebook gain are also quantized and transmitted to the decoder, and thus received in bitstream 1101 by a receiver (not shown). The audio signal decoding method 1250 includes an operation 1256 to calculate the temporal excitation contribution of the decoded signal using the adaptive codebook index and gain, as well as the fixed codebook index and gain (if used), as is typically done in the CELP coding field. To perform operation 1256, the audio signal decoder 1200 includes a calculator 126 for the temporal excitation contribution of the decoded signal.

[0284] The audio signal decoding method 1250 also includes an operation 1257 for calculating the frequency transformation of the decoded time-domain excitation contribution using the same procedure as in operation 156 using DCT transform. To perform operation 1257, the audio signal decoder 1200 includes a calculator 1207 for the frequency transformation of the decoded time-domain excitation contribution.

[0285] As mentioned in the preceding description, the quantization version f of the cutoff frequency tcQ The signal is sent to the decoder and thus received in bitstream 1101 by the receiver (not shown). Audio signal decoding method 1250 includes using the decoded cutoff frequency f recovered from bitstream 1101. tcQ Operation 1258 filters the frequency transformation of the time-domain excitation contribution from calculator 1207 using a process identical or similar to the previously described filtering operation 266. To accomplish operation 1258, the audio signal decoder 1200 includes using the recovered cutoff frequency f... tcQ A filter 1208 performs frequency transformation on the time-domain excitation contribution. The filter 1208 has the same characteristics as... Figure 2 The filter 216 has the same or at least similar structure.

[0286] The filtered frequency transformation of the time-domain excitation contribution from filter 1208 is provided to the positive input of adder 1209, which performs the corresponding addition operation 1259.

[0287] Audio signal decoding method 1250 includes calculating the difference vector f d Operation 1260 involves decoding the energy and gain of each frequency band. To perform operation 1260, the audio signal decoder 1200 includes a calculator 1210. Specifically, the calculator 1210 dequantizes the quantized energy and quantized gain of each frequency band received by the receiver (not shown) from the unified time-domain / frequency-domain encoding device 700 in bitstream 1101 using a process opposite to the quantization process described in this disclosure.

[0288] Audio signal decoding method 1250 includes recovering the difference vector f of frequency quantization. dQ Operation 1261. To perform operation 1261, the audio signal decoder 1200 includes a calculator 1211. The calculator 1211 extracts the quantized position and symbol of the frequency pulses from the bit stream 1101 and copies the selection of frequency bands for quantization and bit allocation in different frequency bands determined by operation 757 and the distributor 707 and used by the unified time-domain / frequency-domain encoding device 700 to encode the input audio signal. The calculator 1211 uses the copied information to recover the frequency quantization difference vector f from the extracted frequency pulse quantization position and symbol. dQ Specifically, for this purpose, the audio signal decoder 1200 responds to the difference vector f in the decoder 1200 that can be used for frequency quantization. dQ The number of bits (bit rate) (see 1220), the total bit rate available for the channel being processed (see 1220), and the sub-mode flag (see 1220), copied as follows Figure 9 The process used in the unified time-domain / frequency-domain coding device 700 shown.

[0289] Specifically:

[0290] - Figure 12 The estimator 1201 and operation 1251 correspond to Figure 9 The estimator 901 and operation 951 are used to calculate the cutoff frequency f according to the quantization. tcQ The function is pre-fixed for quantizing the difference vector f d It is a portion of the available bit budget at lower frequencies.

[0291] - Figure 12 The estimator 1202 and operation 1252 correspond to Figure 9 The estimator 902 and operation 952 are used to estimate the quantized difference vector f. dQ The maximum number of frequency bands N Bmx .

[0292] - Figure 12 The calculator 1203 and operation 1253 correspond to Figure 9 The calculator 903 and operation 953 are used to calculate lower frequency bits.

[0293] - Figure 12 The characterizer 1204 and operation 1254 correspond to Figure 9 Characterizer 904 and operation 954 are used for frequency band characterization.

[0294] - Figure 12 The distributor 1205 and operation 1255 correspond to Figure 9 The allocator 905 and operation 955 are used for the final allocation of bits per frequency band.

[0295] The audio signal decoding method 1250 includes quantizing the difference vector f of the recovered frequency from the calculator 1211. dQ The time-domain excitation contribution f from the frequency transformation and filtering of filter 1208 excF Operation 1259, which combines to form a hybrid time-domain / frequency-domain excitation.

[0296] It is understood that estimators 1201 and 1202, calculator 1203, characterizer 1204, distributor 1205, calculators 1206 and 1207, filter 1208, calculators 1210 and 1211, and adder 1212 use information transmitted in bitstream 1101 to form a reconstructor with mixed time-domain / frequency-domain excitation, which includes a sub-mode flag identifying one of the coded sub-modes selected and used to encode sound signals classified as having an unclear signal type category.

[0297] In the same manner, operations 1251-1261 form a method for reconstructing the hybrid time-domain / frequency-domain excitation using the information transmitted in bitstream 1101.

[0298] The audio signal decoder 1200 includes a converter 1212 for performing an operation 1262 that transforms a mixed time-domain / frequency-domain excitation back to the time domain using, for example, an IDCT (inverse DCT) 220.

[0299] Finally, the synthesized audio signal is calculated in the decoder 1200 through the operation 1263 of filtering the total excitation from the converter 1212 by the LP (linear prediction) synthesis filter 1213. Of course, the LP parameters required by the decoder 1200 to reconstruct the synthesis filter 1213 are sent from the unified time-domain / frequency-domain coding device 700 and extracted from the bitstream 1101, as is well known in the field of CELP coding.

[0300] 8) Hardware Implementation

[0301] Figure 10 This is a simplified block diagram of an example configuration of the hardware components that form the aforementioned unified time-domain / frequency-domain coding device 100 / 700 and method 150 / 750, decoder device 1100 and decoding method 1150.

[0302] The unified time-domain / frequency-domain encoding device 100 / 700 and decoder device 1100 can be implemented as part of a mobile terminal, a portable media player, or any similar device. Device 100 / 700 and decoder device 1100 (in...) Figure 10 The part (marked as 1000) includes input terminal 1002, output 1003, processor 1001, and memory 1004.

[0303] Input 1002 is configured to receive digital or analog data. Figure 1 and Figure 7 The input audio signal 101 / bit stream 1101 is used. Output 1003 is configured to provide an output signal. Input 1002 and output 1003 can be implemented in a common module, such as a serial input / output device.

[0304] Processor 1001 is operatively connected to input 1002, output 1003, and memory 1004. Processor 1001 is implemented as one or more processors for executing code instructions that support, for example... Figure 1-9 The unified time-domain / frequency-domain coding device 100 / 700 shown is used for encoding input audio signals. Figure 11-12 The functions of various components of the decoder device 1100.

[0305] Memory 1004 may include non-transitory memory for storing code instructions executable by processor 1001. Specifically, it includes / stores processor-readable memory for non-transitory instructions that, when executed, cause the processor to implement the operations and components of the unified time-domain / frequency-domain coding apparatus 100 / 700 and method 150 / 750, as well as the decoder apparatus 1100 and decoding method 1150 described in this disclosure. Memory 1004 may also include random access memory or buffers for storing intermediate processing data from various functions performed by processor 1001.

[0306] Those skilled in the art will recognize that the description of the unified time-domain / frequency-domain coding apparatus 100 / 700 and method 150 / 750, as well as the decoder apparatus 1100 and decoding method 1150, is merely illustrative and not intended to be limiting in any way. Other embodiments will readily conceive of those skilled in the art upon receiving this disclosure. Furthermore, the disclosed unified time-domain / frequency-domain coding apparatus 100 / 700 and method 150 / 750, decoder apparatus 1100, and decoding method 1150 can be customized to provide valuable solutions to existing needs and problems in encoding and decoding sound.

[0307] For clarity, not all routine features of the unified time-domain / frequency-domain coding devices 100 / 700 and 150 / 750, as well as the decoder device 1100 and decoding method 1150, are shown or described. It should be understood that in developing any such practical implementation of the unified time-domain / frequency-domain coding devices 100 / 700 and 150 / 750, as well as the decoder device 1100 and decoding method 1150, many implementation-specific decisions may need to be made to achieve the developer's specific goals, such as compliance with application, system, network, and business-related constraints, and these specific goals will vary from implementation to implementation and from developer to developer. Furthermore, it should be understood that development work may be complex and time-consuming, but it remains a routine engineering task for those skilled in the art of sound processing who benefit from this disclosure.

[0308] According to this disclosure, the components / processors / modules, processing operations, and / or data structures described herein can be implemented using various types of operating systems, computing platforms, network devices, computer programs, and / or general-purpose machines. Furthermore, those skilled in the art will recognize that less general-purpose devices, such as hardwired devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can also be used. Where a method comprising a series of operations and sub-operations is implemented by a processor, computer, or machine, and those operations and sub-operations can be stored as a series of non-transitory code instructions readable by the processor, computer, or machine, they can be stored on tangible and / or non-transitory media.

[0309] The unified time-domain / frequency-domain coding apparatus 100 / 700 and method 150 / 750, as well as the decoder apparatus 1100 and decoding method 1150 described herein, may use software, firmware, hardware, or any combination of software, firmware, or hardware suitable for the purposes described herein.

[0310] In the unified time-domain / frequency-domain coding apparatus 100 / 700 and method 150 / 750, as well as the decoder apparatus 1100 and decoding method 1150 described herein, various operations and sub-operations can be performed in various orders, and some operations and sub-operations can be optional.

[0311] Although the present disclosure has been described above by way of its non-limiting illustrative embodiments, these embodiments may be modified freely within the scope of the appended claims without departing from the spirit and essence of the present disclosure.

[0312] 9) References

[0313] This disclosure references the following sources, the entire contents of which are incorporated herein by reference:

[0314] [1] U.S. Patent 9,015,038, “Coding generic audio signals at low bit rate and low delay”.

[0315] [2] 3GPP TS26.445, v.12.0.0, “Codec for Enhanced Voice Services (EVS); Detailed Algorithmic Description”, September 2014.

[0316] [3] 3GPP SA4 Contribution S4-170749 “New WID on EVS Codec Extension for Immersive Voice and Audio Services”, SA4 Meeting #94, June 26-30, 2017, http: / / www.3gpp.org / ftp / tsg_sa / WG4_CODEC / TSGS4_94 / Docs / S4-170749.zip

[0317] [4] U.S. Provisional Application 63 / 010,798, “Method and device for speech / music classification and core encoder selection in a sound codec”.

[0318] [5]ITU-T Recommendation G.718 “Frame error robust narrow-band and wideband embedded variable bit-rate coding of speech and audio from 8-32kbit / s”, June 2008.

[0319] [6] T. Vaillancourt et al., “Inter-tone noise reduction in a low bit rate CELP decoder”, Proceedings of the IEEE Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Taipei, Taiwan, April 2009, pp. 4113-16.

[0320] [7] V. Eksler and M. Jelnek, “Transition mode coding for source controlled CELP codecs”, Proceedings of the IEEE Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), March-April 2008, pp. 4001-4043.

[0321] [8] U. Mittal, J. Pashley and E.M. Cruz-Zeno, “Low Complexity Factorial Pulse Coding of MDCT Coefficients using Approximation of Combinatorial Functions”, Proceedings of the IEEE Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Taipei, Taiwan, April 2007, pp. 289-292.

Claims

1. A unified time-domain / frequency-domain coding device for encoding input audio signals, comprising: A classifier that classifies the input sound signal into one of a plurality of sound signal categories, wherein the sound signal category includes an unclear signal type category, the unclear signal type category indicating that the nature of the input sound signal is unclear; A selector for encoding one of a plurality of encoding sub-patterns when the input sound signal is classified into the unclear signal type category; as well as A hybrid time-domain / frequency-domain encoder is used to encode the input audio signal using a selected coding sub-mode.

2. The time-domain / frequency-domain unified coding device according to claim 1, wherein, The sound signal categories include speech, music, and ambiguous signal types that indicate the input sound signal is neither classified as speech nor music.

3. The unified time-domain / frequency-domain coding device according to claim 2, comprising: A frequency domain encoder is used to encode the input sound signal if the classifier classifies the input sound signal as a music category.

4. The unified time-domain / frequency-domain coding device according to claim 2, comprising: A time-domain encoder is used to encode the input sound signal if the classifier classifies the input sound signal as a speech category.

5. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein, The selector selects the encoding sub-mode in response to the bit rate used to encode the input audio signal and the characteristics of the input audio signal classified into the unclear signal type category.

6. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein the coding sub-mode is identified by a corresponding sub-mode flag.

7. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein, If (a) the bit rate available for encoding the input audio signal is not higher than a first given value and (b) the input audio signal is neither classified as speech nor as music, then the selector selects a backward coding sub-mode that uses a conventional unified time-domain and frequency-domain coding model to encode the input audio signal.

8. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein, If a characteristic similar to "speech" is detected in the input sound signal, the selector selects the first encoding sub-mode.

9. The unified time-domain / frequency-domain coding device according to claim 8, wherein, If (a) the input audio signal is not classified as speech or music by the classifier, and the bit rate available for encoding the input audio signal is higher than a second given value, (b) the probability that the input audio signal is music is not greater than a third given value, and (c) no time attack is detected in the current frame of the input audio signal, then the selector selects the first encoding sub-mode.

10. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein, If a time-based attack is detected in the input audio signal, the selector selects the second encoding sub-mode.

11. The unified time-domain / frequency-domain coding device according to claim 10, wherein, If (a) the input audio signal is not classified as speech or music by the classifier, and the bit rate available for encoding the input audio signal is higher than a fourth given value, (b) the probability that the input audio signal is music is not greater than a fifth given value, and (c) a time attack is detected in the current frame of the input audio signal, then the selector selects the second encoding sub-mode.

12. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein, If a characteristic similar to "music" is detected in the input sound signal, the selector selects the third encoding sub-mode.

13. The unified time-domain / frequency-domain coding device according to claim 12, wherein, If (a) the input sound signal is not classified as speech or music by the classifier and the bit rate available for encoding the input sound signal is higher than a sixth given value, and (b) the probability that the input sound signal is music is greater than a seventh given value, then the selector selects the third encoding sub-mode.

14. The unified time-domain / frequency-domain coding apparatus according to any one of claims 1 to 4, wherein: - If a characteristic similar to "speech" is detected in the input sound signal, the selector selects the first encoding sub-mode; - If a time-based attack is detected in the input audio signal, the selector selects a second encoding sub-mode; and - If a characteristic similar to "music" is detected in the input sound signal, the selector selects the third encoding sub-mode.

15. The unified time-domain / frequency-domain coding device according to claim 14, wherein, The selector (a) selects a given number of subframes frame-by-frame in the third encoding sub-mode to encode the input audio signal, and (b) in the first encoding sub-mode and the second encoding sub-mode, selects a number of subframes less than the given number and based on the number of bit rates available for encoding the input audio signal.

16. A unified time-domain / frequency-domain coding method for encoding input audio signals, comprising: The input sound signal is classified into one of a plurality of sound signal categories, wherein the sound signal category includes an unclear signal type category, the unclear signal type category indicating that the nature of the input sound signal is unclear; Select one of a plurality of encoding sub-modes for encoding the input sound signal when the input sound signal is classified into the unclear signal type category; as well as The input audio signal is mixed time-domain / frequency-domain encoded using the selected coding sub-mode.

17. The time-domain / frequency-domain unified coding method according to claim 16, wherein, The sound signal categories include speech, music, and ambiguous signal types that indicate the input sound signal is neither classified as speech nor music.

18. The unified time-domain / frequency-domain coding method according to claim 17, comprising: If the input sound signal is classified as a music category, the input sound signal is frequency domain encoded.

19. The unified time-domain / frequency-domain coding method according to claim 17, comprising: If the input sound signal is classified as a speech category, the input sound signal is time-domain encoded.

20. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein, Selecting one of a plurality of encoding sub-modes includes: selecting the encoding sub-mode in response to the bit rate used to encode the input audio signal and the characteristics of the input audio signal classified into the unclear signal type category.

21. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, comprising identifying the coding sub-mode by a corresponding sub-mode flag.

22. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein, Selecting one of a plurality of coding sub-modes includes: if (a) the bit rate available for encoding the input audio signal is not higher than a first given value and (b) the input audio signal is not classified as speech or music, then a backward coding sub-mode that uses a conventional unified time-domain and frequency-domain coding model to encode the input audio signal is selected.

23. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein, Selecting one of multiple encoding sub-modes includes: selecting the first encoding sub-mode if a characteristic similar to "speech" is detected in the input sound signal.

24. The unified time-domain / frequency-domain coding method according to claim 23, wherein, If (a) the input audio signal is not classified as speech or music, and the bit rate available for encoding the input audio signal is higher than a second given value, (b) the probability that the input audio signal is music is not greater than a third given value, and (c) no time attack is detected in the current frame of the input audio signal, then the first encoding sub-mode is selected.

25. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein, Selecting one of multiple encoding sub-modes includes: selecting a second encoding sub-mode if a time attack is detected in the input audio signal.

26. The unified time-domain / frequency-domain coding method according to claim 25, wherein, If (a) the input audio signal is not classified as speech or music, and the bit rate available for encoding the input audio signal is higher than a fourth given value, (b) the probability that the input audio signal is music is not greater than a fifth given value, and (c) a time attack is detected in the current frame of the input audio signal, then the second encoding sub-mode is selected.

27. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein, Selecting one of several encoding sub-modes includes: selecting a third encoding sub-mode if a characteristic similar to "music" is detected in the input sound signal.

28. The unified time-domain / frequency-domain coding method according to claim 27, wherein, If (a) the input sound signal is not classified as speech or music and the bit rate available for encoding the input sound signal is higher than a sixth given value, and (b) the probability that the input sound signal is music is greater than a seventh given value, then the third encoding sub-mode is selected.

29. The unified time-domain / frequency-domain coding method according to any one of claims 16 to 19, wherein: - If a characteristic similar to "speech" is detected in the input sound signal, then the first encoding sub-mode is selected; - If a time-based attack is detected in the input audio signal, then a second encoding sub-mode is selected; and - If a characteristic similar to "music" is detected in the input sound signal, then the third encoding sub-mode is selected.

30. The unified time-domain / frequency-domain coding method according to claim 29, wherein, Selecting one of a plurality of encoding sub-modes includes: (a) in the third encoding sub-mode, selecting a given number of subframes frame-by-frame to encode the input audio signal, and (b) in the first and second encoding sub-modes, selecting a number of subframes less than the given number and based on the number of bit rates available for encoding the input audio signal.

31. A unified time-domain / frequency-domain coding device for encoding input audio signals, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to implement: A classifier that classifies the input sound signal into one of a plurality of sound signal categories, wherein the sound signal category includes an unclear signal type category, the unclear signal type category indicating that the nature of the input sound signal is unclear; A selector for encoding one of a plurality of encoding sub-patterns when the input sound signal is classified into the unclear signal type category; as well as A hybrid time-domain / frequency-domain encoder is used to encode the input audio signal using a selected coding sub-mode.

32. A unified time-domain / frequency-domain coding device for encoding input audio signals, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to: The input sound signal is classified into one of a plurality of sound signal categories, wherein the sound signal category includes an unclear signal type category, the unclear signal type category indicating that the nature of the input sound signal is unclear; Select one of a plurality of encoding sub-modes for encoding the input sound signal when the input sound signal is classified into the unclear signal type category; as well as The input audio signal is mixed time-domain / frequency-domain encoded using the selected coding sub-mode.

33. A sound signal decoder, comprising: A receiver of a bitstream that transmits information for reconstructing a hybrid time-domain / frequency-domain excitation, the hybrid time-domain / frequency-domain excitation representing an audio signal classified in an unclear signal type category, the unclear signal type category indicating that the nature of the audio signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the audio signal classified in the unclear signal type category; A reconstructor that responds to a hybrid time-domain / frequency-domain excitation of information including coded sub-patterns for encoding the audio signal transmitted in the bitstream; A converter that transforms the hybrid time-domain / frequency-domain excitation to the time domain; as well as A synthesis filter is used to filter the mixed time-domain / frequency-domain excitation converted to the time domain to produce a synthesized version of the sound signal.

34. The audio signal decoder according to claim 33, wherein, The encoded sub-mode is identified by a sub-mode flag in the bitstream.

35. The audio signal decoder according to claim 33 or 34, wherein, The encoding sub-modes include: (a) a first encoding sub-mode in the case where the sound signal contains characteristics similar to "speech", (b) a second encoding sub-mode in the case where the sound signal contains time attacks, and (c) a third encoding sub-mode in the case where the sound signal contains characteristics similar to "music".

36. The audio signal decoder according to any one of claims 33 to 34, wherein, The reconstructor recovers the frequency representation of the time-domain excitation contribution from the information transmitted in the bitstream, reconstructs the frequency-quantized difference vector between the frequency-domain excitation contribution and the frequency representation of the time-domain excitation contribution, and adds the frequency-quantized difference vector to the frequency representation of the time-domain excitation contribution to generate the hybrid time-domain / frequency-domain excitation.

37. A method for decoding audio signals, comprising: Receive a bit stream that transmits information that can be used to reconstruct a mixed time-domain / frequency-domain excitation, the mixed time-domain / frequency-domain excitation representing a sound signal classified as an unclear signal type category, the unclear signal type category indicating that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the sound signal classified as the unclear signal type category; In response to information including coded sub-patterns for encoding the sound signal transmitted in the bitstream, the hybrid time-domain / frequency-domain excitation is reconstructed; Transform the hybrid time-domain / frequency-domain excitation to the time domain; and The mixed time-domain / frequency-domain excitation, converted to the time domain, is filtered by a synthesis filter to produce a synthesized version of the sound signal.

38. The audio signal decoding method according to claim 37, wherein, The encoded sub-mode is identified by a sub-mode flag in the bitstream.

39. The audio signal decoding method according to claim 37 or 38, wherein, The encoding sub-modes include: (a) a first encoding sub-mode in the case where the sound signal contains characteristics similar to "speech", (b) a second encoding sub-mode in the case where the sound signal contains time attacks, and (c) a third encoding sub-mode in the case where the sound signal contains characteristics similar to "music".

40. The sound signal decoding method according to any one of claims 37 to 38, wherein, Reconstructing the hybrid time-domain / frequency-domain excitation includes: recovering the frequency representation of the time-domain excitation contribution from the information transmitted in the bit stream; reconstructing the frequency quantized difference vector between the frequency representations of the frequency-domain excitation contribution and the time-domain excitation contribution from the information transmitted in the bit stream; and adding the frequency quantized difference vector to the frequency representation of the time-domain excitation contribution to generate the hybrid time-domain / frequency-domain excitation.

41. A sound signal decoder, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to implement: A receiver of a bitstream that transmits information for reconstructing a hybrid time-domain / frequency-domain excitation, the hybrid time-domain / frequency-domain excitation representing a sound signal classified as an unclear signal type category, the unclear signal type category indicating that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the sound signal classified as the unclear signal type category; A reconstructor that responds to a hybrid time-domain / frequency-domain excitation of information including coded sub-modes for encoding the audio signal transmitted in the bitstream; A converter that transforms the hybrid time-domain / frequency-domain excitation to the time domain; as well as A synthesis filter is used to filter the mixed time-domain / frequency-domain excitation converted to the time domain to produce a synthesized version of the sound signal.

42. A sound signal decoder, comprising: At least one processor; as well as A memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to: Receive a bitstream that conveys information that can be used to reconstruct a mixed time-domain / frequency-domain excitation, the mixed time-domain / frequency-domain excitation representing an audio signal classified as an unclear signal type category, the unclear signal type category indicating that the nature of the audio signal is unclear, wherein the information includes one of a plurality of coding sub-patterns for encoding the audio signal classified as the unclear signal type category; In response to information including coded sub-patterns for encoding the audio signal transmitted in the bitstream, the hybrid time-domain / frequency-domain excitation is reconstructed. Transform the hybrid time-domain / frequency-domain excitation to the time domain; and The mixed time-domain / frequency-domain excitation, converted to the time domain, is filtered by a synthesis filter to produce a synthesized version of the sound signal.

Images