Packet loss concealment based on adaptive cross-band filtering

Abstract

A method for packet loss concealment for frame-based audio is provided. The method comprises, in a first mode, storing time-frequency coefficients of a current frame of the audio in a buffer when the time-frequency coefficients of the current frame are available. The method further comprises, in a second mode, predicting the time-frequency coefficients of the current frame based on a plurality of prediction filter coefficients stored in the buffer and time-frequency coefficients of one or more previous frames when the time-frequency coefficients of the current frame are not available, and storing the predicted time-frequency coefficients in the buffer. Corresponding apparatus, program, and computer-readable storage medium are also provided.

Description

Technical Field

[0001] This disclosure relates to techniques for packet loss concealment (frame loss concealment) of frame-based audio. In particular, this disclosure describes methods and apparatus for packet loss concealment of frame-based audio based on frequency domain cross-band prediction. Cross-references to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 624,066 and European Patent Application No. 24164281.8, filed January 23, 2024, each of which is incorporated herein by reference in its entirety. Background Technology

[0003] For audio coding with low latency constraints, lost packets (e.g., lost frames) cannot always be replaced by the coded representation of the original audio. Therefore, the ability to extrapolate the audio at the receiver is required. For the special case of speech signals, many existing solutions exist based on time-domain and frequency-domain predictions, including emerging machine learning methods. For lost packets in longer bursts, it is important that the method not be too realistic or creative, as this risks altering the expected message. For general audio, speech-specific solutions often perform poorly, and a common approach in transform-based coding is to repeat the transform bin of the previous frame and randomly flip the symbols. A speech-specific improvement to this method is described in US Patent Application 16 / 126,940, "Packet Loss Concealment for Critically-Sampled Filter Bank-Based Codecs Using Multi-Sinusoidal Detection." Here, the crossband filter is derived from a sinusoidal signal model, but applying this method to general audio has shown less than ideal results. Summary of the Invention

[0004] Therefore, there is a need for improved techniques for packet loss concealment in frame-based audio, especially general audio. Such techniques are particularly needed for low-latency codecs. Low latency typically depends on various factors, such as frame size, sampling rate, hardware and / or software computing resources. For example, in some cases, low latency is typically less than 40 ms, 20 ms, or 10 ms.

[0005] In view of this need, this disclosure provides a method for packet loss concealment of frame-based audio, as well as corresponding apparatus, programs and computer-readable storage media, which have the features of the respective independent claims.

[0006] One aspect of this disclosure relates to a method for packet loss concealment in frame-based audio. For example, the frame-based audio may involve or include any of the following: mono signal, stereo signal, multi-channel signal, immersive audio, etc. The method may include: in a first mode (e.g., normal mode), storing the time-frequency coefficients of the current frame in a buffer when the time-frequency coefficients of the current frame of the audio are available (e.g., have been effectively received). For example, without limitation, the time-frequency coefficients may be coefficients of a modified discrete cosine transform (MDCT) or coefficients of a complex modulation filter bank transform (such as a quadrature mirror filter (QMF) transform or a discrete Fourier transform (DFT)). The method may further include: in a second mode (e.g., concealment mode), when the time-frequency coefficients of the current frame are not available (e.g., when the current frame has not yet been effectively received), predicting the time-frequency coefficients of the current frame based on a set of prediction parameters stored in the buffer and the time-frequency coefficients of one or more previous frames, and storing the predicted time-frequency coefficients in the buffer. For example, the prediction may involve cross-band prediction. The method may further include outputting the time-frequency coefficients of the current frame and / or the predicted time-frequency coefficients of the current frame, for example for audio rendering or further processing.

[0007] Therefore, the proposed method can extrapolate the time-frequency coefficients (e.g., MDCT coefficients) of previously buffered frames to hide lost frames. This method is particularly applicable to general audio, including music. Here, general audio is understood to encompass any audio that is not strictly speech, such as music, audio from movies, or TV clips.

[0008] In some embodiments, the set of prediction parameters may include multiple prediction filter coefficients. The prediction filter coefficients may include, for each of multiple frequency bands, prediction filter coefficients associated with corresponding time-frequency coefficients of one or more previous frames in the band and in one or more adjacent frequency bands. The multiple frequency bands may involve, for example, the full frequency range of the time-frequency transformation or a low frequency range of the time-frequency transformation (e.g., a frequency threshold selected up to 4 kHz to 6 kHz). In one example, a given frequency band and one or more adjacent frequency bands may involve a frequency band range centered on the given frequency band, for example, three frequency bands centered on the given frequency band.

[0009] Therefore, cross-band prediction can be applied to realistically predict any lost frames based on the previous frames in the buffer.

[0010] In some embodiments, the prediction of the time-frequency coefficients of the current frame in a given frequency band may be based on the time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands, and on the prediction filter coefficients associated with the corresponding time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands.

[0011] In some embodiments, the current frame and frequency band The predicted time-frequency coefficients It can be by Given. Here. It is frequency band Relative to frame In frequency band Time-frequency coefficients in The predicted filter coefficients, Indicates the number of prior frames to be used for prediction, and This indicates the number (or range) of adjacent frequency bands to be used for prediction. Adjacent frequency bands can correspond to: a given number (e.g., one or more) of neighboring frequency bands towards higher frequencies and a given number of neighboring frequency bands towards lower frequencies, relative to the frequency band in question. It should be understood that the total number of frequency bands used for prediction is determined by… Given that a frequency band has only one adjacent frequency band among multiple frequency bands, it can be assumed that the time-frequency coefficients in the corresponding non-existent adjacent frequency band are zero. Alternatively, the equation above can be interpreted as follows: The summation range can be shifted to cover only the existing frequency bands (among multiple frequency bands).

[0012] In some embodiments, the method may further include updating the set of prediction parameters based on the time-frequency coefficients of the current frame in a first mode (e.g., normal mode). Therefore, the prediction parameters (e.g., prediction filter coefficients and / or accumulated per-band values) can be continuously updated without packet loss (e.g., frame loss).

[0013] In some embodiments, the prediction parameters can be updated based on the time-frequency coefficients of a predetermined number of frames, including the current frame and previous frames. The time-frequency coefficients of previous frames can be obtained from a buffer.

[0014] In some embodiments, in a first mode (e.g., normal mode), for a given frequency band, updated prediction filter coefficients can be determined based on the current prediction filter coefficients for the given frequency band and corresponding adjustment amounts to the current prediction filter coefficients. For each prediction filter coefficient, a corresponding adjustment amount may exist. The updated prediction filter coefficients can be determined by adding the corresponding adjustment amount to the corresponding current prediction filter coefficient. Thus, adaptive filtering for frame prediction purposes can be implemented.

[0015] In some embodiments, the adjustment amount for a given frequency band (the adjustment amount for each prediction filter coefficient of the given frequency band) may be determined using least squares, based on the current prediction filter coefficients associated with the corresponding time-frequency coefficients of one or more previous frames in the given frequency band and one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of one or more previous frames in the given frequency band and one or more adjacent frequency bands of the given frequency band.

[0016] In some embodiments, determining the adjustment amount for a given frequency band may include: predicting the time-frequency coefficients of the current frame in the given frequency band based on the adjustment amount (e.g., a functional dependency). Determining the adjustment amount for a given frequency band may further include: minimizing the squared prediction error between the predicted time-frequency coefficients of the current frame in the given frequency band and the time-frequency coefficients of the current frame in the given frequency band for a given change in the adjustment amount for the given frequency band. This may involve determining the gradient of the squared prediction error relative to the adjustment amount for the given frequency band. For example, the given change in the adjustment amount for the given frequency band may be a change in the sum or sum of squares of the adjustment amounts for the given frequency band. The maximum decrease in the squared prediction error may be found in the opposite direction of the gradient.

[0017] This allows for continuous updating of the prediction filter coefficients in normal mode while efficiently receiving frames. Specifically, the prediction filter coefficients are adapted based on the received audio to provide the best prediction for any lost frames based on previous frames.

[0018] In some embodiments, determining the adjustment amount for a given frequency band may include: determining a probabilistic model of the time-frequency coefficients of the current frame in the given frequency band based on current prediction filter coefficients associated with corresponding time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands of the given frequency band. Determining the adjustment amount for a given frequency band may further include: determining the adjustment amount for the given frequency band based on the probabilistic model and based on the time-frequency coefficients of the current frame in the given frequency band. For example, the probabilistic model may include the mean and variance of a normally distributed random variable.

[0019] This provides an alternative to continuously updating the prediction filter coefficients using the least squares method. Similarly, the prediction filter coefficients can be continuously updated in normal mode while efficiently receiving frames, where the prediction filter coefficients are adapted based on the received audio to provide the best prediction for any lost frames based on previous frames.

[0020] In some embodiments, the adjustment amount for a given frequency band and a given current prediction filter coefficient can be determined based on a probability model, the time-frequency coefficients of the current frame in a given frequency band, and the time-frequency coefficients associated with a given current prediction filter coefficient.

[0021] In some embodiments, the predicted time-frequency coefficients in the buffer may not be used to update the prediction parameters. This ensures that the prediction is based on actual frames that have already been received. For long stretches of lost frames (e.g., dozens or more lost frames), a smooth fade-out may be more advantageous than attempting to predict frames indefinitely.

[0022] In some embodiments, the method may further include: in a second mode (e.g., a hidden mode), using least squares to determine prediction filter coefficients based on a predetermined number of time-frequency coefficients from previous frames.

[0023] Therefore, the average computational burden (computational complexity) can be reduced (e.g., in normal mode), while the peak computational burden increases (e.g., during frame loss). The choice can be made between determining the prediction filter coefficients during frame loss or continuously determining them in normal mode, based on the desired balance between the average and peak computational burdens. Further balance can be achieved by using one of these techniques for a first frequency range and another for different second frequency ranges.

[0024] In some embodiments, the prediction filter coefficients for a given frequency band among multiple frequency bands can be determined using the least squares method based on a predetermined number of previous frames in the given frequency band and time-frequency coefficients in one or more adjacent frequency bands.

[0025] In some embodiments, the least squares error to be minimized for a given frequency band can be determined based on the sum of prediction errors of time-frequency coefficients of a predetermined number of previous frames, each predicted time-frequency coefficient being based on the time-frequency coefficients of one or more preceding frames of the corresponding frame in the predetermined number of previous frames.

[0026] In some embodiments, predicting the time-frequency coefficients of the current frame may further include limiting the predicted time-frequency coefficients for a given frequency band to a range based on the cumulative per-band value of the signal power in the given frequency band. This can reduce audible artifacts such as "popping" sounds that may be caused in some cases by an exponentially growing number of predicted samples.

[0027] In some embodiments, the method may further include, in a second mode (e.g., a hidden mode), generating, for each of a plurality of frequency bands, a weighted sum of the predicted time-frequency coefficients of the current frame in that frequency band and the latest available time-frequency coefficients multiplied by random symbols. Thus, depending on perceptual considerations, conventional random symbol repetition methods can be “hybridized” with the proposed technique.

[0028] In some embodiments, the weights of the weighted sum in a given frequency band may be determined based on one or more of the following: per-band value of signal power in the given frequency band, per-band value of predicted signal power in the given frequency band, per-band value of prediction error of predicted signal power in the given frequency band, and / or per-band value of log-relative noise level in the given frequency band. For example, the per-band value may be a cumulative value or a propagation value.

[0029] In some embodiments, for consecutive prediction frames, the weights of the weighted sum can be successively reduced to decrease the relative importance of the latest available time-frequency coefficients multiplied by the random symbol. Thus, if a long span of frames (e.g., dozens or more frames) is lost, a gradual fading of the prediction signal can be achieved.

[0030] In some embodiments, the method may further include: for a pair of channels, multiplying the latest available time-frequency coefficient of one channel by a mixed value of random symbols and adding it to the weighted sum of the other channel. This allows the recovery of, for example, the inter-channel correlation expected in a stereo signal.

[0031] In some embodiments, the mixing value may be determined based on a per-band value of the signal cross product relative to the channel pair. For example, the per-band value of the signal cross product may be a cumulative value or a propagation value.

[0032] In some embodiments, the multiple frequency bands may involve (e.g., correspond to) the lower frequency portion of the frequency range of time-frequency coefficients. The method may then further include: for at least one frequency band not located in the lower frequency portion, multiplying the latest available time-frequency coefficient in the at least one frequency band by a random symbol to use as the time-frequency coefficient of the current frame in the at least one frequency band. This allows for load balancing based on, for example, perceptual considerations.

[0033] According to another aspect, an apparatus for packet loss concealment in frame-based audio is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform all steps of the method according to the preceding aspect and its embodiments.

[0034] According to another aspect, a computer program is described. This computer program may include executable instructions that, when executed by a computing device, are used to perform the methods or method steps outlined throughout this disclosure.

[0035] According to another aspect, a computer-readable storage medium is described. This storage medium can store a computer program adapted to execute on a processor and, when executed on a processor, to perform the methods or method steps outlined throughout this disclosure.

[0036] It should be noted that the methods and apparatus outlined in this disclosure (including their preferred embodiments) can be used independently or in combination with other methods and apparatus disclosed in this document. Furthermore, all aspects of the methods and apparatus outlined in this disclosure can be combined arbitrarily. In particular, the features of the claims can be combined with each other in any manner.

[0037] It will be understood that the features of the apparatus and the steps of the method can be interchanged in various ways. In particular, as those skilled in the art will understand, the details of the disclosed (multiple) methods can be implemented by the corresponding apparatus, and vice versa. Furthermore, any statement above regarding the (multiple) methods (and, for example, their steps) should be understood to also apply to the corresponding apparatus (and, for example, its blocks, stages, units, etc.), and vice versa. Attached Figure Description

[0038] The invention will now be explained by way of example with reference to the accompanying drawings, in which: Figure 1 This is a block diagram schematically illustrating an example of a frequency domain decoder according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating an example method for packet loss concealment of frame-based audio according to an embodiment of the present disclosure; Figure 3 The illustration schematically depicts an embodiment according to the present disclosure. Figure 1 A block diagram of an example of the components of a frequency domain decoder; Figure 4 An example of a frequency domain prediction geometry according to an embodiment of the present disclosure is schematically illustrated; Figure 5 This is a flowchart illustrating an example of a method for updating prediction filter coefficients according to an embodiment of the present disclosure; Figure 6 This is a flowchart illustrating another example of a method for updating prediction filter coefficients according to an embodiment of the present disclosure; Figures 7 to 10 The figures illustrate examples of performance results of the technology according to embodiments of the present disclosure; and Figure 11 This is a block diagram of an example of an apparatus for performing a method according to an embodiment of the present disclosure. Detailed Implementation

[0039] Overview In a broad sense, this disclosure relates to techniques (e.g., methods and apparatus) for packet loss concealment in frame-based audio. For example, the disclosed techniques are applicable to audio in general.

[0040] A key idea of ​​the embodiments of this disclosure is that the filter coefficients of the crossband filter used for predicting lost frames are continuously updated during successful transmission according to an extended adaptive filter method. Specifically, embodiments of this disclosure may include or relate to one or more of the following: • Time-frequency adaptive prediction and filter update rules, • A hybrid rule for combining predicted and random contributions. • Stabilization is achieved by limiting the predicted time-frequency coefficients based on energy estimation, and • Provides a means for controlling peak complexity and average complexity by splitting the calculation of filter coefficients into two variants that operate in different frequency regions.

[0041] Figure 1 This is a block diagram schematically illustrating an example of a frequency domain decoder 100 with packet loss (e.g., frame loss) concealment according to an embodiment of the present disclosure. Figure 1 In the example, the decoder is an MDCT-based audio decoder, but audio decoders based on other forms of time-frequency transforms (e.g., QMF transform or DFT) are also understood to be covered by this disclosure. Therefore, although this disclosure may frequently refer to MDCT coefficients or MDCT binaries, it should be understood that this is not intended to be limiting, and this disclosure generally applies to time-frequency coefficients.

[0042] In the decoder 100, the parameters to be hidden are continuously updated during normal mode operation. The decoder 100 continuously receives MDCT coefficient frames 10 for processing. A buffer (MDCT buffer) 20 is also maintained for the current frame and a predetermined number of previous frames, into which any incoming (i.e., current) frame 10 is stored. In some embodiments, the buffer 20 may be a first-in-first-out (FIFO) buffer of a predetermined length. Prediction parameters used to predict (lost) frames from one or more previous frames are stored in a parameter buffer 50. As long as a frame is validly received (e.g., a frame that can be decoded or otherwise processed to obtain the necessary MDCT coefficients), the prediction parameters are continuously updated by the parameter update module 40, and the updated prediction parameters 45 are stored in the parameter buffer 50.

[0043] In the event of packet loss, i.e., if the current frame of the MDCT coefficients is unavailable (e.g., not effectively received), the hidden mode is invoked based on the current state of the prediction parameters in parameter buffer 50 and the time-frequency coefficients in MDCT buffer 20. Therefore, the hidden module 60 creates a predicted MDCT frame (e.g., the output MDCT frame) 65 based on the aforementioned parameters and coefficients, and then feeds this frame to the output terminal 70. The created MDCT frame 65 is also used to update MDCT buffer 20. This process is repeated for each lost frame until MDCT data is received again, at which point the normal mode is re-entered. Mode selection (depending on whether the current frame of the MDCT coefficients is available) can be performed by the mode selection module (e.g., a switch) 30.

[0044] As configured as described above, decoder 100 can output MDCT frame 80 for further processing (e.g., in the context of audio rendering) regardless of whether the current frame of the MDCT coefficients is successfully received or lost. The output MDCT frame 80 can be the current frame (if available) or its predicted version, which are output by output modules (e.g., adders) 70 respectively.

[0045] The aforementioned hidden operations and parameter updates will be described in more detail below.

[0046] Figure 2 This is a flowchart illustrating an example method 200 for packet loss concealment (e.g., frame loss concealment) of frame-based audio according to embodiments of the present disclosure. Method 200 includes steps S210 and S220, and an optional step S230.

[0047] At step S210, in the first mode (e.g., normal mode), when the time-frequency coefficients (e.g., MDCT coefficients) of the current frame of audio are available, the time-frequency coefficients of the current frame are stored in a buffer (e.g., MDCT buffer 20 of decoder 100).

[0048] As will be described in more detail below, in some embodiments, step S210 may further include: updating a set of prediction parameters (in the first mode) based on the time-frequency coefficients of the (available) current frame. In other words, when there is no packet loss (e.g., frame loss), the prediction parameters can be updated continuously. The update of the prediction parameters may be performed based on the time-frequency coefficients of a predetermined number of frames, including the current frame and previous frames, wherein the previous frames can be obtained from a buffer. In some embodiments, the time-frequency coefficients of previous (e.g., buffered) frames (which are themselves the predicted time-frequency coefficients) may not be used to update the prediction parameters.

[0049] At step S220, in a second mode (e.g., hidden mode), when the time-frequency coefficients of the current frame are unavailable, the time-frequency coefficients of the current frame are predicted based on a set of prediction parameters stored in a buffer and the time-frequency coefficients of one or more previous frames. Step S220 further includes storing the predicted time-frequency coefficients in a buffer. For example, the prediction may involve cross-band prediction.

[0050] As will be described in more detail below, in some embodiments, step S220 may further include determining at least a portion of the prediction parameters (e.g., prediction filter coefficients) based on a predetermined number of time-frequency coefficients from previous frames. This can be done, for example, using the least squares method. Also, it should be understood here that in some embodiments, the time-frequency coefficients of earlier (e.g., buffered) frames (which are themselves the predicted time-frequency coefficients) may not be used (e.g., may be skipped) in determining the prediction parameters. Specifically, determining at least a portion of the prediction parameters at step S220 can be performed in embodiments where the prediction parameters are not planned to be updated at step S210 in normal mode, and vice versa.

[0051] At step S230, in the second mode, for each of the multiple frequency bands, a weighted sum is generated of the predicted time-frequency coefficients of the current frame in that frequency band and the latest available time-frequency coefficients multiplied by a random symbol. This weighted sum can be used as the output frame. As mentioned above, step S230 can be an optional step.

[0052] although Figure 2 Not shown, but method 200 may further include the following steps: outputting the time-frequency coefficients of the current frame (if available) and / or the predicted time-frequency coefficients of the current frame (if the current frame is not available), for example for audio rendering or further processing. For example, in the latter case, the output may involve the result of step S220 or step S230.

[0053] Hidden Operations Next, we will describe the details of the hiding operation, namely, predicting the time-frequency coefficients if the current frame is lost.

[0054] Figure 3 It is a schematic illustration of what can be, for example, in Figure 1 Hidden module 60 and / or in Figure 2 A block diagram of an example of the hidden pattern operation 300 performed in step S220. The hidden pattern operation 300 in this example includes cross-band prediction 310, random symbol mixing 320 (which may be optional), and restriction 330 (which may also be optional).

[0055] The parameter buffer 50 remains unchanged during the second mode (e.g., the hidden mode) because the prediction parameters are not updated in this mode. The prediction parameters (e.g., the parameter buffer 50) may include the following per-band values. • Predicting filter coefficients

[0056] They may further include one or more of the following: • Signal power

[0057] • Instructions for random symbol mixing operations shall take one of the following forms. 1. Predicted Power

[0058] 2. Prediction Error Power

[0059] 3. Log-relative noise level

[0060] For example, given a frame The time-frequency coefficients (e.g., MDCT coefficients). (in, Indicates the signal power per frequency band of a frequency band or compartment (e.g., an MDCT line). It can be calculated as (1) Furthermore, given a frame Predicted power per frequency band of time-frequency coefficients It can be calculated as (2) in, These are the predicted time-frequency coefficients for the current frame.

[0063] Given a frame The prediction error power per frequency band of the time-frequency coefficients It can be calculated as (3) As will be described below, the above per-band quantities can be used , , and The cumulative values ​​(e.g., moving average, first-order cumulative) are used instead of the per-band values ​​mentioned above. These cumulative values ​​depend on both the per-band values ​​of the current frame and the cumulative values ​​(e.g., moving average, first-order cumulative) of previous frames.

[0065] Cross-band prediction Next, we will describe in more detail, for example, in Figure 2 The cross-band prediction is performed in step S220.

[0066] Typically, the set of prediction parameters used for prediction can include multiple prediction filter coefficients. For each of the multiple frequency bands, there may exist corresponding time-frequency coefficients for one or more previous frames in the frequency band and one or more adjacent frequency bands. Associated prediction filter coefficients Multiple frequency bands can cover, for example, the full frequency range of time-frequency conversion or the low frequency range of time-frequency conversion.

[0067] Then, at step S220, the prediction of the time-frequency coefficients of the current (and lost) frame in the given frequency band can be based on the time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands, and on the prediction filter coefficients associated with the corresponding time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands.

[0068] In one example, a given frequency band and one or more adjacent frequency bands can involve a frequency band range centered on the given frequency band, for example, three frequency bands centered on the given frequency band.

[0069] Figure 4 The diagram illustrates an example frequency domain prediction geometry or time-frequency receptive field (i.e., relying on previous time-frequency coefficients to predict the current time-frequency coefficients). The squares in the diagram indicate transform modules, which in turn correspond to time-frequency coefficients (e.g., MDCT coefficients). The horizontal direction represents time, and the vertical direction represents frequency. As described above, each target transform module (i.e., the transform module of the lost frame) can be predicted based on a predetermined time-frequency receptive field through a linear combination of one or more (e.g., multiple) previous modules in time and their frequency-adjacent modules. Figure 4 In the example, in the (time, frequency) index The light gray bins are based on (e.g., via) bins (in and The prediction is based on a linear combination of ( ). In this example, the values ​​over the time range and frequency range values of and Provided.

[0070] In other words, the current frame and frequency band The predicted time-frequency coefficients It can be given by the following formula (4) in, It is frequency band Relative to frame In frequency band (Buffered) time-frequency coefficients The predicted filter coefficients, Indicates the number of prior frames to be used for prediction, and Indicates the number of adjacent frequency bands to be used for prediction.

[0072] For a frequency band that has only one adjacent frequency band among multiple frequency bands (e.g., a frequency band at the edge of a range spanned by these multiple frequency bands), it can be assumed that the time-frequency coefficients in the corresponding non-existent adjacent frequency band are zero, or in other words, when it is necessary to... When performing summation, the available data can be expanded with zeros. Alternatively, in equation (4), the values ​​are... The summation range can be shifted to cover existing frequency bands.

[0073] Random symbol mixing Existing random symbol repetition methods involve multiplying the last unhidden cell by a random symbol as the hiding factor for the current cell in each frequency band.

[0074] Based on this Figure 2 Method 200 may include a step of performing random symbol mixing in step S230. The result of the symbol repetition method is expressed as... It can be based on the mixing factor (or weight). With forecast warehouse mix, (5) Therefore, as described above, step S230 can be equivalent to: in the second mode, for each of the multiple frequency bands... Generate the current frame In this frequency band Predicted time-frequency coefficients Time-frequency coefficients with random symbol repetition (This is a weighted sum of the available time-frequency coefficients from the most recently received valid reception multiplied by a random symbol). In this paper, the weights of the weighted sum in a given frequency band can correspond to the aforementioned mixing factors. .

[0076] If the prediction filter All zero and (That is, all weights) are set to one, then the random symbol repetition method is restored. On the other hand, if If all values ​​are zero, then there is no random contribution to the output.

[0077] Mixing factors (or weights) It can be determined (e.g., calculated) based on one or more of the following: signal power in a given frequency band. Value per frequency band, predicted signal power in a given frequency band The prediction error of the predicted signal power in a given frequency band and the value per frequency band. Values ​​per frequency band, and / or log-relative noise levels in a given frequency band The values ​​per frequency band. For example, these values ​​per frequency band can be cumulative values ​​or propagation values ​​(e.g., moving average, first-order cumulative).

[0078] For example, mixing factors The state of the parameter buffer can be calculated as one of the following formulas. (6) (7) (8) For consecutively hidden frames, the energy contributed by the random symbol mixing can be reduced over time by recursively decaying the mixing factor in each processed frame. In other words, for consecutively predicted frames, the weights of the weighted sum can be successively reduced. This ensures a smooth fade-out for consecutively lost frames.

[0082] Random symbol mixing for stereo signals For two stereo signals (or typically for a pair of channels), prediction processing can be performed separately for each of the two stereo signals. The processing for the first channel can be the same as described above. For the second channel, the signal covariance can be calculated. The signal covariance can then be used to derive the mixing weights of the random symbol mixed signal such that the covariance of the random symbol mixed signal approximately matches the measured (i.e., expected) covariance. This could involve mixing the random symbol coefficients of the first channel into the random symbol mixture of the second channel (and vice versa).

[0083] For example, random symbol mixing of the first and second channels can be performed via the following formula. (9) (10) The last index indicates the vocal tract index (0 or 1), and where (11) (12) (13) covariance C(n) It can be based on two input signals and Calculated as

[0089] (13a) in, This represents the expected value, which can be calculated in practice, similar to signal power. S(n) The expected value, for example, through a moving average filter, a first-order IIR filter, or other suitable methods. In the case of stereo, the number of channels... ( ) signal power The expected value can be given, for example, by the following formula:

[0090] (13b) In other words, the random symbol mixing at step S230 may include: for a pair of channels, multiplying the latest available time-frequency coefficient of one channel by the mixed value of the random symbols and adding it to the weighted sum (i.e., random symbol mixing) of the other channel. This mixed value may be determined based on the per-band value of the signal cross product relative to the pair of channels. For example, the per-band value of the signal cross product may be a cumulative value or a propagation value (e.g., a moving average, a first-order cumulative value).

[0091] limit There is no simple criterion for the stability of linear prediction of recursive vector values. For hiding over several time frames, through... Figure 1 The feedback loop of the MDCT buffer 20 may therefore cause instability in the output. A simple remedy proposed in this disclosure is to optionally apply a limit to the output bin (the predicted time-frequency coefficient) such that its absolute value never exceeds the per-band limit. If not intended to impose a limit, this limit value can be calculated via signal power parameters, for example, to avoid a power increase exceeding 3 dB. . (14) Thus, when constraints are used, the final output of the hiding process can be, . (15) In other words, predicting the time-frequency coefficients of the current frame can also involve limiting the predicted time-frequency coefficients of a given frequency band to a range based on the per-band value (e.g., cumulative or propagated value, such as a moving average or first-order cumulative) of the signal power in the given frequency band before outputting the final result or before applying random symbol mixing.

[0094] Parameter update In some implementations (especially when prediction parameters are continuously updated and not only determined when frames are lost), the state of the parameter buffer (i.e., the prediction parameters) is updated during normal (non-hidden) operation (i.e., in the first mode). Specifically, for a given frequency band, the updated prediction filter coefficients can be determined based on the current prediction filter coefficients for that band and their corresponding adjustment amounts. For each prediction filter coefficient, there can be a corresponding adjustment amount. The updated prediction filter coefficients can be determined by adding the corresponding adjustment amount to the corresponding current prediction filter coefficient. The adjustment amount can alternatively be referred to as a prediction filter coefficient update, i.e., an update for each of a plurality of prediction filter coefficients.

[0095] The following sections will describe two different implementations for determining the adjustment amount (or, in general, the update): the least squares method and a probabilistic model-based method.

[0096] Least squares method This idea involves adjusting the predictor coefficients (i.e., the current predictor filter coefficients). The current state of the data is adjusted additively (by the amount of adjustment). This incrementally improves the estimation of the current MDCT frame. These updates are made for each target frequency band. Execute independently.

[0097] Furthermore, given a frequency band Adjustment amount Based on one or more previous frames ( In a given frequency band In and in a given frequency band One or more adjacent frequency bands ( The corresponding time-frequency coefficients in ) Associated current prediction filter coefficients And based on one or more previous frames in a given frequency band In and in a given frequency band Time-frequency coefficients in one or more adjacent frequency bands To determine. Specifically, the adjustment amount. It was determined using the least squares method.

[0098] Based on the method, a given frequency band is determined. Adjustment amount Example of method 500 in Figure 5 The flowchart illustrates this. Method 500 includes steps S510 and S520.

[0099] In step S510, for example, based on the adjustment amount using the following formula. To predict the current frame In a given frequency band Time-frequency coefficients in

[0100] (16) Updates can be made for each target frequency band. Execute independently. To find the adjustment amount. In step S520, for a given frequency band Given a change in the adjustment amount, minimize the current frame. In a given frequency band The predicted time-frequency coefficients With the current frame In a given frequency band Time-frequency coefficients in Squared prediction error For example, squared prediction error It can be given by the following formula (17) Minimize the squared prediction error This may involve determining the squared prediction error. Relative to a given frequency band Adjustment amount The gradient. For example, a given change in the adjustment amount for a given frequency band can be a change in the sum or sum of squares of the adjustment amounts for that given frequency band. For coefficient adjustment Given a change in the sum of squares, the squared prediction error The maximum reduction can be Compared to The gradient is obtained in the opposite direction. This can be equivalent to updating (adjusting) the prediction filter coefficients using the formula given below. (18) Wherein gain It is positive. Since the prediction target is a scalar, it can be determined by... Adjusted to To obtain the vanishing prediction error ,in, (19) A trade-off can be struck between update speed and prediction quality. In some embodiments, the method presented in this disclosure... • Intended for use Among them, the predetermined value This depends on the physical time step of the time-frequency transform (e.g., MDCT coding system). For example, a good choice for a step size of approximately 5 ms could be...

[0105] • Further restrictions This allows for coefficient updates. The sum of squares is less than

[0106] This can be expressed in one step as (20) The prediction parameters of the buffer are then passed through The updated, or in other words, the updated prediction filter coefficients are determined by adding the corresponding adjustments to the corresponding current prediction filter coefficients.

[0108] To update the power estimate , and It can be achieved, for example, by applying a simple first-order accumulation using the following formula. (twenty one) (twenty two) (twenty three) For example, for a 5 ms step size system, a suitable choice could be... and .

[0112] Probabilistic modeling An alternative approach to determining the adjustment amount (or, in general, the update) could be to consider a predictor that provides a parameterized probabilistic model of the target warehouse.

[0113] Based on the method, a given frequency band is determined. Adjustment amount Example of method 600 in Figure 6 The flowchart illustrates this. Method 600 includes steps S610 and S620.

[0114] According to the method, at step S610, a probabilistic model is determined based on the current prediction filter coefficients associated with the corresponding time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of one or more previous frames in the given frequency band and in one or more adjacent frequency bands of the given frequency band. The probabilistic model determined in this step may include the mean of a normally distributed random variable. And variance. For example, we can assume It is a normally distributed random variable with a mean of (twenty four) And the variance is ,in, It is a current estimate of the noise level relative to the signal power. The logarithmic control parameters. For measurement... With target warehouse The matching quality can be determined, for example, using the non-negative log-likelihood (NLL) loss, which in this case can be given, for example, by the following equation. (25) in, It is a constant. It should be understood that this loss... It can be used to optimize probabilistic models and / or maximize the probability given by the probabilistic model. With target warehouse Matching.

[0117] In some embodiments, parameters It can be directly used to obtain the blending factor in the above-mentioned hidden mode. .

[0118] At step S620, the adjustment amount for the given frequency band is determined based on the probability model and the time-frequency coefficients of the current frame in the given frequency band. For example, the adjustment amount for the given frequency band and the given current prediction filter coefficients can be determined based on the probability model, the time-frequency coefficients of the current frame in the given frequency band, and the time-frequency coefficients associated with the given current prediction filter coefficients.

[0119] In one example, the prediction coefficient and noise level parameters Additive updates (e.g., adjustments) can be performed relative to each other. and Scaling Calculate using the negative gradient to obtain (26) as well as (27) Similar considerations to those used for the least squares method can also be used to adjust the update gain. and The value, for example, It can be defined as in equation (20), and Can be with Proportional. Furthermore, the updated results... Setting a lower limit for the absolute value (e.g., -10) may be advantageous.

[0122] In the case of probabilistic modeling, only the signal power estimate needs to be updated. Therefore, the same method as the least squares method can be applied, such as the update process of equation (21).

[0123] Modifications to achieve lower complexity Combined with random repetition in high frequency If only as described above (for example, refer to...) Figure 3 The proposed technique can be simplified by applying the hidden function to certain frequency ranges and applying conventional techniques to the remaining frequency ranges.

[0124] Therefore, in some embodiments, the aforementioned multiple frequency bands may involve (e.g., corresponding to) the lower frequency portion of the frequency range of time-frequency coefficients (e.g., MDCT). Then, for at least one frequency band not within this range, a conventional random symbol repetition method can be used. That is, for at least one frequency band not in the lower frequency portion, the latest available time-frequency coefficient in that at least one frequency band can be multiplied by a random symbol and used as the time-frequency coefficient of the current frame in that at least one frequency band.

[0125] For example, it may be advantageous to run adaptive filtering only in the low to mid-frequency range (e.g., up to a threshold selected in the range of 4-6 kHz), and then use a cheaper method of repeating random symbols for frequencies above that range.

[0126] Alternatively, the prediction filter length at higher frequencies can be increased. Set to compare the prediction filter length at lower frequencies (e.g., time Smaller (e.g.) time This is also reasonable from a perceptual perspective.

[0127] Block-based computation of predictive filter coefficients As a means of reducing average complexity at the expense of peak complexity, prediction filter coefficients can be calculated based on previously stored time-frequency coefficients (e.g., MDCT coefficients) when frame loss occurs.

[0128] Therefore, in some implementations (especially when the prediction parameters are not continuously updated but only determined when a frame is lost), the state of the parameter buffer (i.e., the prediction parameters or prediction filter coefficients) is determined during the hiding operation (i.e., in the second mode). Thus, the prediction filter coefficients can be updated directly before predicting the time-frequency coefficients of the lost frame.

[0129] Specifically, the prediction filter coefficients can be determined based on the time-frequency coefficients of a predetermined number of previous frames. Specifically, determining the prediction filter coefficients for a given frequency band among multiple frequency bands can be based on the time-frequency coefficients of a predetermined number of previous frames in the given frequency band and in one or more adjacent frequency bands.

[0130] Minimum number of time-frequency (e.g., MDCT) frames (including the current frame) stored. This can depend on the number of prediction filter coefficients per prediction time-frequency (e.g., MDCT) bin. And can be selected to satisfy, for example (28) in, and This is to ensure that the optimized prediction coefficients capture the short-term stationary statistics of the signal. Therefore, the predetermined number of frames mentioned above can be selected as more than [a certain number]. .

[0132] One approach is to use least squares to determine the prediction filter coefficients based on a predetermined number of time-frequency coefficients from previous frames.

[0133] For example, the prediction filter coefficients can be computed by running the adaptive least squares process described above on the stored time-frequency coefficients. In another example, the prediction filter coefficients that minimize the total least squares prediction error for all stored time-frequency coefficients (block-based) can be computed, as follows for the frequency band index. As shown, each stored time-frequency coefficient is predicted based on its own predetermined time-frequency receptive field. (29) Here, we assume the time index of the current frame is .

[0135] Generally, it is necessary to target a given frequency band. The minimum least squares error can be determined based on the sum of prediction errors of time-frequency coefficients of a predetermined number of previous frames, wherein each predicted time-frequency coefficient can be based on the time-frequency coefficients of one or more preceding frames of the corresponding frame in the predetermined number of previous frames.

[0136] The minimization of the prediction error in equation (29) can be restated in matrix notation as follows: (30) in, It has lines and A matrix of columns, yes A column vector of prediction filter coefficients, and It is warehouse of A column vector of target values. Then, the least squares solution can be found as... (31) The block-based method described in this paper has advantages in cases of multiple lost frames, such as... Figure 7 and Figure 8 As shown, this illustration depicts the SNR measurement of hidden MDCT coefficients of size 240 across 20 scale factor bands. The error mode is 10 frames lost per 100 frames (SNR is shown only in lost and hidden frames). Figure 7 The adaptive prediction coefficient update using 8 stored MDCT coefficients is shown, and Figure 8 The calculation of block-based prediction coefficients is shown on the same 8-store MDCT coefficients.

[0139] Block-based and adaptive computation of combined prediction coefficients By dividing the computation of prediction filter coefficients into an adaptive least squares method during normal decoding (normal mode) and a block-based method during frame loss (hidden mode), peak computational complexity and battery consumption can be balanced. For example, for lower frequency ranges (e.g., up to a threshold selected in the 4-6 kHz range), prediction coefficients can be adaptively computed during normal operation, while the block-based method can be used for higher frequencies during frame loss. Therefore, for higher frequencies, computational complexity is reduced during normal operation, thus reducing the average computational load, while the peak computational load increases during packet loss. Preferably, the prediction filter length of the block-based method can be relatively small, which reduces the buffer size required to store past MDCT coefficients. For example, the prediction filter length can be selected based on perceptual considerations.

[0140] Operations in the first frame after hiding Possible quality improvements can be achieved by estimating (e.g., predicting) previous (hidden) time-frequency coefficients (e.g., MDCT coefficients) based on the current time-frequency coefficients. The right half of the time-domain window for the previous frame's time-frequency coefficients can then be modified by crossfading in and out with the right half of the time-domain window containing the estimated time-frequency coefficients. This can improve aliasing cancellation in the current frame.

[0141] Performance Results The Multistimulus Resonance Assessment (MUSHRA) hearing test with hidden references and anchors was conducted to evaluate the benefits of variations of the proposed method. Figure 9 and Figure 10 The test conditions are as follows: • Mute: Replaces lost frames with mute.

[0142] • mdct copy: a traditional method for repeating random symbols.

[0143] • MDCT Prediction (MDCT Pred): The proposed method using a 6-tap filter and least squares estimation.

[0144] • The method proposed by mdct prediction (low comp) uses a 3-tap filter below 6 kHz and the mdct_copy method above 6 kHz.

[0145] Figure 9 The illustration shows the results of a hearing test with random loss (Random Loss) where 7.5% of frames are lost in a random manner. Figure 10 The figure illustrates the results of a hearing test (burst loss) on a burst segment of 10 lost frames (with 100 good frames in between). It can be seen that the proposed method significantly outperforms traditional methods in both random and burst loss scenarios.

[0146] Device Finally, this disclosure also relates to apparatus (e.g., computer-implemented apparatus or apparatus with general processing capabilities) for performing or implementing the methods and techniques described herein. For example, such apparatus may relate to an audio decoder. The audio decoder may be, for example, a low-latency audio decoder suitable for general audio.

[0147] Figure 11An example of such a device 1100 is shown. Specifically, device 1100 includes a processor 1110 and a memory 1120 coupled to the processor 1110. Memory 1120 may store instructions for the processor 1110. Depending on the use case and / or implementation, among other things, the processor 1110 may also receive suitable input data 1130 (e.g., input frames of time-frequency coefficients, etc.). Depending on the use case and / or implementation, the processor 1110 may be adapted to perform or implement the methods / techniques described throughout this disclosure (e.g., Figure 2 Method 200 Figure 5 Method 500 and / or Figure 6 Method 600) and generate corresponding output data 1140 (e.g., prediction / output frames of time-frequency coefficients, etc.).

[0148] This disclosure also relates to corresponding computer programs, computer program products, and computer-readable storage media for storing such computer programs or computer program products.

[0149] explain Aspects of the methods and apparatus / systems described herein can be implemented in a suitable computer-based audio processing network environment (e.g., a server or cloud environment) to process digital or digitized audio files. A portion of the audio system may include one or more networks comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0150] One or more components, blocks, processes, or other functional units may be implemented by a computer program executed by a processor-based computing device controlling the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media such as optical, magnetic, or semiconductor storage media.

[0151] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules, which may be illustrated and described for discussion purposes as if most components were implemented solely in hardware. However, those skilled in the art, and based on a reading of this specific embodiment, will recognize that in at least one embodiment, the electronic aspects may be implemented in software (e.g., stored on a non-transitory computer-readable medium) executable by one or more electronic processors (e.g., microprocessors and / or application-specific integrated circuits (“ASICs”)). Therefore, it should be noted that embodiments may be implemented using multiple hardware and software-based devices and multiple different structural components. For example, the aforementioned apparatus (e.g., an encoder) may include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., system buses) connecting the various components.

[0152] While one or more embodiments have been described by way of example and specific examples, it should be understood that one or more embodiments are not limited to the disclosed embodiments. Rather, they are intended to cover various modifications and similar arrangements that will be obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation in order to cover all such modifications and similar arrangements.

[0153] Similarly, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” or “having,” and variations thereof, is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “coupled,” and variations thereof, are used extensively and cover direct and indirect installation, connection, support, and coupling.

[0154] The enumerated example embodiments Various aspects and implementations of this disclosure can also be understood from the following enumerated exemplary embodiments (EEE), which are not claims.

[0155] EEE1. A method for packet loss concealment in frame-based audio, comprising: In the first mode, when the time-frequency coefficients of the current frame of the audio are available, the time-frequency coefficients of the current frame are stored in a buffer; and In the second mode, when the time-frequency coefficients of the current frame are unavailable, the time-frequency coefficients of the current frame are predicted based on a set of prediction parameters stored in the buffer and the time-frequency coefficients of one or more previous frames, and the predicted time-frequency coefficients are stored in the buffer.

[0156] EEE2. According to the method of EEE1, wherein the set of prediction parameters includes a plurality of prediction filter coefficients, the plurality of prediction filter coefficients including, for each of a plurality of frequency bands, prediction filter coefficients associated with corresponding time-frequency coefficients of one or more previous frames in the frequency band and in one or more adjacent frequency bands.

[0157] EEE3. The method according to EEE2, wherein the prediction of the time-frequency coefficients of the current frame in a given frequency band is based on the time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands, and based on the prediction filter coefficients associated with the corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands.

[0158] EEE4. The method described according to EEE2 or EEE3 when subordinate to EEE2, wherein the current frame and frequency band The predicted time-frequency coefficients Given by the following formula

[0159] in, It is frequency band Relative to frame In frequency band Time-frequency coefficients in The predicted filter coefficients, Indicates the number of prior frames to be used for the prediction, and Indicates the number of adjacent frequency bands to be used for the prediction.

[0160] EEE5. The method according to any one of EEE1 to EEE4, further comprising: in the first mode, updating the set of prediction parameters based on the time-frequency coefficients of the current frame.

[0161] EEE6. The method according to EEE5, wherein the prediction parameters are updated based on the time-frequency coefficients of a predetermined number of frames, including the current frame and previous frames.

[0162] EEE7. The method according to EEE5 or EEE6 when subordinate to EEE2, wherein, in the first mode, for a given frequency band, the updated prediction filter coefficients are determined based on the current prediction filter coefficients of the given frequency band and the corresponding adjustment amount of the current prediction filter coefficients.

[0163] EEE8. The method according to EEE7, wherein the adjustment amount of the given frequency band is determined using least squares, based on the current prediction filter coefficients associated with the corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and the one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of the one or more previous frames in the given frequency band and the one or more adjacent frequency bands of the given frequency band.

[0164] EEE9. The method according to EEE7 or EEE8, wherein determining the adjustment amount for the given frequency band includes: Predict the time-frequency coefficients of the current frame in the given frequency band based on the adjustment amount; and For a given change in the adjustment amount of the given frequency band, minimize the squared prediction error between the predicted time-frequency coefficients of the current frame in the given frequency band and the time-frequency coefficients of the current frame in the given frequency band.

[0165] EEE10. The method according to EEE7 or EEE8, wherein determining the adjustment amount for the given frequency band includes: Based on the current prediction filter coefficients associated with corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands of the given frequency band, a probabilistic model is determined for the time-frequency coefficients of the current frame in the given frequency band; and The adjustment amount for the given frequency band is determined based on the probability model and the time-frequency coefficients of the current frame in the given frequency band.

[0166] EEE11. The method according to EEE10, wherein the adjustment amount of the given frequency band and the given current prediction filter coefficients is determined based on the probability model, the time-frequency coefficients of the current frame in the given frequency band, and the time-frequency coefficients associated with the given current prediction filter coefficients.

[0167] EEE12. The method according to EEE6 or any EEE belonging to EEE6, wherein the predicted time-frequency coefficients in the buffer are not used to update the prediction parameters.

[0168] EEE13. The method according to any one of EEE2 or EEE3 and EEE4 when subordinate to EEE2, further comprising: in the second mode, using least squares to determine the prediction filter coefficients based on a predetermined number of time-frequency coefficients from previous frames.

[0169] EEE14. The method according to EEE13, wherein the prediction filter coefficients of a given frequency band among the plurality of frequency bands are determined using the least squares method based on the time-frequency coefficients of the predetermined number of previous frames in the given frequency band and in the one or more adjacent frequency bands.

[0170] EEE15. The method according to EEE14, wherein the least squares error to be minimized for the given frequency band is determined based on the sum of prediction errors of the time-frequency coefficients of the predetermined number of previous frames, each predicted time-frequency coefficient being based on the time-frequency coefficients of one or more preceding frames of the corresponding frame in the predetermined number of previous frames.

[0171] EEE16. The method according to any one of the preceding EEEs, wherein predicting the time-frequency coefficients of the current frame further comprises: limiting the predicted time-frequency coefficients of a given frequency band to a range based on the cumulative per-band value of the signal power in the given frequency band.

[0172] EEE17. The method according to any one of the preceding EEEs further comprises: in the second mode, generating, for each of the plurality of frequency bands, a weighted sum of the predicted time-frequency coefficients of the current frame in that frequency band and the latest available time-frequency coefficients multiplied by random symbols.

[0173] EEE18. The method according to EEE17, wherein the weights of the weighted sum in a given frequency band are determined based on one or more of the following: the per-band value of the signal power in the given frequency band, the per-band value of the predicted signal power in the given frequency band, the per-band value of the prediction error of the predicted signal power in the given frequency band, and / or the per-band value of the log-relative noise level in the given frequency band.

[0174] EEE19. According to the method of EEE18, wherein, for consecutive predicted frames, the weights of the weighted sum decrease successively.

[0175] EEE20. The method according to any one of EEE17 to EEE19 further comprises: for a pair of channels, adding the latest available time-frequency coefficient of one channel multiplied by a random symbol to the weighted sum of the other channel.

[0176] EEE21. The method according to EEE20, wherein the mixing value is determined based on a per-band value of the signal cross product relative to the channel pair.

[0177] EEE22. The method according to EEE2 or any EEE belonging to EEE2, wherein the plurality of frequency bands correspond to the lower frequency portion of the frequency range of the time-frequency coefficients; and The method further includes: for at least one frequency band not located in the lower frequency portion, multiplying the latest available time-frequency coefficient in the at least one frequency band by a random symbol and using it as the time-frequency coefficient of the current frame in the at least one frequency band.

[0178] EEE23. An apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is adapted to perform a method according to any one of EEE1 to EEE22.

[0179] EEE24. A program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of EEE1 to EEE22.

[0180] EEE25. A computer-readable storage medium storing a program according to EEE24.

Claims

1. A method for packet loss concealment in frame-based audio, comprising: In the first mode, when the time-frequency coefficients of the current frame of the audio are available, the time-frequency coefficients of the current frame are stored in a buffer; In the second mode, when the time-frequency coefficients of the current frame are unavailable, the time-frequency coefficients of the current frame are predicted based on a plurality of prediction filter coefficients stored in the buffer and the time-frequency coefficients of one or more previous frames, and the predicted time-frequency coefficients are stored in the buffer. And in the first mode, the plurality of prediction filter coefficients are updated based on the time-frequency coefficients of the one or more previous frames and the time-frequency coefficients of the current frame; or Before predicting the time-frequency coefficients of the current frame, in the second mode, the plurality of prediction filter coefficients are updated based on the time-frequency coefficients of the one or more previous frames.

2. The method according to claim 1, wherein, The plurality of prediction filter coefficients include, for each of the plurality of frequency bands, prediction filter coefficients associated with the corresponding time-frequency coefficients of one or more previous frames in that frequency band and in one or more adjacent frequency bands.

3. The method according to claim 2, wherein, The prediction of the time-frequency coefficients of the current frame in a given frequency band is based on the time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands, and on the prediction filter coefficients associated with the corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands.

4. The method according to claim 2 or claim 3, wherein, Current frame and frequency band The predicted time-frequency coefficients Given by the following formula in, It is frequency band Relative to frame In frequency band Time-frequency coefficients in The predicted filter coefficients, Indicates the number of prior frames to be used for the prediction, and Indicates the number of adjacent frequency bands to be used for the prediction.

5. The method according to any one of the preceding claims, wherein, The predicted time-frequency coefficients in the buffer are not used to update the multiple prediction filter coefficients.

6. The method according to any one of the preceding claims, wherein, In the first mode, updating the plurality of prediction filter coefficients includes: determining a prediction filter coefficient update for the given frequency band based on the current prediction filter coefficients associated with the corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands of the given frequency band.

7. The method according to claim 6, wherein, The prediction filter coefficients are updated based on the least squares method.

8. The method according to claim 7, wherein, The least squares method includes: determining the prediction filter coefficient update based on the gradient of the squared prediction error between the predicted time-frequency coefficients of the current frame in the given frequency band and the time-frequency coefficients of the current frame in the given frequency band.

9. The method according to claim 8 when subordinate to claim 4, wherein, The prediction filter coefficient update is given by the following formula. in, It is a positive gain, and 。 10. The method according to claim 9, wherein, It depends on the physical time step of the time-frequency transformation used to determine the time-frequency coefficients.

11. The method according to claim 9 or 10, wherein, in, It is a predetermined parameter that is less than one.

12. The method of claim 6, wherein, Determining the prediction filter coefficient update for the given frequency band includes: Based on the current prediction filter coefficients associated with corresponding time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands of the given frequency band, and based on the time-frequency coefficients of the one or more previous frames in the given frequency band and in the one or more adjacent frequency bands of the given frequency band, a probabilistic model is determined for the time-frequency coefficients of the current frame in the given frequency band; and Based on the probability model and the time-frequency coefficients of the current frame in the given frequency band, the prediction filter coefficient update for the given frequency band is determined.

13. The method of claim 12, wherein, The probability model corresponds to the normal distribution.

14. The method according to claim 13 when subordinate to claim 4, wherein, The mean of the normal distribution is given by the following formula. Furthermore, the variance of the normal distribution is given by the following formula. in, It is a current estimate of the noise level relative to the signal power. The logarithmic control parameters.

15. The method according to claim 13 or 14, wherein, The normal distribution is optimized based on non-negative log-likelihood loss.

16. The method of claim 14, wherein, The prediction filter coefficient update is given by the following formula. in, The update is given by the following formula. in, in, It is a predetermined parameter less than one, and and Proportional.

17. The method according to claim 1, wherein, In the second mode, updating the plurality of prediction filter coefficients based on the time-frequency coefficients of the one or more previous frames includes: determining the prediction filter coefficients by minimizing the sum of least-squares prediction errors for the time-frequency coefficients of a predetermined number of the one or more previous frames, wherein each predicted time-frequency coefficient is based on the time-frequency coefficients of one or more preceding frames of the corresponding frame in the predetermined number of the one or more previous frames.

18. The method according to claim 17 when subordinate to claim 2, wherein, The predetermined number of time-frequency coefficients of the one or more previous frames are time-frequency coefficients in a given frequency band and in the one or more adjacent frequency bands.

19. The method according to claim 1, wherein, The prediction filter coefficients are updated in both the first mode and the second mode, wherein, in any one of claims 7 to 11, the update of the prediction filter coefficients in the first mode is performed for the lower frequency portion of the frequency range of the time-frequency coefficients, and in any one of claims 17 to 18, the update of the prediction filter coefficients in the second mode is performed for the higher frequency portion of the frequency range of the time-frequency coefficients.

20. The method of claim 19, wherein, The threshold used to determine the lower frequency portion and the higher frequency portion is between 4 kHz and 6 kHz.

21. The method according to any one of the preceding claims, wherein, Predicting the time-frequency coefficients of the current frame further includes limiting the predicted time-frequency coefficients of a given frequency band to a range based on the cumulative per-band value of the signal power in the given frequency band.

22. The method according to any one of the preceding claims, further comprising, in the second mode, generating, for each of the plurality of frequency bands, a weighted sum of the predicted time-frequency coefficients of the current frame in that frequency band and the latest available time-frequency coefficients multiplied by random symbols.

23. The method according to claim 22, wherein, The weights of the weighted sum in a given frequency band are determined based on one or more of the following: the per-band value of the signal power in the given frequency band, the per-band value of the predicted signal power in the given frequency band, the per-band value of the prediction error of the predicted signal power in the given frequency band, and / or the per-band value of the log-relative noise level in the given frequency band.

24. The method according to claim 22 or 23, wherein, For consecutive predicted frames, the weights of the weighted sum decrease successively.

25. The method according to any one of claims 22 to 24, further comprising, for a pair of channels, adding the latest available time-frequency coefficient of one channel multiplied by a random symbol to the weighted sum of the other channel.

26. The method of claim 25, wherein, The mixed value is determined based on the per-band value of the signal cross product relative to the channel pair.

27. The method according to claim 2 or any claim dependent on claim 2, wherein, The plurality of frequency bands correspond to the lower frequency portion of the frequency range of the time-frequency coefficients; and The method further includes: for at least one frequency band not located in the lower frequency portion, multiplying the latest available time-frequency coefficient in the at least one frequency band by a random symbol and using it as the time-frequency coefficient of the current frame in the at least one frequency band.

28. An apparatus comprising a processor and a memory, the memory being coupled to the processor and storing instructions for use with the processor, wherein, The processor is adapted to perform the method according to any one of claims 1 to 27.

29. A program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 27.

30. A computer-readable storage medium storing the program according to claim 29.

Images