How to perform packet loss concealment in a complex filter bank domain

Abstract

This disclosure relates to a method and system for concealing lost or corrupted frames of audio content in a sequence of audio frames. The method includes the steps of obtaining at least one first frame related to a frame sequence and determining whether a second frame following at least one first frame is valid or invalid. When the second frame is identified as invalid, the method includes the step of generating a TF tile value set for a replacement frame to replace the second frame by applying a sinusoidal expansion process to generate a time-frequency (TF) tile value set for a replacement frame based on the degree of tonality of at least one preceding frame, or by applying a linear prediction process to generate a TF tile value set for a replacement frame.

Description

[Technical Field]

[0001] This disclosure relates, in general terms, to systems, apparatus, and methods for processing audio. [Background technology]

[0002] Unless otherwise specified, the approaches described in this chapter are not prior art to the claims of this application, and their inclusion in this chapter does not make them prior art.

[0003] There are many types of codecs for encoding audio content into a format suitable for inclusion in a bitstream. For example, there are several codecs that operate in the frequency domain, where segments of a time-domain audio signal are converted into vector-value samples, each sample represented in the encoded bitstream by a set of time-frequency (TF) tiles combined with some side information. Each TF tile is associated with a specific frequency band, and at the decoding end, the samples carrying the TF tiles are reconstructed by decoding the bitstream, and then the time-domain samples are regenerated by applying an inverse transform to the TF tiles.

[0004] There are many types of time-frequency transforms (and their corresponding inverse transforms). Examples include the Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), Modified Discrete Cosine Transform (MDCT), or sub-band techniques that represent the original time-domain signal by sampling in the sub-band region. One example of a sub-band transform is the use of a Quadrature Mirror Filter (QMF) bank, which has the property that each bandpass (or sub-band) signal can be critically sampled so that the inverse operation, which involves a complete reconstruction of the original time signal, is still possible.

[0005] The QMF-based transformation class operates in the complex-valued domain, in which case a time-domain signal is said to be represented in the CQMF domain after being transformed into the complex-valued domain. In some cases, CQMF banks are designed to introduce minimum latency in both the forward (CQMF analysis) and reverse (CQMF synthesis) directions, making CQMF banks particularly useful in conversational voice applications and applications that benefit from low latency, such as telephone or video conferencing applications.

[0006] A specific type of CQMF bank is used in the codec standardized for the 3GPP® Immersive Voice and Audio Services (IVAS) codec. This CQMF bank is called a Complex Low-Delay Filter Bank (CLDFB), and according to the CLDFB design, when CLDFB analysis is applied to a 48kHz sampled audio signal, one TF tile (represented by one TF tile value) represents a frequency band of 400Hz bandwidth, and a CLDFB sample covers a time slot of 1.25ms.

[0007] Services like 3GPP IVAS are often deployed over error-prone radio channels between transmitters and receiver units. In practice, a TF tile of multiple CQMF samples is called a frame, which is encoded as a block of data, with each frame containing a CQMF sample representing audio content for a predetermined period (e.g., 20ms). Coated frames are transmitted in packetized batches of one or more coded frames transmitted over the radio channel. However, there is a risk that one or more frames may be corrupted or not delivered to the receiver unit at all, resulting in what is called "packet loss."

[0008] Several error detection mechanisms (such as cyclic redundancy checks, CRCs) may be applied in the receiving unit to detect whether a packet (carrying one or more frames) was received without errors or whether the packet is corrupted. In some cases, the packet may arrive too late to be used by the receiving unit. In either case, only the audio frames contained in packets received in a timely and error-free manner can be used for audio decoding and extraction of time-sensitive audio signals.

[0009] For frames that cannot be used for decoding, the receiving unit may use techniques to generate alternative frames. One technique for forming an alternative frame is to use a zero frame (i.e., a frame in which each spectral coefficient is equal to zero) as the alternative signal. Another technique is to repeat the most recent frame received without error.

[0010] The technique of generating alternate frames is called frame loss concealment or packet loss concealment (PLC). The general purpose of these techniques is to minimize the impact on audio quality at the receiver by making frame loss as inaudible as possible. When relying on waveform coding, packet loss concealment may be based on frame iteration techniques, or when relying on parametric coding, the coding parameters of the most recent good frame are iterated over.

[0011] Furthermore, various PLC technologies are known, for example, from the 3GPP codec for Enhanced Voice Services (EVS). This codec includes numerous PLC technologies that can be selected depending on whether speech coding technology (ACELP) or audio coding technology (transform coding) is used for coding the signal.

[0012] The disclosures made in this specification are presented with respect to these and other considerations. [Overview of the Initiative]

[0013] Techniques for processing audio signals are described. Various embodiments described in this specification provide apparatus, systems, and methods for mitigating the effects of loss frames that may occur in the transmission of audio signals. Various examples are described of improving the concealment of loss frames by processing audio in a Complex Quadrature Mirror Filter (CQMF) domain.

[0014] The drawbacks of existing concealment techniques (e.g., PLC) are that they provide lower quality than desired and often introduce perceptible audio artifacts. Furthermore, many techniques are computationally complex (e.g., in terms of computation time, memory requirements, and number of computations), require powerful processing hardware, and / or increase the latency of the audio encoding-decoding chain. At the same time, audio decoding is frequently performed on very power-constrained end devices such as earphones or AR glasses, which means that many concealment techniques are directly unsuitable for achieving low latency and high-quality concealment in some applications. For this purpose, this disclosure identifies an improved concealment technique that can recreate high-quality alternate frames with reduced complexity operation.

[0015] According to a first aspect of the present invention, a method is provided for concealing a lost or corrupted frame with audio content within a frame sequence. The method includes the steps of: obtaining at least one first frame associated with a frame sequence, wherein the at least one first frame is either a frame in the frame sequence or a generated replacement frame; and identifying whether a second frame following the at least one first frame is valid or invalid. If the second frame is identified as invalid, the method includes the steps of: analyzing the at least one first frame to determine its tonality; comparing the tonality to a threshold to determine whether the tonality exceeds the threshold; thereby generating a set of time-frequency (TF) tile values ​​for a replacement frame to replace the second frame. If the tonality exceeds the threshold, the method includes the steps of: applying a sinusoidal expansion process to generate a set of TF tile values ​​for the replacement frame; and applying a linear prediction process to generate a set of TF tile values ​​for the replacement frame if the tonality does not exceed the threshold. The method further includes the step of outputting a replacement frame to replace the second frame.

[0016] Frames that are successfully received and therefore determined to be undamaged during decoding are identified as valid frames. Frames that are determined to be damaged, or that were not received, or that are otherwise unavailable during decoding (e.g., lost), are identified as invalid frames. For invalid frames at least, measures are taken to generate replacement frames to conceal them.

[0017] For the purpose of generating a set of TF tile values ​​for a replacement frame, the preceding first frame may be a replacement frame for a preceding frame that was previously determined to be a valid frame or an invalid frame.

[0018] In the first embodiment of the method, the TF tile value of the replacement frame is generated adaptively by adaptively selecting one of two processes for generating the TF tile value of the replacement frame. The choice of which of the two processes to use, sinusoidal expansion or linear prediction, is based on the degree of tonality of the preceding (first) frame. When the degree of tonality is higher, sinusoidal expansion is expected to generate a more accurate TF tile value, so sinusoidal expansion is used when the degree of tonality exceeds a threshold, and when the degree of tonality is lower, linear prediction is used when the degree of tonality is below a threshold, so linear prediction is expected to generate a more accurate TF tile value. In this way, it is possible to generate TF tile values ​​that result in a more realistic and reliable replacement frame (which functions as a continuation of the preceding first frame), which results in higher quality PLC technology.

[0019] There are various methods for quantifying the quality of PLC (Power Line Communication) technology. For example, a subjective listening test is performed in which the listener scores the quality of the audio signal generated through a simulation of a transmission system prone to errors that result in frame loss, and the PLC technology being tested is used to mask the perceived effect of frame loss. The subject may be asked to score the perceived clarity or quality of the audio. Using these tests, different PLC technologies can be compared based on their scores. As an example, the signal discontinuity that appears when invalid frames are replaced with generated replacement frames is known to cause perceptual artifacts, and thereby the quality of PLC technology can be roughly quantified by analyzing the smoothness of the audio signal in the transition between valid frames and generated replacement frames that replace invalid frames.

[0020] A second aspect of the present invention provides a method for concealing a lost or corrupted frame in a Complex Quadrature Mirror Filter (CQMF) domain. The method includes the step of receiving at least one first frame of a CQMF sample, each CQMF sample spanning one or more time-frequency (TF) tiles, each TF tile associated with a frequency band and a complex TF tile value, respectively. The method further includes the step of determining whether a second frame of the CQMF sample following at least one first frame is valid or invalid. If the second frame is identified as invalid, the method includes the steps of determining a complex parameter for at least one of the frequency bands of each of the one or more TF tiles of the CQMF sample in at least one first frame, based on at least two of the TF tile values ​​in at least one first frame, and generating a replacement TF tile value by modifying at least one TF tile value of the first frame using the complex parameter. The method further includes the steps of forming a replacement frame for the second frame based on the replacement TF tile value, and outputting the replacement frame.

[0021] In the CQMF domain, the PLC technology of this disclosure can be computationally efficient. A complex parameter based on at least two TF tile values ​​of at least one preceding frame may be used to generate a set of TF tile values ​​for the replacement frame. In some implementations, the same complex parameter is used iteratively to generate two or more (or all) TF tile values ​​for the replacement frame.

[0022] According to a third aspect of the present invention, a device is provided which includes a processor and memory, and the device is configured to perform the method of the first or second aspect.

[0023] According to a fourth aspect of the present invention, a non-temporary medium storing software is provided, the software including instructions for controlling one or more devices to perform the method of the first or second aspect.

[0024] According to a fifth aspect of the present invention, a computer program product is provided which, when the program is executed by a computer, includes instructions that cause the computer to execute the method of either the first or second aspect.

[0025] A sixth aspect of the present invention provides a receiving unit including a decoder configured to receive a bitstream. The bitstream includes data packets, each data packet carrying a frame containing a plurality of time-frequency (TF) tile values, and the decoder is configured to decode the received data packets of the bitstream to obtain the TF tile value of the current frame of the current data packet. The receiving unit further includes a frame replacement module configured to identify whether the current frame is invalid and, in response to identifying that the current frame is invalid, generate a set of TF tile values ​​for a replacement frame to replace the current invalid frame. The frame replacement module includes a tonality extractor configured to analyze at least one preceding frame preceding the current frame and determine the degree of tonality of the preceding frame, and an adaptive sine wave expansion and linear prediction module configured to determine whether the degree of tonality exceeds a threshold, and, if the degree of tonality exceeds a threshold, generate a set of TF tile values ​​for a replacement frame using a sine wave expansion process, and, if the degree of tonality does not exceed a threshold, generate a set of TF tile values ​​for a replacement frame using a linear prediction process. The receiving unit further includes a synthetic filter bank configured to receive the TF tile values ​​of the replacement frame and convert the TF tile values ​​of the replacement frame into time-domain audio segments.

[0026] The embodiments described herein may generally be described as "technical," where the term "technical" may refer to systems, apparatus, methods, computer-readable instructions, modules, components, hardware logic, and / or operations, as suggested by the context in which it applies to this specification.

[0027] Features and technical benefits other than those explicitly stated above will become apparent upon reading the following detailed description and related drawings. This summary is provided to introduce the selection of technology in a simplified form and is not intended to identify key or essential features of the claimed subject matter as defined by the attached claims.

[0028] The inventions of the third, fourth, fifth, and sixth embodiments are characterized by the same or equivalent points as the inventions of the first or second embodiment. Any function described in relation to the method may have corresponding features in a device or computer program product. [Brief explanation of the drawing]

[0029] The present invention will be described in further detail with reference to the accompanying drawings.

[0030] [Figure 1a] This is a block diagram showing an encoder and decoder having a frame replacement module.

[0031] [Figure 1b] This is a block diagram showing a frame replacement module with adaptive selection of sinusoidal extension and linear prediction.

[0032] [Figure 2] This is a diagram showing a frame containing multiple samples.

[0033] [Figure 3] This diagram shows a frame sequence where one frame is invalid and is replaced by a generated replacement frame.

[0034] [Figure 4] This flowchart shows the processing in a frame replacement module using several implementation methods.

[0035] [Figure 5] This is a diagram illustrating the sine wave expansion process.

[0036] [Figure 6a] This is a diagram showing the linear prediction process.

[0037] [Figure 6b] This is a diagram showing the linear prediction process.

[0038] [Figure 6c] This is a diagram showing the linear prediction process.

[0039] [Figure 6d] This is a diagram showing the linear prediction process.

[0040] [Figure 7] This diagram illustrates a linear prediction process using a weighted coupling with alternative frames.

[0041] [Figure 8] This diagram shows the crossfade from the generated replacement frame to the subsequent valid frame.

[0042] [Figure 9] A schematic block diagram of an exemplary apparatus or architecture that may be used to carry out embodiments of the present invention is shown. [Modes for carrying out the invention]

[0043] The following description includes numerous details, such as system, device configuration, timing, operation, etc., to provide an understanding of one or more aspects of the present disclosure. It will be immediately apparent to those skilled in the art that these specific details are merely examples and are not intended to limit the scope of the application.

[0044] In this specification, the terms “and,” “or,” and “and / or” are used. Such terms should be interpreted as having an inclusive meaning. For example, “A and B” may mean at least: “both A and B,” or “at least both A and B.” For example, “A or B” may mean at least: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” For example, “A and / or B” may mean at least: “A and B,” or “A or B.” Particular attention should be paid to this when an exclusive OR is intended (e.g., “either A or B,” or “at most one of A and B”).

[0045] The term "includes" and its variations shall be interpreted as a broad term meaning "includes, but not limited to." The terms "one exemplary implementation" and "exemplary implementation" should be interpreted as "at least one exemplary implementation." The term "another implementation" should be interpreted as "at least one other implementation." The terms "determined," "determine," or "to determine" should be interpreted as "to obtain," "to receive," "to calculate," "to calculate," "to estimate," "to predict," or "to derive." Furthermore, in the following descriptions and claims, unless otherwise noted, all technical and scientific terms used in this specification shall have the same meaning as those generally understood by those skilled in the art to which this disclosure belongs.

[0046] This specification describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Generally, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0047] Throughout this disclosure and in the relevant claims and / or drawings, various acronyms may appear, which are listed below. Other commonly used acronyms and technical terms may be omitted from this list for brevity. A short list of acronyms is provided below for the reader's easy reference. IVAS: Immersive Voice and Audio Services PLC: Packet Loss Concealment CRC: Cyclic Redundancy Check QMF: Quadrature Mirror Filter CQMF: Complex Quadrature Mirror Filter CLDFB: Complex Low Delay Filter Bank ACELP: Algebraic code-excited linear prediction 3GPP: 3rd Generation Partnership Project EVS: Enhanced Voice Services IP: Internet Protocol UDP: User Datagram Protocol RTP: Real-time Transport Protocol BFI: Bad Frame Indicator

[0048] Figure 1a schematically shows the transmitting unit 11 and the receiving unit 12. The transmitting unit 11 includes an analysis filter bank 13 and an encoder 18. The receiving unit 12 includes a decoder 19, a frame replacement module 15, and a composite filter bank 14.

[0049] In the transmitting unit 11, the analysis filter bank 13 is configured to acquire a time-domain audio segment of an audio signal and, accordingly, convert the time-domain audio segment into a time-frequency (TF) representation. The time-domain audio segment contains time-domain samples of the audio signal, and the analysis filter bank 13 converts the time-domain audio samples into TF samples. Each TF sample may represent multiple time-domain samples. Each TF sample may contain multiple TF tiles, each TF tile representing its respective frequency band. The encoder 18 is configured to receive TF samples in TF representation and, in response, encode the TF samples into a bitstream, which may be output for use by the receiving unit 12. The encoder 18 may form a bitstream by collecting a predetermined number of subsequent TF samples to form a frame of TF samples, thereby coding the frame into data packets that constitute the bitstream.

[0050] In the receiving unit 12, the decoder 19 is configured to receive data packets from the bitstream and, accordingly, decode the data packets from the bitstream to hold frames of TF samples. The frame replacement module 15 is configured to receive frames of TF samples from the decoder 19, and if the frames of TF samples are invalid, the frame replacement module 15 generates replacement frames of TF samples. The composite filter bank 14 receives frames of TF samples (or their replacement frames) and is configured to generate time-domain samples of time-domain audio segments for each TF sample. The composite filter bank 14 in the receiving unit 12 is configured to perform inverse filtering operations on the analysis filter bank 13 of the transmitting unit 11.

[0051] In summary, the encoding and decoding processes used by the system in Figure 1a can be described as follows: A time-domain audio segment received by the transmitting unit 11 is processed by the analysis filter bank 13 to obtain TF samples in the TF domain. The TF samples are grouped into frames, which are encoded into a bitstream by the encoder 18 and then output (e.g., stored, transmitted, carried). The bitstream is received by the receiving unit 12 and processed by the decoder 19 to obtain frames of TF samples. The frames of TF samples are processed by the frame replacement module 15 to generate replacement frames with TF samples when needed. The replacement frames of TF samples are processed by the synthesis filter bank 14 to generate time-domain samples of the restored time-domain audio segment. As will be further described below, the function of the frame replacement module 15 is to ensure that the adverse effects of frames with unreceived, corrupted, or otherwise unavailable TF samples are mitigated by generating replacement frames to replace frames that are unreceived, corrupted, or otherwise unavailable.

[0052] In some implementations, the analysis filter bank 13 is a Quadrature Mirror Filter (QMF) bank, and thereafter, the TF sample is a QMF sample in the QMF domain.

[0053] In some other implementations, the analysis filter bank 13 is a CQMF (Complex Quadrature Mirror Filter) bank, such as the CLDFB (Complex Low-Delay Filter Bank) specified in the 3GPP IVAS (Immersive Voice and Audio Services) codec. In such implementations, the TF samples are CQMF samples in a CQMF domain (such as the CLDFB domain).

[0054] The analysis filter bank 13 processes the audio segment received in the time domain. The time domain audio segment consists of time domain audio samples. The audio samples in the time domain are converted to a TF representation in the time-frequency domain, for example, by the Fourier transform method as described above. The TF representation is then used in the output or transmitted within a bitstream containing the TF representation samples. For example, each TF representation sample in the bitstream may correspond to a QMF or CQMF sample, as will be described in more detail below.

[0055] Each analysis filter bank 13 consists of multiple filters, each having a passband defined over a specific frequency range. For example, a simple analysis filter bank with three filters may include a low-pass filter having a passband for frequencies lower than a first frequency (f1), a band-pass filter having a passband for frequencies between the first frequency (f1) and a second frequency (f2), and a high-pass filter having a passband for frequencies higher than the second frequency (f2). Each TF sample includes one or more TF tiles, each TF tile may correspond to a specific frequency range, and each TF tile may be mapped to a corresponding frequency band for a specific filter in the analysis filter bank 13. When a filter from the analysis filter bank 13 is applied to a time-domain audio segment, the time-domain samples of the audio segment are multiplied by the filter characteristics (e.g., filter coefficients for the passband or stopband, or their interpolation) to obtain an output value for each time-domain sample. The output values ​​may then be decimated. Optionally, the decimated output values ​​form TF tile values ​​that each occupy a TF tile of the TF sample.

[0056] When the QMF bank is used as the analysis filter bank 13, the output from the analysis filter bank 13 is in the form of QMF samples, each of which represents a portion of the audio content. Each QMF sample includes a plurality of TF tiles, with each TF tile representing a respective frequency band. Thus, a QMF sample can include a plurality of TF tiles, and each TF tile maps to a specific range of frequencies as described above. The number of TF tiles within a QMF sample corresponds to the number of filters in the analysis filter bank 13. Therefore, when a time-domain audio segment is processed by the analysis filter bank 12, each filter within the analysis filter bank 12 outputs a corresponding stream of TF tile values associated with the frequency band of that particular filter.

[0057] That is, each QMF sample corresponds to a time slot of the time-domain audio segment, and each of one or more TF tile values of the same QMF sample (e.g., the same time slot) represents the frequency band of that time slot in the time-domain audio segment.

[0058] The specific passbands of the filters within the analysis filter bank 13 may not overlap, or may partially overlap. For example, each QMF filter can be defined as a band-pass filter having a passband centered on a unique center frequency. A first exemplary band-pass filter can include two frequencies (f1, f2) that define a first bandwidth (BW1) of a first passband, while an adjacent second band-pass filter can include two frequencies (f3, f4) that define a second bandwidth (BW2) of a second passband. If the exemplary passband frequencies are defined such that f1 < f2 < f3 < f4, the first passband and the second passband do not overlap. However, if the exemplary passband frequencies are defined such that f1 < f3 < f2 < f4, the first passband and the second passband partially overlap.

[0059] The analysis filter bank 13 may alternatively include a CQMF bank. Each filter in the CQMF bank outputs a stream of complex-valued TF-tile values, which are grouped together in time to form a CQMF sample. As a further example, if the analysis filter bank 13 is CLDFB of the 3GPP IVAS codec, each filter has a 400 Hz passband and outputs substantially non-overlapping complex TF-tile values.

[0060] Since each filter in the analysis filter bank 13 is relatively narrowband (compared to a full-band audio representation, for example, when sampled at 48kHz), it is generally understood that the analysis filter bank 13 contains a large number of filters, such as 10 or more filters, or even 100 or more filters. For example, if CLDFB is used as the analysis filter bank 13, there could be 60 filters, each filter having a 400Hz non-overlapping passband used to convert a time-domain segment to a CLDFB sample, and each CLDFB sample containing 60 complex TF tile values.

[0061] QMF (e.g., CQMF) samples output by the analysis filter bank 13 may be encoded by the encoder 18 and then transmitted to the receiving unit 12 (e.g., wirelessly), stored, or otherwise transmitted. As described above, the encoder 18 may group multiple QMF (e.g., CQMF) samples into frames and encode each frame as a coded data packet.

[0062] The decoder 19 is configured to decode the bitstream by decoding coded data packets in order to obtain frames of QMF samples. The QMF samples of the decoded data packet frames are provided to a composite filter bank 14 which reconstructs samples of time-domain audio segments. The composite filter bank 14 in the receiving unit 12 is configured to perform an inverse filtering operation against the analysis filter bank 13 in the transmitting unit 11.

[0063] The sequence of coded data packets (each packet carrying a frame of QMF samples) is then delivered to the receiving unit 12 via a bitstream, and the decoder 19 performs a decoding operation to obtain QMF samples of the coded data packets. In general, coding QMF samples is a lossy operation (due to, for example, quantization errors), and the QMF samples obtained after decoding may be approximations of the coded QMF samples.

[0064] If the receiving unit 12 successfully receives all data packets (i.e., all frames) associated with the time-domain audio segment, all the information is available for the composite filter bank 14 to reconstruct the time-domain audio segment (or at least an approximation of it), enabling the receiving unit 12 to reconstruct the time-domain audio segment (or at least an approximation of it). However, if one or more data packets associated with the time-domain audio signal are not received by the receiving unit 12, or if one or more received data packets are determined to be corrupted, the receiving unit 12 cannot access all frames to fully reconstruct the corresponding time-domain audio segment. For this purpose, the frame replacement module 15 is used to generate a set of TF tile values ​​for replacement frames, which replace one or more frames of data packets that were not received or were corrupted.

[0065] In Figure 1a, the frame replacement module 15 is shown as a separate module from the decoder 19. However, this is merely an example, and the frame replacement module 15 may be an integral part of the decoder 19. Generally, the decoder 19 and the frame replacement module 15 are implemented as hardware, firmware, or software within a client device or server that functions as a receiving unit 12.

[0066] Figure 2 schematically shows a frame 20 of a QMF (e.g., CQMF) sample. Frame 20 contains multiple TF tiles 21, which are grouped into multiple QMF samples 22 having time indices n=0, .,N-1. Each QMF sample 22 contains F TF tiles 21, and each TF tile 21 of a QMF sample has its own frequency band B i It is associated with the index i=1,2,.,F.

[0067] In the example in Figure 2, each frequency band B i This is represented by row 23, and all TF tiles 21 in the same row contain TF tile values ​​associated with the same frequency band and the same corresponding QMF in the QMF bank used to extract the TF tile values.

[0068] The frame replacement module is configured to operate using a frame 20, as shown in Figure 2. For example, the frame replacement module is configured to generate TF tile values ​​to insert all TF tiles 21 of the replacement frame (if there is one or more preceding frames) in order to replace a frame that is lost or otherwise unavailable.

[0069] The exemplary frame 20 in Figure 2 contains five frequency bands, i.e., F=5, and five QMF samples 22, i.e., N=5, resulting in a total of 25 TF tiles 21. It should be understood that the frame 20 shown in Figure 2 is merely an example, and the frame size can be determined almost arbitrarily depending on the implementation. For example, the Complex Low-Delay Filter Bank (CLDFB) used in the 3GPP IVAS codec uses 60 filters, each filter having a bandwidth of 400 Hz, and each sample 22 represents 1.25 ms. A typically used frame size is a total of 8*60=480 TF tiles 21 and 16*60=960 TF tiles 21 in frame 20, representing 10-20 ms of audio content, or approximately 8-16 CLDFB samples. CLDFB is applied to a 48kHz sampled audio signal, thereby capturing the lower 24kHz of the 48kHz sampled audio signal with 60 filters with a 400Hz bandwidth. However, since the frequency spectrum of a 48kHz audio signal is symmetrical with respect to half the sampling frequency (24kHz, also known as the aliasing frequency), the frequency spectrum above 24kHz can be reconstructed after decoding by mirroring the frequency spectrum of lower frequencies around 24kHz.

[0070] Figure 1b schematically shows a frame replacement module 15 configured according to several embodiments. The frame replacement module 15 includes a tonality extractor 17 and an adaptive sinusoidal expansion and linear prediction module 16.

[0071] When at least one preceding frame 20A, sometimes referred to as the first frame 20A, is provided, the frame replacement module 15 is configured to generate TF tile values for the replacement frame 20B'. The frame replacement module 15 includes an adaptive sine wave extension and linear prediction module 16 that uses either a sine wave extension process or a linear prediction process to generate the TF tile values of the replacement frame using 20B'. The frame replacement module 15 further includes a tonality extractor 17 configured to extract the degree of tonality for each frequency band of the preceding first frame 20A. The degree of tonality is provided to the adaptive sine wave extension and linear prediction module 16, and the adaptive sine wave extension and linear prediction module 16 uses either a sine wave extension process or a linear prediction process to generate the TF tile values of the replacement frame using 20B', and the type of process used is based on the degree of tonality of each frequency band.

[0072] In FIG. 1b, the preceding first frame 20A of QMF samples is input to the frame replacement module 15. The first frame 20A is provided to a tonality extractor 17 that extracts the degree of tonality T i for each frequency band B i . For example, the tonality extractor 17 extracts individual tonality degree values T1, T2, T3, T4 for each of the frequency bands B1, B2, B3, B4. The degree of tonality T i for each frequency band B i in the first frame 20A is provided to the adaptive sine wave extension and linear prediction module 16 together with the first frame 20A. The adaptive sine wave extension and linear prediction module 16 determines whether to use a sine wave extension process or a linear prediction process for each frequency band B i based on the band-specific degree of tonality T i . Using either a sine wave extension process or a linear prediction process, the adaptive sine wave extension and linear prediction module 16 generates the TF tile values of the replacement frame 20B', and using the selected type of processing, for band B iOutputs replacement frame 20B' with the generated TF tile values.

[0073] In summary, the frame replacement module 15 is configured, according to several implementations, to analyze at least one preceding first frame 20A and generate a set of TF tile values ​​that can be used to form a replacement frame 20B' that replaces a subsequent second frame. The frame replacement module 15 may include an adaptive sinusoidal expansion and linear prediction module 16 that generates the TF tile value set using either a sinusoidal expansion process or a linear prediction process. The type of process used may be selected individually for each frequency band based on the degree of tonality of each frequency band in the first frame, as determined by the tonality extractor 17 of the frame replacement module 15. Thus, the TF tile values ​​for one or more frequency bands in the replacement frame 20B' may be generated using sinusoidal expansion, and the TF tile values ​​for one or more other frequency bands may be generated using linear prediction based on the degree of tonality of each individual frequency band.

[0074] The frame replacement module 15 may be further configured to store one or more of these frames so that it can access the TF tile value of the preceding first frame 20A when a replacement frame 20B for the current second frame should be generated.

[0075] Figure 3 schematically shows the frame sequence of three frames 20A, 20B, and 20C. The frame sequence includes the first frame 20A, the second frame 20B which follows the first frame 20A in time, and the third frame 20C which follows the second frame 20B in time. Each frame contains multiple QMF samples (columns), each QMF sample contains multiple TF tiles, and each TF tile is in its respective frequency band B i It is associated with (a row).

[0076] Each of the frames 20A, 20B, and 20C in the frame sequence may be encoded by an encoder in the transmitting unit into a corresponding number of coded data packets (each data packet carrying one frame), and the data packets are transmitted to the receiving unit as a bitstream. However, one or more data packets may be lost or corrupted en route to the receiving unit, thereby rendering the frames contained in these lost or corrupted data packets unavailable to the receiving unit. In the frame sequence of Figure 3, the second frame 20B may be an unavailable (e.g., lost or corrupted) frame in the receiving unit, and the purpose of the frame replacement module in the receiving unit is, in that case, to generate a replacement frame 20B' based on information from the preceding frame, which can be used to replace the unavailable second frame 20B.

[0077] A method for forming a replacement frame will be described with reference to the flowchart in Figure 4 and the frame sequence in Figure 3. This method may also be performed by the frame replacement module 15 described above.

[0078] Step S1, "Acquire the first frame," acquires the first frame 20A. For example, the first frame 20A is associated with the bitstream received by the frame replacement module and / or decoder. The first frame 20A may be either a replacement frame (for example, a frame previously generated by the frame replacement module in response to the loss or corruption of the original first frame) or the original first frame delivered to the frame replacement module and / or decoder.

[0079] In step S2a, “Check the validity of the second frame,” the decoder and / or frame replacement module checks whether the second frame 20B following the first frame 20A is available and undamaged. A frame that is available for decoding and undamaged is identified as a valid frame. If a frame is unavailable, damaged, or otherwise unavailable, the frame is identified as invalid.

[0080] Frames (and / or data packets carrying frames) may be intended to arrive at the receiving unit as a continuous sequence. However, if a frame (or data packet) arrives lost, delayed, or corrupted (i.e., invalid) when it should be processed by the receiving unit (for example, to maintain uninterrupted audio playback), this should be identified as allowing a replacement frame to be generated. Therefore, in step S2a, the receiving unit checks whether a valid second frame is available, for example, to give sufficient time to generate a replacement frame.

[0081] In step S2b, "Is the second frame valid?", the result from step S2a is used to determine whether the second frame 20B is available or unavailable due to loss, corruption, or other reasons. This determination is called the determination of whether the second frame 20B is a valid or invalid frame. Therefore, if the second frame 20B is unavailable in step S2a, or if it is determined that the second frame 20B is corrupted in step S2a, the method determines in step S2b that the second frame is invalid. On the other hand, if the second frame 20B is available in step S2a, or if it is determined that the second frame 20B is not corrupted in step S2a, the method determines in step S2b that the second frame is valid.

[0082] The decoder (and / or frame substitution module) may operate frame-synchronously, meaning the decoder decodes a frame every T seconds, where T is a predetermined time interval. For example, T is equal to the temporal duration of one frame, such as 20 ms. In some embodiments, frames 20A, 20B are transported to the receiving unit as data packets using one or more data transmission protocols. Exemplary data transmission protocols include IP, UDP, and RTP. The decoder in the receiving unit can collect the data packets and store them in a buffer. Data packets may arrive asynchronously, and some data packets may not arrive at all. For example, one or more data packets may not arrive due to a buffer overflow caused by congestion of data traffic along the transmission path. In addition, some packages may arrive with errors detected by the CRC mechanism. A packet that is too late to be available when the decoder needs it to operate frame-synchronously is an example of an invalid data packet, and the frame associated with this data packet is an example of an invalid frame. A data packet that never arrives at the receiving unit (e.g., a lost packet) is another example of an invalid data packet, and the frame associated with this data packet is an example of an invalid frame. A data packet that arrives with an error detected by the CRC mechanism is yet another example of an invalid data packet, and the frame associated with this data packet is an example of an invalid frame.

[0083] For each instance in which an invalid frame (e.g., each invalid packet) is identified, a Bad Frame Indicator (BFI) flag is set. When a decoder decodes a frame associated with a set BFI flag, the portion of the buffer containing that frame is usually ignored, and the packet loss concealment process is initiated. Therefore, identifying whether a frame is valid or invalid can include identifying whether the BFI flag is set for that frame, thereby identifying the frame as invalid if the BFI flag is set, and as valid if the BFI flag is not set.

[0084] Therefore, the decoder and / or frame replacement module can identify that a frame is invalid even if the frame is not received. In frame synchronization operation, it can be assumed that a frame should have reached the decoder and / or frame replacement module of the receiving unit within a predetermined time interval. The start of the predetermined time interval can be measured from the last successful reception of a frame or data packet. Therefore, a frame may be labeled (identified) as invalid if, after (successfully) receiving a preceding frame or data packet, it is determined that the frame or data packet is unavailable after the expiration of the predetermined time interval.

[0085] If it is determined in step S2b that the second frame 20B is invalid, a set of TF tile values ​​for the replacement frame 20B' is generated.

[0086] In some implementations, the type of processing used to generate TF tile values ​​is selected individually for each frequency band based on the degree of tonality of the corresponding frequency band in at least one preceding frame. For this purpose, the generated replacement frame may include a mixture of frequency bands containing TF tile values ​​generated using sinusoidal extension and frequency bands containing TF tile values ​​generated using linear prediction. Of course, the same general processing may also be applied to frames having a single frequency band.

[0087] To generate the TF tile value set for the replacement frame, the method can proceed to step S3 "Get the tonality of the first frame," which involves analyzing the first frame 20A to determine the degree of tonality of the first frame 20A. In some implementations, the degree of tonality is determined individually for each frequency band, resulting in one degree of tonality for each of multiple frequency bands.

[0088] Determining the degree of tonality of a frame (e.g., the first frame 20A) may involve analyzing the TF tile values ​​of the frame (e.g., individually in each frequency band). For example, the first frame 20A may be temporarily stored and accessible, and determining the degree of tonality may involve analyzing the TF tile values ​​of the first frame 20A.

[0089] In some implementations, determining the degree of tonality of a frame (e.g., the first frame 20A) involves evaluating the TF-tile values ​​of at least one first frame 20A to identify the (e.g., complex) phase difference between different TF-tile values ​​of that frame, and calculating the phase standard deviation based on the identified phase difference between different TF-tile values ​​of that frame. The phase standard deviation may be used as an indicator of the degree of tonality, with lower phase standard deviations being used as an indicator of a higher degree of tonality.

[0090] Alternatively, determining the degree of tonality of a frame (e.g., the first frame 20A) involves evaluating the TF-tile values ​​of at least one first frame to identify the amplitude ratios between different TF-tile values, and calculating the amplitude standard deviation based on the identified amplitude ratios between the different TF-tile values. The amplitude standard deviation may be used as an indicator of the degree of tonality, with lower amplitude standard deviations being used as an indicator of a higher degree of tonality.

[0091] As described above, the degree of tonality may be determined individually for each frequency band. The same method for determining tonality may be used for each frequency band, or different methods for determining tonality may be used for different frequency bands. Non-limiting examples of the different methods considered are further described below.

[0092] Referring further to Figure 1b, several embodiments of the replacement module 15 are shown. The frame replacement module 15 includes a tonality extractor 17 that analyzes the first frame 20A and determines the degree of tonality for each frequency band individually. The degree of tonality for each frequency band is T Bi As shown, in the example shown in Figure 1b, the degree of tonality T for the four frequency bands B1 , T B2 , T B3 , T B4 However, this is provided to the adaptive sinusoidal extension and linear prediction module 16 by the tonality extractor 17.

[0093] The method then proceeds to step S4, "Does the tonality exceed the threshold?", which is the degree of tonality T for each frequency band. B1 , T B2 , T B3 , T B4This includes determining whether the value exceeds a predetermined threshold. This step may be performed by the adaptive sinusoidal extension and linear prediction module 16. The predetermined threshold may be the same for each frequency band, or different predetermined thresholds may be used.

[0094] In one example, the given threshold is the phase threshold. It is possible to determine whether a particular frequency band is tonal by comparing its phase standard deviation to the phase threshold. If the phase standard deviation is below the phase threshold, the frequency band is identified as tonal. If the phase standard deviation is greater than or equal to the phase threshold, the frequency band is determined to be noise-like. The phase threshold of the standard deviation can be determined by calculation, experiment, measurement, and other such techniques. In one example, the phase threshold of the standard deviation is approximately π / 6.

[0095] As another example, a given threshold is the amplitude threshold. By comparing the amplitude standard deviation with the amplitude threshold, it is possible to determine whether a particular frequency band is tonal. If the amplitude standard deviation is below the amplitude threshold, the particular frequency band is identified as tonal. If the amplitude standard deviation of a particular frequency band is greater than or equal to the amplitude threshold, the particular frequency band is determined to be noisy. The amplitude threshold for the standard deviation can be determined by calculation, experiment, measurement, and other such techniques. In one example, the amplitude threshold for the amplitude standard deviation is approximately 0.1.

[0096] For each frequency band in the first frame 20A whose degree of tonality indicates that the TF tile value is tonal, the method can proceed to step S5, “Generate TF tile values ​​using sinusoidal expansion,” which involves generating a set of TF tile values ​​for the frequency bands of the replacement frame using the sinusoidal expansion process. As seen in the example in Figure 1b, the individual TF tile values ​​for frequency band number B2 in the replacement frame 20B' are reconstructed using sinusoidal expansion (abbreviated as SE).

[0097] The sinusoidal expansion process may further include calculating the expanded TF tile. The expanded TF tile value for a frame (e.g., first frame 20A) is the TF tile value that occurs later in time than QMF sample N-1 for that frame (e.g., first frame 20A), as described below.

[0098] On the other hand, for frequency bands where the degree of tonality indicates that the TF tile values ​​are those of a noise-like signal, the method proceeds to step S6a, “Generate TF tile values ​​using a linear predictor,” which involves generating TF tile values ​​for the replaced frame using a linear prediction process. As seen in the example in Figure 1b, the individual TF tile values ​​for frequency band numbers B1, B3, and B4 in the replaced frame 20B' are reconstructed using linear prediction (abbreviated as LP).

[0099] In this way, the adaptive sinusoidal extension and linear prediction module 16 generates TF tile values ​​using sinusoidal extension for frequency bands with a relatively high degree of tonality, and generates TF tile values ​​using linear prediction for frequency bands with a relatively low degree of tonality.

[0100] The linear prediction process may include calculating an extended TF tile value using the linear prediction process based on at least two TF tile values ​​within the first frame 20A.

[0101] Optionally, the method then proceeds to step S6b “Form a weighted superposition,” which involves forming a weighted superposition between the TF tile values ​​generated in the linear prediction process and the alternative frames for each frequency band. The alternative frames may be copies of the preceding first frame 20A. In some implementations, the weighted superposition is performed on a predetermined number of temporally earliest TF tile values ​​of the replacement frames, as will be described in more detail below.

[0102] It is understood that the frame structure of the replacement frame may be equal to the frame structure of the invalid second frame 20B. The sinusoidal expansion process and the linear prediction process may be used to generate a subset (e.g., a proper or strict subset) of the TF tile values ​​of the replacement frame, or to generate all of the TF tile values ​​of the replacement frame. As described above, it is further understood that the steps of generating TF tile values ​​using sinusoidal expansion or linear prediction in steps S5 and S6a are performed individually for each frequency band. This makes it possible to generate TF tile values ​​for one or more frequency bands using sinusoidal expansion, based on the degree of tonality calculated for each frequency band in step S3, while generating TF tile values ​​for one or more other frequency bands of the same frame using linear prediction.

[0103] It should be understood that generating a set of TF tile values ​​for a replacement frame can be done individually for each of multiple frequency bands when the frame contains TF tile values ​​that span two or more frequency bands. As mentioned above, TF tile values ​​are generated individually for each frequency band using either a sinusoidal expansion process or a linear prediction process.

[0104] After generating a set of TF tile values ​​for the replacement frame using sinusoidal augmentation and / or linear prediction in step S5 and / or steps S6a, S6b, the method may proceed to an optional step S7, “Pre-calculate augmented TF tile values,” which includes determining augmented TF tile values ​​using a selected TF tile value generation process. The augmented tile values ​​are tile values ​​that are temporally later than the tile values ​​of sample N-1 of the replacement frame. As described below, the TF tile value generation process operates in principle continuously and can generate any number of augmented samples after generating a set (or optionally all) of TF tile values ​​for the replacement frame. By generating augmented TF tile values ​​in step S7, the augmented TF tile values ​​may be stored or, in some cases, made available for the third frame 20C following the second frame 20B. The augmented TF tile values ​​are used to form a perceptually high-quality transition to the third frame 20C when the third frame 20C is a valid frame. If the third frame 20C is another invalid frame, the augmented TF tile values ​​may be retained or regenerated when a replacement frame for the third frame 20C is generated.

[0105] Next, the method can proceed to step S8a, "Output replacement frame," which involves outputting a replacement frame as a replacement for the second frame. The replacement frame output in step S8a is still in the QMF (e.g., CQMF) domain. To convert the QMF (e.g., CQMF) frame to a time-domain representation, the method can proceed to step S8b, "Perform synthesis filtering and output time-domain frame," which may involve processing the replacement frame with a synthesis filter to form an output time-domain audio frame.

[0106] Returning to step S2, if it is determined that the second frame 20B is a valid frame, the method proceeds to step S9, “Was the first frame valid?” which includes determining whether the first frame 20A was a valid or invalid frame. Determining whether the first frame 20A was valid may include identifying whether the BFI flag associated with the first frame 20A (or the data packet carrying the first frame 20A) was set. For example, for each frame 20A, 20B, 20C processed by the replacement module and / or decoder, the BFI flag value indicates whether the frame was valid or invalid. The BFI flag may be temporarily stored to allow identification of whether the preceding frame was valid or invalid.

[0107] If, in step S9, it is determined that the preceding frame was valid, the method may proceed to step S10a, "Output the second frame without modification," which may include outputting the second frame without modification. That is, if both the first frame 20A and the second frame 20B are identified as valid frames, there is no need to perform frame substitution, and all information is available to the decoder, thereby outputting the second frame 20B without modification. For example, if the frame substitution module and / or decoder obtain a sequence of only valid frames, frame substitution is not required, and the frame substitution module outputs the unmodified frames. Optionally, to convert the QMF frames to a time-domain representation, the method may proceed to step S10b, "Perform a synthesis filter and output a time-domain frame," which may include processing the substituted frames with a synthesis filter to form the output time-domain audio frame.

[0108] If, in step S9, the preceding frame is determined to be an invalid frame, the method proceeds to step S11, "Generate cross-faded TF tile values," which includes obtaining an extended set of TF tile values ​​for the first frame 20A and generating one or more cross-faded TF tile values ​​for the second frame 20B. Generating cross-faded TF tile values ​​may include crossfading the extended TF tile values ​​with the TF tile values ​​of the second frame in the cross-fade region. Note that in step S11, the first frame 20A is invalid (replaced by a replacement frame) and the second frame 20B is valid. This achieves a perceptually high-quality transition from the replacement frame to the second frame 20B. Similar to step S7, the extended set of TF tile values ​​for the first frame 20A may be generated using a sinusoidal extension process or a linear prediction process, which may already be available since the extended TF tile values ​​were generated when the first frame 20A was replaced by a replacement frame.

[0109] Step S11 may be repeated individually for each frequency band. Depending on the determined tonality of the frequency band of the preceding frame, the extended TF tile values ​​for one frequency band may be generated using sinusoidal extension, while the extended TF tile values ​​for another frequency band may be generated using linear prediction.

[0110] The crossfade region may include a predetermined number of K samples from among the earliest samples in the second frame 20B.

[0111] In some implementations, the crossfade TF tile value is formed as a crossfade weighted sum of the TF tile value from the TF tile value extension set from the first frame 20A and the TF tile value from the second frame 20B. The two additives of the crossfade weighted sum are weighted by their respective crossfade weight coefficients. The crossfade weight coefficients for the TF tile value from the second frame 20B increase over time between subsequent QMF samples. The crossfade weight coefficients for the TF tile value extension set decrease over time between subsequent QMF samples in a complementary manner.

[0112] In other words, the crossfade weighting coefficient may be inversely proportional to the sample index n. This results in an increase in the number of TF tile values ​​in the second frame 20B as the sample index n increases, and a decrease in the number of extended TF tile values ​​included in each crossfade TF tile value. In some examples, the crossfade weighting coefficient may change linearly or non-linearly (e.g., exponentially) along with the sample index to control the transition from the generated TF tile values ​​to the TF tile values ​​in the second frame 20B.

[0113] The method can then proceed to step S12a, "output a replacement frame having crossfaded TF tile values," which includes outputting a replacement frame to replace the second frame 20B, the replacement frame containing crossfaded TF tile values. The replacement frame may contain crossfaded TF tile values ​​in the crossfaded region, while in the trailing region which is temporally later than the crossfaded region, the replacement frame contains the unmodified TF tile values ​​of the second frame 20B.

[0114] Optionally, to convert the QMF frame to a time-domain representation, the method can proceed to step S12b "Perform a synthesis filter and output a time-domain frame," which involves processing the replaced frame with a synthesis filter to form the output time-domain audio frame.

[0115] The sinusoidal expansion and linear prediction processes for generating TF tile value sets are applicable to any type of frame carrying at least two temporally subsequent QMF samples; that is, each frame contains at least two temporally separated TF tile values. In some implementations, each frame contains complex TF tile values. For example, each frame is generated using a CQMF bank (such as CLDFB).

[0116] Here, we assume that frames 20A, 20B, and 20C in Figure 3 are complex frames generated in the CQMF bank. Frames 20A, 20B, and 20C carry TF tiles 21, each containing a complex TF tile value representing the frequency bandwidth and time duration of the time-domain audio signal.

[0117] Under normal operation, if none of frames 20A, 20B, and 20C are identified as invalid (e.g., lost or corrupted), the streams of frames 20A, 20B, and 20C are available for decoding in the intended order. When a given frame is identified as invalid (e.g., lost or corrupted), there is generally at least one frame that precedes the given frame received without error. However, the sinusoidal expansion process and the linear prediction process can operate recursively, as described below, to generate TF tile values ​​for replacing any number of invalid frames.

[0118] To explain the sinusoidal expansion process and the linear prediction process, without loss of generality, we assume that the second frame 20B is an invalid frame (from which a replacement frame should be generated), and that the preceding first frame 20A is available, or at least a replacement frame for the first frame 20A is available.

[0119] The third frame 20C, following the second frame 20B, can be identified as either a valid or invalid frame, as described below.

[0120] The two TF tile value generation processes (i.e., the sinusoidal expansion process and the linear prediction process) can be used separately. That is, implementations that utilize only sinusoidal expansion or linear prediction are conceivable. Alternatively, the two processes may be used together sequentially and / or in parallel for different frequency bands. In some implementations, adaptive frame replacement techniques are used, where one of the sinusoidal expansion and linear prediction processes is used for each invalid frame or for each frequency band of each invalid frame, based on the degree of tonality of the preceding frame (or its replacement frame) or each frequency band of the preceding frame.

[0121] In the frame sequence shown in Figure 3, the second frame 20B is an invalid frame, and a set of TF tile values ​​(for example, all the TF tile values ​​of the replacement frame 20B' which has the same frame structure as the second frame 20B) is generated to form the replacement frame 20B'.

[0122] When operating in the CQMF domain, both the sinusoidal expansion process and the linear prediction process are performed by determining the complex parameter z based on at least two TF tile values ​​in the first frame 20A. The TF tile 21 in the first frame 20A, whose TF tile value is used to determine the complex parameter z, is in the same frequency band B as the set of TF tile values ​​to be generated. x It could be of that nature. For example, frequency band B x When a complex TF tile value in the second frame 20B belonging to TF tile 21 should be generated, the complex parameter z is the frequency band B in the first frame 20A. x Determined from the TF tile value.

[0123] The complex parameter z can describe the ratio between two TF tile values ​​in the first frame 20A. The complex parameter z can also describe the difference between two parameters derived from two TF tile values. For example, the complex parameter z describes the difference between the phases of two TF tile values. The complex parameter z allows for the generation of TF tile values ​​in the second frame 20B by extrapolating from the TF tile values ​​of the first frame 11 using the complex parameter z. The TF tile value on which z extrapolation is performed can be one of the TF tile values ​​used to determine the complex parameter z, or a different TF tile value. In some implementations, the TF tile value used for extrapolation is the temporally most recent TF tile value in the first frame 20A.

[0124] The complex parameter z is the phase parameter (e.g., e) as explained below. jΔφ (or Δφ), amplitude parameter κ, and one or more complex predictor coefficients γ i It could be one or more of the following.

[0125] The sinusoidal extension process and the linear prediction process may also be performed in a real-valued QMF domain (with some limitations), where the complex parameter z is replaced, for example, by a real-valued parameter describing the ratio between two TF tile values ​​in the first frame 20A.

[0126] The sinusoidal expansion and linear prediction processes are described in detail here. This explanation focuses on a specific frequency band B. x While each of the technologies in question is assumed, it is understood that the process can operate in a similar manner across multiple frequency bands.

[0127] Figure 5 schematically illustrates a sinusoidal expansion process for generating TF tile values ​​for a replaced frame 20B' based on at least one first frame 20A. The sinusoidal expansion process is performed by the frame replacement module to generate TF tile values ​​for one or more frequency bands of the replaced frame.

[0128] The first frame 20A is either a valid frame or a replacement frame for an invalid preceding first frame. The subsequent second frame is an invalid frame, and the TF tile value of the replacement frame 20B' for the second frame should be generated.

[0129] The first frame 20A is in a specific frequency band B x Multiple frequency bands B including i The CQMF sample n is characterized, and n ranges from 0 to N-1. The second frame and the replacement frame 20B' have the same general structure as the first frame 20A. Since the second frame is invalid, the TF tile value set of the replacement frame 20B' is generated.

[0130] The sinusoidal expansion process is performed in a specific frequency band B x Regarding this, we assume that the first frame 20A contains a sinusoidal signal component.

[0131] In the real-valued QMF domain, the sinusoidal signal component x(n) is modeled as follows:

number

[0132] Therefore, the sinusoidal extension of the QMF domain representation of the frame to the substituted frame 20B' can be determined as follows:

number

number

number

[0133] Turning to the CQMF bank domain, each TF tile 21 contains a complex TF tile value and does not contain real numbers, as in the case of real-valued QMF bank domains or time-domain audio signals. To apply the sinusoidal extension technique in the CQMF domain, the complex sinusoidal signal component in the first frame 20A can be described as follows:

number

number

[0134] The complex sinusoidal signal component described in equation (6) has sample-by-sample phase progression, which can be described as a whole by a single complex progression parameter b. The complex progression parameter b can be expressed by the following equation:

number

[0135] The TF tile value y(n) of the replacement frame 20B' for the second frame is the termination parameter a-1 It can be generated by multiplying by b raised to the power of (n+1). That is, y(n) can be obtained as follows:

number

number

[0136] That is, using equation (8) or (9), the TF tile value set for the replacement frame 20B' for the second frame is a -1 and / or b (including Δφ) may be generated. -1 The most accurate sinusoidal extension is achieved when both and b are known, but it is assumed that either one is sufficient to achieve TF tile value generation.

[0137] For example, a -1 If only a is known, the default value of b may be used. An example of a default b value can be obtained by setting f0 to the center frequency of a particular frequency band. For example, for a critically sampled frequency band with a bandwidth of 400 Hz, a suitable choice of f0 is 200 Hz. This may not correspond to the sinusoidal component of the frequency band, but this may be a sufficient approximation. On the other hand, if only b is known, a -1The default value may be used. This may not correspond to the TF tile value of the last CQMF sample in the first frame, but the frequency preservation of sinusoidal components as described by b may be sufficient to form an acceptable replacement frame.

[0138] In some implementations, a vector with N elements is determined, and each element corresponds to an index n in the permutation frame 20B' ranging from 0 to N-1. Each of the N elements in the vector is b n It is proportional to (for example, equal to), and thereby the set of TF tile values ​​of the replacement frame 20B' is the TF tile value of the first frame 20A (for example, the end parameter a -1 ) can be generated by multiplying by a vector.

[0139] The phase parameter Δφ can be determined in various ways. Under the assumption that the sample-by-sample phase progression is approximately constant in the first frame 20A, any two subsequent TF tile values ​​in the first frame 20A can be selected, and the phase parameter Δφ can be determined as the phase difference between the two TF tile values. In practice, the phase of an individual TF tile value x(n) can be determined by taking the arcus tangent of the ratio between the imaginary part and the real part of the TF tile value x(n). An implementation of the arcus tangent function (e.g., atan2) returns the phase modulo 2π, so the phase is unwrapped, and the unwrapped phase θ(n) can be obtained.

[0140] The phase parameter Δφ may then be determined as the difference between the (unwrapped) phases of two subsequent TF tile values, for example, Δφ = θ(n+1) - θ(n) for any n = 0, 1, .N-2.

[0141] It should also be noted that the sample-by-sample phase difference can also be determined from the selection of non-successor TF tile values. For example, if the phase difference between the first TF tile value x(n) and the second TF tile value x(n+k) is determined, the sample-by-sample phase difference can be calculated by dividing the difference by k-1 for any k=1,2,...,N-1.

[0142] To mitigate the effects of statistical deviations, the phase parameter Δφ is set to the phase context window W. p It may be determined as the average phase difference between the TF tile values ​​of two (for example, consecutive) CQMF samples within the sample.

[0143] Phase context window W p This may span multiple consecutive CQMF samples in the first frame 20A. For example, the phase context window W p This extends to all CQMF samples of the first frame 20A, or a subset of the CQMF samples of the first frame 20A (e.g., a proper or strict subset). As another example, the topological context window extends to CQMF samples of more than one frame, such as at least two first frames that are consecutive to each other but precede the second frame that is replaced by the replacement frame 20B'. In general, the TF tile value generation process as defined in this specification is not limited to the analysis of only the immediately preceding first frame, but can analyze multiple preceding first frames (which are replacement frames for valid or invalid frames) to reduce statistical variability.

[0144] CQMF samples closer to the temporal endpoint of the first frame 20A (i.e., as n approaches N-1) are temporally closer to the second frame and may be more important for generating the TF tile value in the replacement frame 20B' compared to CQMF samples that are temporally earlier in the first frame 20A, thus affecting the phase context window W. pThis may include, at a minimum, the last CQMF sample at n=N-1, the second-to-last CQMF sample N-2, optionally the third-to-last CQMF sample N-3, and optionally further consecutive CQMF samples at the end of the first frame 20A. Average phase parameter Δφ mean This sums the (unwrapped) phase differences between the TF tile values ​​of consecutive CQMF samples and the phase context window W. p This can be calculated by dividing by the number of CQMF samples within the phase context window W. p However, if it is the entire valid first frame 20A with N CQMF samples, the average phase parameter Δφ mean This can be determined by the following formula:

number

[0145] Furthermore, if θ(n) is an unwrapped phase, the mean phase parameter Δφ mean This can be determined (for a phase context window covering the entire effective frame 20A) as follows:

number

[0146] Average phase parameter Δφ mean Instead of determining the average phase parameter Δφ mean Instead, it is assumed that the median or mode phase progression parameter can be determined and used.

[0147] In some implementations, each TF tile value is available in polar coordinates (i.e., each TF tile value is represented by amplitude and unwrapped phase values), or each TF tile value is converted to polar coordinates. That is, each TF tile value can be expressed in the form of the following equation:

number

[0148] Termination parameter a -1 Turning our attention to this, according to some implementations, this parameter may be determined directly as the TF tile value of the last CQMF sample (at n=N-1) in the first frame 20A. That is, a -1 = x(N-1). Alternatively, multiple TF tile values ​​within the first frame 20A are analyzed to determine the expected value of x(N-1) which mitigates the statistical deviation, and the expected value of x(N-1) is a -1 It is used as follows. For example, a linear predictor (described later) can be used to predict the value of x(N-1) based on one or more preceding TF tile values ​​in the first frame 20A. Optionally, the expected value of x(N-1) is identified as the predicted value of x(N-1) using the linear predictor and the observed value of x(N-1) in the first frame 20A.

[0149] This allows the expected value of x(N-1) to be determined based on the observed values ​​of x(N-1). For example, the expected value of x(N-1) may be determined by a smoothing operation that removes statistical variation (e.g., noise). The smoothing operation may be, for example, a regression based on the CQMF sample x(N-1) and one or more preceding CQMF samples.

[0150] A further useful improvement to the sine wave expansion process described above is to consider the case of decreasing or increasing amplitude. Rather than simply expanding a sine wave with a fixed amplitude, analyzing the case of decreasing or increasing amplitude may improve the perceived quality of the replacement frame. The amplitude of b from equation (7) above is 1, which is a -1 It is understood that this means the amplitude is maintained for all CQMF samples of y(n).

[0151] By redefining the complex progression parameter b as follows:

number

[0152] In some implementations, a similar method may be used with respect to the phase parameter Δφ in equations (10) and (11) above to determine the amplitude parameter κ, namely, the amplitude context window W m Determines the average amplitude ratio between subsequent TF tile values ​​within the amplitude context window W. m This is the phase context window W p They can be the same size or different sizes.

[0153] Since κ is an exponential parameter, the amplitude context window W m For all CQMF samples included, it is appropriate to determine the average value of the amplitude ratio per sample.

[0154] κ can also be calculated as the geometric mean (rather than the arithmetic mean) across the amplitude context window, which yields the following equation:

number

[0155] In some implementations, an amplitude parameter κ greater than 1 is undesirable because it leads to an exponential increase in amplitude, which can be problematic for the resulting audio quality and / or cause an overload effect. Therefore, the amplitude parameter κ may be implemented so that it is less than or equal to 1. In practice, this limitation allows the amplitude of the sinusoidal extension from the second frame 20B to the replacement frame 20B to be maintained or decreased, but not increased.

[0156] The sinusoidal extension process extends the sinusoidal signal component of the first frame 20A to generate the TF tile value of the replaced frame 20B' for the second frame 20B. The frequency spectrum of the pure sinusoidal signal component is in the form of a single spectral peak located at the frequency f0 of the sinusoidal signal component. That is, the extended sinusoidal signal component is a single tone of a single frequency, which is a very narrowband signal. Theoretically, a single frequency peak can be described as having zero bandwidth. In some implementations, a narrowband (single frequency) generated signal is desirable, especially if the signal represented by the first frame 20A is also narrowband and has a high degree of tonality. The generated TF tile value of the replaced frame 20B' for the second frame 20B is a reasonable and accurate extension of the tone signal in the first frame 20A.

[0157] On the other hand, in some implementations, the signal in the first frame 20A has a low degree of tonality and is characterized by a non-zero bandwidth. For example, many types of audio signals are characterized by a spectral maximum at a specific frequency, while being surrounded by additional frequency components that form a non-zero bandwidth. To more accurately generate the TF tile value of the replacement frame 20B' of the second frame 20B for this type of non-tone signal, the sinusoidal expansion process may be modified to implement one or both of the phase perturbation parameter δφ(n) and the amplitude perturbation parameter δκ(n).

[0158] The purpose of the phase perturbation parameter δφ(n) and amplitude perturbation parameter δκ(n) is to modify the phase parameter Δφ and amplitude parameter κ determined by the sinusoidal expansion process to generate a non-zero bandwidth expansion y(n) that more closely resembles the non-zero bandwidth of the signal represented by the TF tile value x(n) of the first frame 20A. The phase perturbation parameter δφ(n) and amplitude perturbation parameter δκ(n) can be described as additive perturbation noise. The additive perturbation noise can be determined individually for each generated TF tile value, or it can be derived from a random sequence or a pseudo-random sequence.

[0159] For each generated TF tile value, a phase perturbation parameter δφ(n) is determined and can be used to perturb the phase parameter Δφ to form a perturbed phase parameter Δφp(n). The perturbed phase parameter Δφp(n) is used in place of the phase parameter to generate the TF tile values ​​of the replacement frame 20B'. For example, the phase perturbation parameter δφ(n) may define an additive perturbation, and the perturbed phase parameter Δφp(n) is then calculated by the following equation:

number

[0160] The phase perturbation parameter δφ(n) is a predetermined standard deviation σ for each CQMF sample n. δφThe random parameter values ​​for the replacement frame 20B' may be determined by deriving them from a distribution having a predetermined standard deviation σ. δφ This may be based on the standard deviation of the sample-by-sample phase difference Δθ(n) in a context window (which may be equal to or different from the phase context window) that extends over the entire effective frame 20A for the entire first frame 20A, or over at least a portion of at least one first frame 20A. For example, a predetermined standard deviation σ δφ This may be equal to or proportional to the standard deviation of the sample-by-sample phase difference Δθ(n) for all CQMF samples n in the first frame 20A (or its context window).

[0161] In some implementations, running a random number generator for each TF tile value to be generated is undesirable because it can add computational complexity and increase latency. For this purpose, in some implementations, the phase perturbation parameter δφ(n) for one TF tile value to be generated is the sample-by-sample phase difference Δθ(n) and the average sample-by-sample phase difference Δφ from the first frame 20A (or its context window). mean It is obtained as the difference between. To obtain another phase perturbation parameter δφ(n) for another TF tile value that is generated, the sample-by-sample phase difference Δθ(n) of another pair of TF tile values ​​and the average sample-by-sample phase difference Δφ in the first frame 20A are used. mean The difference between the two is calculated. This process is then repeated for multiple pairs of TF tile values ​​in the first frame 20A to generate a new phase perturbation parameter δφ(n) for generating TF tile values ​​in the replacement frame 20B' of the second frame. This essentially characterizes the same standard deviation without the operation of a random number generator.

[0162] In some implementations, the sample-by-sample phase difference Δθ(n) from the effective frame 20A (or its context window) and the average sample-by-sample phase difference Δφ are used. mean The difference between the two is scaled before being used to generate the TF tile value for the replacement frame 20B' for the second frame.

[0163] Sample-by-sample phase difference Δθ(n) and average phase difference Δφ mean Obtaining the phase perturbation parameter δφ(n) as the difference between and allows for the determination of an individual phase perturbation parameter δφ(n) for each TF tile value to be generated, although this is not always necessary. For example, to reduce computational complexity, a set of phase perturbation parameters δφ(n) may be determined for a subset of TF tile value pairs in the first frame 20A, and this set of phase perturbation parameters δφ(n) may then be repeated (e.g., circularly) for the TF tile values ​​generated for the replacement frame 20B' of the second frame.

[0164] Similarly, the amplitude perturbation parameter δκ(n) is determined for each TF tile value to be generated, and the amplitude parameter κ is perturbed to perturb the perturbation phase parameter κ p (n) can be used to form the perturbation phase parameter κ. p (n) is used to generate the TF tile value in place of the original amplitude parameter κ. For example, the amplitude perturbation parameter δκ(n) may define an additive perturbation, and the perturbation phase parameter κ p (n) is then calculated by the following formula:

number

[0165] Alternatively, to avoid using a random number generator, the amplitude perturbation parameter δκ(n) is calculated using the relative amplitude change of two adjacent CQMF samples within the effective frame 20A and the average amplitude parameter κ. mean The additional amplitude perturbation parameter δκ(n) may be determined by calculating the difference between the two. Similarly, the additional amplitude perturbation parameter δκ(n) may be determined by the relative amplitude change θρ(n) of different pairs and the mean amplitude parameter κ mean This can be determined by calculating the following: In this case, the perturbation to be applied may be multiplied by the following formula:

number

[0166] Figures 6a–6d schematically illustrate the linear predictor process for generating a set of TF tile values ​​for the replacement frame 20B' of the second frame. The linear predictor process recursively generates TF tile values ​​for one or more frequency bands of the replacement frame using complex predictor coefficients γ. i This includes defining a linear predictor using [a specific method / tool]. The linear predictor process is performed by the frame replacement module to generate TF tile values ​​for one or more frequency bands of the replacement frame.

[0167] In Figures 6a to 6d, the preceding first frame 20A (or, therefore, the generated replacement frame) is used to generate the TF tile value of the replacement frame 20B' for the subsequent invalid frame.

[0168] Linear prediction processing is an alternative to the sinusoidal expansion process. Depending on the degree of tonality of the frequency band of the first frame 20A, the TF tiles of the replaced frame 20B' can be generated by the frame replacement module using either the sinusoidal expansion process or the linear prediction process.

[0169] In the linear prediction process, a specific frequency band B of the first frame 20A is used. x The TF tile values ​​are analyzed. For example, the TF tile values ​​of at least two CQMF samples in the first frame are analyzed. The TF tile values ​​are analyzed to determine a linear predictor that generates predicted TF tile values ​​x^(n) based on one or more preceding TF tile values ​​x(ni), i>0.

[0170] A first-order linear predictor generates a predicted TF tile value based on one preceding TF tile value, e.g., x(n-1); a second-order linear predictor generates a predicted TF tile value x^(n) based on two preceding TF tile values, e.g., x(n-1) and x(n-2); a third-order linear predictor generates a predicted TF tile value x^(n) based on three preceding TF tile values, and so on for fourth-order and higher-order linear predictors.

[0171] In some implementations, the linear predictor is first-order, second-order, or third-order to keep computational complexity low. Because some implementations of the linear predictor operate within a single, specific frequency band with limited bandwidth (e.g., approximately 400 Hz for CLDFB, or at least less than 1 kHz), the order of the linear predictor can be low, for example, only first-order or second-order.

[0172] The linear predictor uses a specific frequency band B within the first frame 20A (or its context window). x The prediction error can be defined to minimize the prediction error for all TF tile values ​​within (or the predictor context window W of frame 20A). For example, the prediction error is defined as the prediction error for all TF tile values ​​within (or the predictor context window W of frame 20A) within the first frame 20A. L Predictor context window W within or spanning multiple frames L This is the expected mean squared error between the predicted TF tile value x^(n) in the first frame 20A and the actual TF tile value x(n) for all TF tile values ​​(within). For example, the prediction error is E[(x^-x) 2 It may also be expressed as ].

[0173] In some implementations, the linear predictor is defined as follows: [Number] Here, the index i ranges from 1 to M, where M is the order of the predictor, and γi is a complex-valued parameter. This type of linear predictor generates a TF tile value based on at least one preceding and directly adjacent TF tile value. As can be seen from Equation (18), the linear predictor has the same number of predictor parameters γ as the predictor order i including. That is, a first-order linear predictor includes only the parameter γ1, and a second-order linear predictor includes the predictor parameters γ1 and γ2. As an example, a third-order linear predictor generates the TF tile value x^(n) as follows: [Number] Complex prediction parameter γ i To determine the complex prediction parameter γ, an optimization problem can be formed to fit the complex prediction parameter γ that minimizes the prediction error (e.g., E[(x^ - x) 2 ) for all TF tile values in a specific frequency band of the first frame 20A (or the context window covering the first frame and optionally earlier frames). i

[0174] In some implementations, one or more predictor coefficients γ i are calculated according to Levinson-Durbin based on the autocorrelation sequence r xx (n) of the TF tile values of the first frame 20A. In the case of a first-order linear predictor, the single complex predictor coefficient γ1 can simply be extracted as follows: [Number] ​To generate the TF tile values of the second frame 20B, the linear predictor uses, as a starting condition, one or more TF tile values of the first frame 20A to predict the tile values of the second frame 20B. For example, the linear predictor is used to generate the TF tile value y(n) of the invalid frame 20B as follows: [Number] Here, the CQMF sample y(j) for a negative CQMF sample index j is taken from the temporally later end of the preceding first frame 20A, which means the following equation: [Number] Figures 6a and 6b schematically show how the third-order linear predictor from Equation (21) was defined to generate an accurate prediction of the TF tile values in the first frame 20A. In Figure 6a, the third-order predictor uses three complex predictor parameters γ1, γ2, γ3 that operate on the TF tile values of the preceding CQMF samples N-3, N-4, N-5 to generate a prediction of the TF tile value in a particular band B of CQMF sample N-2 x In the same way, the third-order predictor may generate a prediction of the TF tile value in a particular band B of CQMF sample N-1 using three complex predictor parameters γ1, γ2, γ3 that operate on the TF tile values of the preceding CQMF samples N-2, N-3, N-4 as shown in Figure 6b x In Figures 6a and 6b show the tasks configured for the linear predictor to perform (i.e., to generate an accurate prediction of the TF tile values of the first frame 20A), but the same predictor is used to generate the TF tile values in a particular frequency band B in the replacement frame 20B' of the second frame

[0175] However, Figures 6a and 6b show the tasks configured for the linear predictor to perform (i.e., to generate an accurate prediction of the TF tile values of the first frame 20A), but the same predictor is used to generate the TF tile values in a particular frequency band B in the replacement frame 20B' of the second frame x In Figures 6a and 6b show the tasks configured for the linear predictor to perform (i.e., to generate an accurate prediction of the TF tile values of the first frame 20A), but the same predictor is used to generate the TF tile values in a particular frequency band B in the replacement frame 20B' of the second frame

[0176] Figure 6c shows a predictor from Figures 6a and 6b (using the same complex prediction parameters γ1, γ2, γ3) that predicts the TF tile value of the first CQMF sample n=0 in the replacement frame 20B' for the second frame 20B using the last TF tile values ​​of the first frame 20A in the CQMF samples N-1, N-2, and N-3, according to equations (21) and (22) above. Once the TF tile value of the CQMF sample n=0 in the replacement frame 20B' for the second frame 20B is predicted, the linear predictor may then predict the TF tile value of the CQMF sample n=1 in the replacement frame 20B' for the second frame 20B, using the (already predicted) TF tile value of the CQMF sample n=0 in the replacement frame 20B' and the preceding TF tile values ​​of the first frame 20A in the CQMF samples N-1 and N-2, as shown in Figure 6d. Similarly, the linear predictor continues to operate through the TF tile values ​​of the replacement frame 20B' of the second frame 20B, for a specific frequency band B x It is possible to generate all TF tile values ​​in the replacement frame 20B'.

[0177] For a first-order linear predictor of the type that operates on the immediately preceding CQMF sample according to equation (19), it should be noted that the complex predictor parameter γ1 corresponds to the complex coefficient b from equation (10). Thus, determining the linear predictor parameter γ1 (e.g., using equation (20)) is another alternative to determining the complex progression coefficient b, which incorporates both the amplitude parameter κ and the phase parameter Δφ.

[0178] Linear predictors of the type derived from equation (19) have been shown to tend to predict decaying TF tile values. That is, the linear predictor progressively generates TF tile values ​​with decreasing amplitudes based on the complex predictor parameter γ. iThere is a tendency to predict this. The reason for this behavior is that the poles of the inference filter defined by the linear predictor are often inside the unit circle, whereas only poles precisely located on the unit circle can generate TF-tile values ​​of sustained amplitude. A replaced frame 20B' with decaying TF-tile values ​​may result in a decoded time-domain audio signal that is perceived as unstable.

[0179] For this purpose, it is assumed that for the first-order predictor, the calculated complex predictor parameter γ1 can be modified by setting its amplitude to 1, or, if |γ1| is not equal to 1, to a predetermined value at least close to 1, or less than a predetermined value close to 1. The modified complex predictor parameter γ*1 thus obtained can be used by the linear predictor instead of the calculated complex predictor parameter γ1 when generating the TF tile values ​​of the replacement frame of the second frame.

[0180] Furthermore, it should be noted that, in the case of a first-order predictor, the calculated complex predictor parameter γ1 is a multiplicative value that describes how the TF tile value progresses from one CQMF sample to the next. That is, the complex predictor parameter γ1 for a first-order predictor is b = κe jΔφ The complex progression parameter is equal to b such that = γ1. Therefore, if the complex prediction parameter γ1 is modified so that its amplitude is 1, the phase progression portion of the complex progression parameter b is preserved. At the same time, this relationship shows that the complex prediction parameter γ1 of the first-order predictor can be used to determine the phase progression parameter Δφ (as the phase of the complex prediction parameter γ1) and / or the amplitude progression parameter κ (as the amplitude of the complex prediction parameter γ1). Thus, the complex prediction parameter γ1 introduces another method for determining the phase progression parameter Δφ and / or the amplitude progression parameter κ of the sinusoidal expansion process by first determining the complex prediction parameter γ1 and then calculating Δφ and / or κ based on γ1. The converse is also applicable, which means that γ1 can be calculated from Δφ (or Δφ and κ).

[0181] For higher-order predictors (e.g., second- or third-order predictors), it is also possible to compensate for signal reduction by modifying the predictor parameters from their calculated values. In some implementations, this is done by calculating the roots of the predictor polynomial and then modifying the predictor parameter γ* so that the roots move closer to or on the unit circle while maintaining the phase value of the roots. i This is achieved by identifying such parameters. Using such modified predictor parameters, the amplitude of the generated TF tile values ​​generally decreases more slowly or not at all, which can facilitate an improvement in the perceived quality of the replacement frame.

[0182] Figure 7 shows the first frame 20A, the generated replacement frame 20B', and the substitute frame 20A'.

[0183] The TF tile values ​​of the first frame 20A are used to generate the TF tile values ​​of the replacement frame 20B' using a sinusoidal expansion process or a linear prediction process in each frequency band. Furthermore, the TF tile values ​​of the replacement frame 20B' are further modified by weighted summation of the corresponding TF tile values ​​of the replacement frame 20A'.

[0184] In Figure 7, frequency band B in the replacement frame 20B' x The TF tile values ​​are generated using the linear prediction process described above. The generated TF tile values ​​in the replacement frame 20B' are further modified by adding the corresponding TF tile values ​​in the alternative frame 20A'. For example, the alternative frame 20A' is a copy of the first frame 20A.

[0185] In other words, the TF tile value of the replacement frame 20B' of the second frame 20B can be formed as a weighted sum of the TF tile value generated (using sinusoidal expansion or linear prediction) and the alternative frame 20A'. In one example, the alternative frame 20A' is equal to the first frame 20A or its cyclically shifted version.

[0186] The exponential decay caused by the amplitude parameter κ of the sinusoidal extension (equal to the amplitude |γ1| of the first-order predictor parameter of the first-order predictor) is weighted 1-κ such that the generated TF tile value of the substitution frame 20B' of the second frame 20B is formed as a weighted sum of the TF tile value of the substitution frame 20A' and the TF tile value generated from the sinusoidal extension process or the linear prediction process. n+1 (or 1-|γ1|) n+1 This can be compensated for by adding a weighted alternative frame 20A'.

[0187] For example, when linear prediction is used, each CQMF sample y(n) in the replacement frame 20B' of the second frame 20B may be generated as follows:

number

number

[0188] A high-quality replacement frame is provided by reintroducing an alternative frame 20A' with weights that compensate for the exponential decay of the generated TF tile values. This forms a crossfade from the TF tile values ​​generated by the sinusoidal expansion or linear prediction process to the TF tile values ​​of the alternative frame 20A', which has the perceptual advantage of avoiding the signal discontinuity that would occur if the frame iteration technique were applied without generating TF tile values ​​by sinusoidal expansion or linear prediction.

[0189] Following the techniques described above, it is possible to generate a TF tile value for a replacement frame 20B' for an invalid frame. Next, we will describe the process for transitioning back from the generated replacement frame 20B' generated for the second frame 20B to the subsequent valid third frame 20C (received without errors), and for generating a TF tile value when two or more consecutive frames are invalid.

[0190] Figure 8 shows the replacement frame 20B' generated to replace the second frame, the subsequent valid third frame 20C, and the replacement frame 20C' generated to replace the valid third frame 20C.

[0191] The generation of replacement frames may be performed for all invalid frames in a frame sequence. Furthermore, the generation of replacement frames may be performed for valid frames during a given situation. For example, if a replacement frame 20B' is generated to replace an invalid second frame, and the subsequent third frame 20C is a valid frame, then a replacement frame 20C' for the third frame 20C may be generated and used as the replacement frame 20C' for the third frame 20C. The purpose of replacing a valid frame following an invalid frame with a replacement frame may be to provide a smooth transition from the replacement frame (which is an approximation of the original frame) to the valid frame.

[0192] The TF tile value y(n) of the replaced frame 20B' is generated using the sinusoidal expansion process and / or linear prediction process as described above. The third frame 20C is a valid frame following the replaced frame 20B' for the second frame. Various techniques can be used to avoid signal discontinuities that would be disruptive to the listener, in order to provide a high-quality transition from the replaced frame 20B' to the third frame 20C. Each frequency band B of the third frame 20C i The CQMF samples are labeled x2(n) for n=0,1,.N-1.

[0193] In some implementations, a crossfade technique is applied to form the replacement frame 20C' of the third frame for the transition from the replacement frame 20B' of the second frame to the TF tile value of the third frame 20C. The crossfade technique involves generating an extended TF tile value of the replacement frame 20B' of the second frame that exceeds N-1 CQMF samples of the replacement frame 20B' of the second frame, and using these extended CQMF samples that exceed N-1 CQMF samples, the replacement frame 20C' of the third frame 20C is achieved by crossfading the third frame 20C into a crossfade region and crossfading the third frame 20C to the TF tile value. That is, a sinusoidal extension process and / or a linear prediction process is used to generate K consecutive extended TF tile values ​​for the replacement frame 20B' of the second frame having a CQMF sample index n≧N. These K extended TF tile values ​​overlap in time with the first K TF tile values ​​of the valid third frame 20C.

[0194] The number K of extended TF tile values ​​beyond the N-1 CQMF sample generated may be adjusted based on the desired length of the crossfade region. In one example, TF tile values ​​for two CQMF samples beyond the N-1 CQMF sample, i.e., TF tile values ​​for CQMF samples N and N+1, are generated. In the CLDFB domain, the two CLDFB samples correspond to 2.5 ms of audio content.

[0195] The K earliest TF tile values ​​in the third frame 20C are labeled k=0,1,.,K-1, and the crossfade may be performed according to the following equation:

number

[0196] Crossfade interpolation parameter α k This may be adjusted to decrease as k increases. This results in a replacement frame 20C' having a progressive boost of the first K TF tile values ​​in the third frame 20C and a corresponding progressive decay of the generated extended TF tile value y(N+k). In one example, the crossfade interpolation parameter α k This defines linear interpolation, and α k = (Kk) / (K+1) can be set. However, to define a nonlinear crossfade profile, for example, the crossfade interpolation parameter α k It is understood that other definitions are possible.

[0197] For the TF tile values ​​of the replacement frame 20C' of the third frame 20C that are temporally later than the first K TF tile values ​​of the crossfade region (i.e., TF tile values ​​n=K,K+1,.,N), the TF tile values ​​are set to the TF tile values ​​of the third frame 20C. The TF tile values ​​of the third frame 20C that are temporally later than the crossfade region may be called trailing region TF tile values, which are equal to the corresponding TF tile values ​​of the third frame 20C.

[0198] Crossfade interpolation parameter α kcan be adjusted over K extended TF tile values to achieve a desired crossfade profile, and it is further possible to utilize a sine wave extension having a decay linear prediction parameter γ1 or an amplitude parameter κ smaller than 1. According to this technique, the crossfade period is not dominated by a predetermined number K of CQMF samples, but by the decay rate of the linear prediction parameter γ1 or the amplitude progression parameter κ.

[0199] In one embodiment, a linear prediction process or a sine wave extension process is used to generate extended TF tile values beyond the CQMF sample N-1 of the second frame 20B, whereby these extended TF tile values are crossfaded with the TF tile values of the third frame 20C using a crossfade weighted sum, so that the extended TF tile values are gradually de-emphasized, and the crossfade weight factor corresponds to the amplitude of the prediction parameter γ1 or the decay of the amplitude parameter κ. For example, in the case of a linear prediction process, the K TF tiles in the crossfade region of the replacement frame 20C' of the third frame 20C are determined as follows:

Equation

[0200] The replacement frame 20B' of the third frame 20C is generated according to equation (26) for the first K CQMF samples or until |γ1| k+1 or κ k+1 is below a predetermined value, and thereafter, it is assumed that the TF tile values of the third frame 13 are used without modification.

[0201] In some scenarios, there are two or more consecutive invalid frames, and there are two techniques that can be used to generate the TF tile values of replacement frames for two or more invalid frames, namely updated extension and continued extension.

[0202] In the updated extension, the first invalid frame is replaced by a replacement frame according to the above using the preceding valid frame. To generate a replacement frame for the second invalid frame following the first invalid frame, the replacement frame for the first invalid frame is treated like a valid frame, and the process described above is repeated by generating a replacement frame for the second invalid frame based on the (generated) TF tile value of the replacement frame for the first invalid frame.

[0203] Specifically, the phase progression parameter Δφ and / or the amplitude progression parameter κ are calculated based on the replacement frame of the first invalid frame, and / or the complex prediction coefficient γ i is calculated from the replacement frame of the first invalid frame. This means that in the case of sine wave extension, the phase progression parameter Δφ and / or the amplitude progression parameter κ used to perform sine wave extension to generate the TF tile value of the replacement frame for the second invalid frame may be different from the Δφ and / or κ used to perform sine wave extension to generate the TF tile value of the replacement frame for the first invalid frame. The same applies to any phase or amplitude perturbation parameter that can be appropriately updated. Similarly, in the case of linear prediction, one or more complex prediction coefficients γ i used to generate the TF tile value of the replacement frame for the second invalid frame i are different from the one or more complex prediction coefficients γ

[0204] In the continued extension, the linear prediction or sine wave extension process uses the parameters Δφ, κ, and γ iWithout updating, the process continues from the replacement frame of the first invalid frame to the replacement frame of the second invalid frame. The sinusoidal expansion process or linear prediction process can, in principle, continue indefinitely and generate more than N-1 TF tile values ​​per frame than the number of CQMF samples. Thus, in the continued expansion, the linear predictor process or sinusoidal expansion process continues to generate TF tile values ​​beyond the N CQMF samples that constitute the replacement frame for the first invalid frame, so that the next N CQMF samples are used as replacement frames for the second invalid frame. The process may continue to generate TF tile values ​​for CQMF samples that form the third, fourth, and fifth replacement frames for the corresponding invalid frames, with each batch of N TF tile values ​​generated by the sinusoidal expansion process or linear prediction process constituting a new replacement frame. The complex prediction coefficient γ compared to the updated expansion. i Alternatively, the phase progression parameter Δφ and / or the amplitude progression parameter κ are therefore retained throughout the generation of TF tile values ​​for all replacement frames. Similarly, the same amplitude and / or phase perturbation parameters may be retained.

[0205] Figure 9 shows a schematic block diagram of an exemplary electronic device or architecture 200 (e.g., device 200) suitable for implementing exemplary embodiments of the present disclosure. Architecture 200 includes, but is not limited to, server and client devices, systems, modules and methods, as described with reference to Figures 1a, 1b, and 3. As shown, architecture 200 includes a central processing unit (CPU) 201 capable of performing various operations according to a program stored, for example, in read-only memory (ROM) 202 or a program loaded, for example, from a storage unit 208 into random access memory (RAM) 203. The CPU 201 may be an electronic processor 201 that may include one or more processor cores, and in some examples, the processor 201 may be multiple processors. The RAM 203 also stores data necessary when the CPU 201 performs various operations, as needed. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204. The input / output (I / O) interface 205 is also connected to bus 204.

[0206] The following components are connected to the I / O interface 205: an input unit 206 which may include a keyboard, mouse, etc.; an output unit 207 which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 208 which may include a hard disk or another suitable storage device; and a communication unit 209 which may include a network interface card such as a network card (e.g., wired or wireless).

[0207] In some implementations, the input unit 206 includes one or more microphones located at different positions (depending on the host device) that enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, or other appropriate formats).

[0208] In some implementations, the output unit 207 includes a system with varying numbers of speakers. The output unit 207 can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, or other appropriate formats) depending on the capabilities of the host device.

[0209] In some embodiments, the communication unit 209 is configured to communicate with other devices (e.g., via a network). The drive 210 is also connected to the I / O interface 205, if necessary. A removable medium 211, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or another suitable removable medium, is mounted on the drive 210, and as a result, computer programs read from it are installed in the storage unit 208, if necessary. Those skilled in the art will understand that although the device 200 is described as including the components described above, in actual applications it is possible to add, remove, and / or replace some of these components, and that all such changes or variations are all included within the scope of this disclosure.

[0210] According to exemplary embodiments of the present disclosure, the processing described above may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product which includes a computer program tangibly embodied on a machine-readable medium. The computer program includes program code for performing the method. In such embodiments, the computer program may be downloaded and implemented from a network via a communication unit 209 and / or installed from a removable medium 211, as shown in Figure 9.

[0211] Typically, various exemplary embodiments of this disclosure may be implemented in hardware or dedicated circuitry (e.g., control circuits), software, logic, or any combination thereof. For example, the unit described above may be executed by a control circuit (e.g., CPU 201 in combination with the other components in Figure 9). Thus, the control circuit can perform the operations described in this disclosure. Some embodiments may be implemented in hardware, while others may be firmware or software implementations that may be executed by other computing devices, which may include a control unit, a microprocessor, and / or a control circuit. Various embodiments of the exemplary embodiments of this disclosure have been illustrated and described using block diagrams, flowcharts, or some other graphical representations, but it should be understood that the blocks, devices, systems, techniques, or methods described in this specification may, in non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or control units or other computing devices, or any combination thereof.

[0212] Furthermore, the various blocks shown in the flowchart may be considered as steps of a method, and / or as operations resulting from operations in computer program code, and / or as a plurality of coupled logic circuit elements configured to perform related functions. For example, embodiments of the present disclosure include a computer program product which includes a computer program tangibly embodied on a machine-readable medium. The computer program includes program code configured to perform the methods described above.

[0213] In the context of this disclosure, a machine-readable medium may be any tangible medium that contains or can store a program for use by or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be intangible and may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatus, or any suitable combination thereof. More specific examples of machine-readable storage media may include one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0214] Computer program code for performing the methods of this disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to one or more processors of a general-purpose computer, a dedicated computer, or other programmable data processing device having control circuits, and when executed by one or more processors of the computer or other programmable data processing device, the program code will perform functions / operations specified in flowcharts and / or block diagrams. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0215] Embodiments described herein perform the generation of TF tile values ​​for one or more replacement frames in order to replace one or more frames identified as invalid, and / or to replace one or more valid frames when a preceding frame was valid. The described method and process may be implemented in an iterative manner, and the method and process is repeated for each frame in a sequence of frames to generate replacement frames or refrain from generating replacement frames based on the validity of each frame in the sequence of frames.

Claims

1. A method for concealing lost or corrupted frames with audio content within a frame sequence, wherein the method is: A step of obtaining at least one first frame associated with the frame sequence, wherein the at least one first frame is a frame of the frame sequence or a generated replacement frame. The steps include determining whether a second frame following at least one first frame is valid or invalid, When the second frame is identified as invalid, the following occurs: Analyze at least one of the first frames to determine the degree of tonality. The degree of tonality is compared with a threshold to determine whether the degree of tonality exceeds the threshold. If the degree of tonality exceeds the threshold, a sinusoidal expansion process is applied to generate a set of time-frequency (TF) tile values ​​for the replacement frame. If the degree of tonality does not exceed the threshold, a linear prediction process is applied to generate the TF tile value set for the replacement frame. The steps include generating the TF tile value set for a replacement frame to replace the second frame, The steps include outputting the replacement frame in order to replace the second frame, A method that includes this.

2. When the second frame is identified as valid, The steps include identifying whether the at least one first frame is valid or invalid, If at least one of the first frames is identified as invalid, then: Obtain the TF tile value extension set of at least one first frame, In the crossfade region, one or more crossfaded TF tile values ​​are generated by crossfading the TF tile value set of the second frame with the TF tile value extension set of at least one first frame. The second frame is output as the replacement frame for replacing the second frame, along with the one or more crossfated TF tile values. The steps include generating a set of TF tile values ​​for the replacement frame to replace the second frame, The method according to claim 1, further comprising:

3. The step of obtaining the TF tile value extension set of at least one first frame is: The method according to claim 2, comprising the step of generating the TF tile value expansion set based on the TF tile values ​​of at least one first frame, based on either the sinusoidal expansion process or the linear prediction process.

4. The method according to claim 3, wherein each frame includes N subsequent TF tile values ​​associated with N subsequent samples, and the extended set of TF tile values ​​includes K subsequent TF tile values ​​extending to K subsequent samples that are temporally later than the N samples of the first frame.

5. The method according to claim 2, wherein the crossfade region includes a predetermined number of temporally earliest TF tile values ​​from the set of TF tile values ​​of the second frame.

6. The method according to claim 2, wherein the step of generating one or more crossfaded TF tile values ​​includes the step of calculating a crossfaded weighted sum of the TF tile value of the second frame and the TF tile value of the TF tile value of the extended set of TF tile values ​​of the at least one first frame for each TF tile value in the crossfaded region.

7. The method according to claim 6, wherein the crossfade weight coefficient of the crossfade weighted sum represents the portion of the crossfade weighted sum that should be composed of the TF tile values ​​of the TF tile value extension set, and the weight coefficient for subsequent crossfade TF tile values ​​in the crossfade region decreases with respect to crossfade TF tile values ​​later in time.

8. The method according to claim 2, wherein the replacement frame includes the crossfaded TF tile values ​​in the crossfade region, and the replacement frame further includes a trailing region which includes the TF tile values ​​of the second frame located outside the crossfade region.

9. If the degree of tonality of the at least one first frame exceeds the threshold, the steps include calculating an extended TF tile value using sinusoidal expansion based on at least two TF tile values ​​of the at least one first frame, If the degree of tonality of the first frame does not exceed the threshold, the steps include: calculating an extended TF tile value using a linear prediction process based on at least two TF tile values ​​of at least one first frame; The method according to claim 2, further comprising:

10. The steps of generating a set of TF tile values ​​for the replacement frame are: The steps include determining a complex parameter based on at least two TF tile values ​​of different CQMF samples in the at least one first frame, The steps include generating a set of TF tile values ​​for the replacement frame by modifying the TF tile values ​​of at least one first frame using the complex parameter, The method according to claim 1, further comprising:

11. The method according to claim 10, wherein each CQMF sample is a sample of the CLDFB (Complex Low-Delay Filter Bank) representation as defined in the 3GPP IVAS (Immersive Voice and Audio Services) Codec.

12. The step of generating a set of TF tile values ​​for the aforementioned replacement frame is: The method according to claim 10, comprising the step of generating a first substituted TF tile value in a first CQMF sample of the substituted frame by modifying the TF tile value of at least one first frame using the complex parameter.

13. The step of generating a set of TF tile values ​​for the aforementioned replacement frame is: The method according to claim 12, further comprising the step of generating a second substituted TF tile value in a second CQMF sample of the substitution frame, either after or before the first CQMF sample of the substitution frame, by modifying the generated first substituted TF tile value of the first CQMF sample in the substitution frame with the complex parameter.

14. A step of determining a vector, wherein the vector comprises N elements having an index n in the range of 0 to N-1, and each of the N elements is proportional to the (n+1) power of the complex parameter, It further includes, The step of generating the TF tile value set for the replacement frame is: The method according to claim 10, further comprising the step of multiplying the TF tile value of the first frame by the vector.

15. The method according to claim 10, wherein the complex parameter includes one or more of a real part, an imaginary part, or both a real part and an imaginary part.

16. Applying the sinusoidal expansion process to generate a set of TF tile values ​​for the replacement frame is: The method further includes calculating the complex parameter as a complex progression parameter that parameterizes a complex sinusoidal curve based on the at least two TF tile values ​​of different CQMF samples in the at least one first frame, The method according to claim 10, wherein generating the set of TF tile values ​​for the replacement frame includes performing sinusoidal expansion based on the complex progression parameter.

17. Calculating the aforementioned complex progression parameter is The method according to claim 16, comprising calculating the phase parameter of the complex progression parameter as the average phase difference between the TF tile values ​​of the phase context window of the TF tile values ​​of at least one first frame.

18. The method according to claim 17, further comprising the step of generating the TF tile value set for the replacement frame using the phase parameter.

19. A step of calculating at least one phase perturbation parameter based on the phase deviation of at least one TF tile value in the first frame from the phase parameter, Before generating the TF tile value set for the replacement frame using the phase parameters, the steps include adjusting the phase parameters using the phase perturbation parameters, The method according to claim 17, further comprising:

20. Calculating the aforementioned complex progression parameter is The method according to claim 16, comprising calculating the amplitude parameter of the complex progression parameter as the average amplitude ratio between subsequent TF tile values ​​in the amplitude context window of the TF tile values ​​of at least one first frame.

21. The method according to claim 20, further comprising the step of generating the TF tile value set for the replacement frame using the amplitude parameter.

22. The steps include determining an amplitude perturbation parameter based on the deviation of the amplitude of at least one TF tile value in at least one first frame from the amplitude parameter, Before generating the TF tile value set for the replacement frame using the amplitude parameters, the steps include adjusting the amplitude parameters using the amplitude perturbation parameters, The method according to claim 20, further comprising:

23. Applying the linear prediction process to generate a set of TF tile values ​​for the replacement frame is: The further includes calculating the complex parameter as at least one complex predictor coefficient of a linear predictor configured to predict the TF tile value in a subsequent CQMF sample in the at least one first frame based on the TF tile value in at least one preceding CQMF sample in the at least one first frame, The method according to claim 10, wherein generating the TF tile value set for the replacement frame comprises performing a linear prediction of the TF tile value set based on the at least one complex predictor coefficient.

24. Calculating the complex parameter as at least one complex predictor coefficient is, The method according to claim 23, comprising identifying a linear complex coefficient to be multiplied by the TF tile value of a preceding CQMF sample in order to predict the TF tile value of a subsequent CQMF sample in the predictor context window of at least one first frame, wherein the linear complex coefficient is identified by minimizing the prediction error in the predictor context window.

25. Calculating the complex parameter as at least one complex predictor coefficient is, The method according to claim 24, further comprising determining a quadratic complex predictor coefficient that is multiplied by a second preceding TF tile value of a second preceding CQMF sample value in order to predict the TF tile value in the subsequent CQMF sample within the predictor context window, wherein the primary and secondary predictor coefficients are identified by minimizing the prediction error within the predictor context window.

26. Applying the linear prediction process to generate the TF tile value set for the replacement frame is: A set of TF tile value predictions is generated using the aforementioned at least one complex predictor coefficient, A modified TF tile value set is formed as a weighted sum of the TF tile value prediction set and the TF tile value set of at least one first frame, Forming the replacement frame based on the modified TF tile value set, The method according to claim 23, which includes the following:

27. The method according to claim 26, wherein the weight coefficients of the weighted sum are based on the amplitude of the at least one complex predictor coefficient.

28. The method according to claim 27, wherein the first weighting coefficient for the first modified TF tile value is different from the second weighting coefficient for the second modified TF tile value, and the ratio between the first weighting coefficient and the second weighting coefficient is equal to the amplitude of the at least one complex predictor coefficient.

29. The method according to claim 26, wherein the replacement frame includes the modified TF tile value set in a decay region, and the replacement frame further includes a repeating region including the unmodified TF tile value of the first frame, the decay region being earlier in time than the repeating region.

30. Analyzing at least one of the first frames to determine the degree of tonality is, Evaluating the TF tile values ​​of at least one first frame to identify the phase difference between different TF tile values, Based on the identified differences in phase between different TF tile values, the phase standard deviation is calculated, The phase standard deviation is compared with a phase threshold, and if the phase standard deviation is less than the phase threshold, it is determined that the at least one first frame is tonal, and if the phase standard deviation is greater than or equal to the phase threshold, it is determined that the at least one first frame is noisy. The method according to claim 1, including the method described in claim 1.

31. Analyzing at least one of the first frames to determine the degree of tonality is, Evaluating the TF tile values ​​of at least one first frame to identify the amplitude ratio between different TF tile values, Based on the identified amplitude ratios between different TF tile values, the amplitude standard deviation is calculated, The amplitude standard deviation is compared with the amplitude threshold to determine that at least one first frame is tonal if the amplitude standard deviation is less than the amplitude threshold, and that at least one first frame is noisy if the amplitude standard deviation is greater than or equal to the amplitude threshold. The method according to claim 1, including the method described in claim 1.

32. Each frame contains multiple samples, each sample contains multiple TF tile values ​​across multiple frequency bands, and the method is, When the second frame is identified as invalid, for each of the multiple frequency bands, the following applies: Analyzing at least one of the first frames to determine the degree of tonality for each frequency, For each frequency band, the degree of tonality is compared with a threshold to determine whether the degree of tonality exceeds the threshold. In the frequency band where the degree of tonality exceeds the threshold, a sinusoidal expansion process is applied to generate the TF tile value set for the replacement frame. In a frequency band where the degree of tonality does not exceed the threshold, a linear prediction process is applied to generate the TF tile value set for the replacement frame. The method according to claim 1, comprising the step of generating a set of TF tile values ​​for the replacement frame.

33. The method according to claim 1, wherein the second frame is identified as invalid if it is unavailable for decoding or is corrupted during the decoding time.

34. A device comprising a processor and memory, wherein the device is configured to perform the method according to any one of claims 1 to 33.

35. A non-temporary medium storing software, wherein the software includes instructions for controlling one or more devices to perform the method according to any one of claims 1 to 33.

36. A computer program including instructions, wherein, when the computer program is executed by a computer, the instructions cause the computer to execute the method according to any one of claims 1 to 33.

37. A method for concealing lost or corrupted frames in a complex quadrature mirror filter (CQMF) domain, A step of receiving at least one first frame of a CQMF sample, wherein each CQMF sample spans one or more time-frequency (TF) tiles, and each TF tile is associated with a frequency band and a complex TF tile value, The steps include determining whether a second frame of the CQMF sample following at least one first frame is valid or invalid, When the second frame is identified as invalid, For at least one of the frequency bands of each of the one or more TF tiles of the CQMF sample of the at least one first frame, The steps include determining a complex parameter based on at least two of the TF tile values ​​in the at least one first frame, The steps include generating a replacement TF tile value by modifying at least one TF tile value of the first frame using the complex parameter, The steps of forming a replacement frame for the second frame based on the replacement TF tile value, The steps include outputting the aforementioned replacement frame, A method that includes this.

38. The method according to claim 37, wherein each CQMF sample is a sample of the CLDFB (Complex Low-Delay Filter Bank) representation as defined in the 3GPP IVAS (Immersive Voice and Audio Services) Codec.

39. A receiving unit, A decoder configured to receive data packets in a bitstream, wherein each data packet is associated with a current frame containing a plurality of time-frequency (TF) tile values, and the decoder is further configured to decode the received data packets to obtain the TF tile values ​​of the current frame. A frame replacement module is configured to identify whether the current frame is invalid, and if it is identified that the current frame is invalid, to generate a set of TF tile values ​​for the replacement frame, wherein the frame replacement module is configured to: A tonality extractor that analyzes at least one preceding frame and determines the degree of tonality of the at least one preceding frame, wherein the at least one preceding frame is temporally earlier than the current frame, A frame replacement module includes an adaptive sinusoidal extension and linear prediction module configured to generate the TF tile value set for the replacement frame based on the determined tonality degree of the at least one preceding frame, A synthesis filter bank configured to receive the TF tile values ​​of the replacement frame and convert the TF tile values ​​of the replacement frame into time-domain audio segments, A receiving unit including a receiver.

40. Each frame contains multiple frequency bands, and each frame contains multiple TF tile values ​​associated with each frequency band, for each frequency, The aforementioned tonality extractor is: The degree of tonality is determined by analyzing at least one preceding frame. The system is further configured to determine whether the degree of tonality exceeds the threshold, The adaptive sinusoidal extension and linear prediction module is, If it is determined that the degree of tonality exceeds the threshold, the TF tile value set of the replacement frame is generated using a sinusoidal expansion process. The receiving unit according to claim 39, further configured to generate the TF tile value set of the replacement frame using a linear prediction process if it is determined that the degree of tonality does not exceed the threshold.

41. The frame replacement module is, When the current frame is identified as valid, the system identifies whether the at least one preceding frame is valid or invalid. When at least one first frame is identified as invalid, obtain the TF tile value extension set of the at least one preceding frame. Generate one or more crossfaded TF tile values ​​for the replacement frame, which is a crossfade of the TF tile value set of the current frame with the TF tile value extension set of the at least one preceding frame in the crossfade region of the replacement frame. The receiving unit according to claim 39, further configured as follows.

42. The receiving unit according to claim 39, wherein the composite filter bank is a CQMF bank, and each frame includes a plurality of CQMF samples, and each sample includes at least one TF tile value associated with a frequency band.

43. The adaptive sinusoidal extension and linear prediction module is, The complex parameters are determined based on at least two TF tile values ​​of different CQMF samples in the aforementioned at least one preceding frame. The TF tile values ​​of at least one preceding frame are modified using the complex parameters, and the set of TF tile values ​​for the replacement frame is generated using sinusoidal expansion or linear prediction. The receiving unit according to claim 42, configured as follows.

44. A system comprising a receiving unit and a transmitting unit according to any one of claims 39 to 43, wherein the transmitting unit is An analysis filter bank configured to acquire a time-domain audio segment and output a time-frequency (TF) tile value representing the time-domain audio segment, An encoder configured to receive the TF tile values, encode the TF samples into data packets, each data packet containing a frame with multiple TF tile values, and transmit the data packets as a bitstream to the receiving unit, A system that includes this.

Images