Packet loss concealment

Abstract

This invention describes a method for processing audio signals for grouping lost audio signals. The audio signal includes a sequence of frames, each frame containing representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. One method includes: receiving the audio signal; and generating a reconstructed audio signal based on the received audio signal in the predetermined channel format. Generating the reconstructed audio signal includes: determining whether at least one frame of the audio signal has been lost; and if the number of consecutively lost frames exceeds a first threshold, then fading the reconstructed audio signal into a predefined spatial configuration. A method for encoding an audio signal is also described. Apparatus for implementing the method, as well as corresponding programs and computer-readable storage media, are further described.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 049,323, filed July 8, 2020 (reference: D20068USP1), and U.S. Provisional Application No. 63 / 208,896, filed June 9, 2021 (reference: D20068USP2), which are hereby incorporated by reference. Technical Field

[0003] This disclosure relates to methods and apparatus for processing audio signals. The disclosure further describes decoder processing in the codec of an Immersive Speech and Audio System (IVAS) codec in the event of packet (frame) loss, for example, to achieve the best possible audio experience. This principle is known as Packet Loss Concealment (PLC). Background Technology

[0004] Audio codecs used for encoding spatial audio, such as IVAS, involve metadata containing reconstruction parameters (e.g., spatial reconstruction parameters) that achieve an accurate spatial construction of the encoded audio. While packet loss concealment may be possible on the actual audio signal, the loss of this metadata can lead to significantly incorrect spatial reconstruction of the audio, and thus audible artifacts.

[0005] Therefore, it is necessary to improve the grouping loss hiding of metadata that includes reconstruction parameters (such as spatial reconstruction parameters). Summary of the Invention

[0006] Based on the foregoing, this disclosure provides a method for processing audio signals, a method for encoding audio signals, and corresponding devices, computer programs, and computer-readable storage media, possessing the features of the respective independent claims.

[0007] According to one aspect of this disclosure, a method for processing an audio signal is provided. The method can be performed at a receiver / decoder. The audio signal may comprise a sequence of frames. Each frame may contain representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined (or predefined) channel format. The audio signal may be a multi-channel audio signal. The predefined channel format may be first-order ambisonics (FOA), for example, having W, X, Y, and Z audio channels (components). In this case, the audio signal may contain up to four audio channels. The multiple audio channels of the audio signal may be associated with downmixed channels obtained by downmixing audio channels of a predefined channel format. The reconstruction parameters may be spatial reconstruction (SPAR) parameters. The method may include receiving an audio signal. The method may further include generating a reconstructed audio signal based on the received audio signal in a predefined channel format. The generation of the reconstructed audio signal may be based on the received audio signal and reconstruction parameters (and / or estimates of the reconstruction parameters). Furthermore, the generation of the reconstructed audio signal may involve upmixing the audio(s) of the audio signal(s). Upmixing multiple audio channels to a predefined channel format can involve reconstructing audio channels of a predefined channel format based on multiple audio channels and their decorrelation versions. The decorrelation versions can be generated based on at least some of the multiple audio channels of the audio signal and reconstruction parameters. For this purpose, the upmixing matrix can be determined based on the reconstruction parameters. Generating the reconstructed audio signal may also include determining whether at least one frame of the audio signal has been lost. Then, if the number of consecutively lost frames exceeds a first threshold, the generation may include fading the reconstructed audio signal into a predetermined (or predefined) spatial configuration. In one instance, the predefined spatial configuration may be associated with an omnidirectional audio signal. For a reconstructed FOA audio signal, this would mean retaining only the W audio channel. For example, the first threshold could be 4 or 8 frames. For example, the duration of a frame could be 20 ms.

[0008] As defined above, the proposed method can mitigate inconsistent audio in the event of packet loss, particularly for long-duration packet loss, and provides a consistent user-space experience. This may be particularly important in Enhanced Voice Services (EVS) frameworks, where the EVS hidden signals of individual audio channels may be inconsistent with each other in the event of packet loss.

[0009] In some embodiments, a predefined spatial configuration may correspond to a spatially uniform audio signal. For example, for FOA, the reconstructed audio signal faded in and out to the predefined spatial configuration may only contain the W audio channel. Alternatively, the predefined spatial configuration may correspond to a predefined direction of the reconstructed audio signal. In this case, for example, for FOA, one of the X, Y, and Z components may fade in and out to a scaled version of W, and the other two of the X, Y, and Z components may fade in and out to zero.

[0010] In some embodiments, fading the reconstructed audio signal into and out of a predefined spatial configuration may involve linear interpolation between an identity matrix and a target matrix indicating the predefined spatial configuration, based on a predefined fade-out time. In this case, the overmixing matrix used for audio reconstruction may be determined (e.g., generated) based on the matrix product of the emphasizing overmixing matrix and the interpolated matrix. Here, the emphasizing overmixing matrix may be derived based on reconstruction parameters.

[0011] In some embodiments, the method may further include: if the number of consecutively lost frames exceeds a second threshold greater than or equal to a first threshold, then gradually fading out the reconstructed audio signal. Gradually fading out (i.e., muting) the reconstructed audio signal can be achieved by applying a gradually attenuating gain to the reconstructed audio signal, multiple audio channels of the audio signal, or any upmixing coefficients used to generate the reconstructed audio signal. Gradually fading out can be performed according to a (second) predetermined fade-out time (time constant). For example, the reconstructed audio signal may be muted by 3dB per (lost) frame. For example, the second threshold may be 8 frames.

[0012] This further enhances the ability to provide a consistent user experience in the event of packet loss, especially for long periods of packet loss.

[0013] In some embodiments, the method may further include: if at least one frame of the audio signal has been lost, then generating an estimate of the reconstruction parameters of at least one lost frame based on one or more reconstruction parameters of the previous frames. The method may further include generating a reconstructed audio signal of at least one lost frame using the estimate of the reconstruction parameters of the at least one lost frame. This may be applicable if the number of lost frames is less than a predetermined number (e.g., less than a first threshold). Alternatively, this may be applicable until the reconstructed audio signal has completely spatially faded in and / or completely faded out (muteed).

[0014] In some embodiments, each reconstruction parameter may be explicitly encoded once every given number of frames in the frame sequence and differentially encoded (temporally) between the remaining frames. Furthermore, estimating a given reconstruction parameter of a lost frame may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter. Alternatively, the estimation may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. In special cases, the estimation may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (e.g., for a reconstruction parameter associated with a band having only one adjacent frequency band). Therefore, a given reconstruction parameter may be extrapolated across time or interpolated across reconstruction parameters, or, in the case of a reconstruction parameter of, for example, the lowest / highest frequency band, extrapolated from a single adjacent frequency band. Differential coding may follow a (interleaved) differential coding scheme, according to which each frame contains at least one explicitly encoded reconstruction parameter and at least one differentially encoded reconstruction parameter referenced to a previous frame, wherein the set of explicitly encoded and differentially encoded reconstruction parameters varies from frame to frame. The contents of these sets may be repeated after a predetermined frame period. It should be understood that the reconstructed parameter values ​​can be determined by correctly decoding those values.

[0015] This allows for the provision of reasonable reconstruction parameters (e.g., SPAR parameters) in the event of packet loss, enabling a consistent spatial experience based on, for example, EVS hidden signals. Furthermore, this enables the provision of optimal reconstruction parameters (e.g., SPAR parameters) after packet loss when temporal differential coding is applied.

[0016] In some embodiments, the method may further include determining a reliability metric for a most recently determined value of a given reconstruction parameter. The method may further include determining, based on the reliability metric, whether to estimate the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter or based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter (in special cases, a single reconstruction parameter). The reliability metric may be determined based on the years (e.g., in frames) of the most recently determined value of the given reconstruction parameter and / or the years (e.g., in frames) of the most recently determined values ​​of reconstruction parameters other than the given reconstruction parameter.

[0017] In some embodiments, the method may further include: if the number of frames for which the value of a given reconstruction parameter cannot be determined exceeds a third threshold, then estimating the given reconstruction parameter of the lost frame based on the most recently determined value of a reconstruction parameter other than the given reconstruction parameter. The method may further include: otherwise estimating the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter.

[0018] In some embodiments, each frame may include reconstruction parameters associated with a corresponding frequency band. The given reconstruction parameters of a lost frame may be estimated based on one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameters are associated.

[0019] In some embodiments, a given reconstruction parameter can be estimated by interpolation between reconstruction parameters associated with a frequency band that is different from the frequency band to which the given reconstruction parameter is associated. In a special case, for a frequency band at the boundary of the covered frequency range (i.e., the highest or lowest frequency band), a given reconstruction parameter for a lost frame can be estimated by extrapolation from reconstruction parameters associated with a frequency band adjacent to (or closest to) the highest or lowest frequency band.

[0020] In some embodiments, a given reconstruction parameter can be estimated by interpolation between reconstruction parameters associated with frequency bands adjacent to the frequency band to which the given reconstruction parameter is associated. Alternatively, if the frequency band to which the given reconstruction parameter is associated has only one adjacent frequency band, then the reconstruction parameter can be estimated by extrapolation from the reconstruction parameters associated with that adjacent frequency band.

[0021] According to another aspect of this disclosure, a method for processing an audio signal is provided. For example, the method may be performed at a receiver / decoder. The audio signal may comprise a sequence of frames. Each frame may comprise representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. The method may comprise receiving the audio signal. The method may further comprise generating a reconstructed audio signal based on the received audio signal in a predefined channel format. Generating the reconstructed audio signal may include determining whether at least one frame of the audio signal has been lost. The generation may further comprise: if at least one frame of the audio signal has been lost, then generating an estimate of the reconstruction parameters of at least one lost frame based on the reconstruction parameters of previous frames. Furthermore, the generation may comprise generating a reconstructed audio signal of at least one lost frame using the estimate of the reconstruction parameters of the at least one lost frame.

[0022] In some embodiments, each reconstruction parameter may be explicitly encoded once every given number of frames in the frame sequence and differentially encoded (temporally) between the remaining frames. Then, estimating the given reconstruction parameter of the lost frame may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter. Alternatively, the estimation may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. In a special case, the estimation may involve estimating the given reconstruction parameter of the lost frame based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (e.g., for a reconstruction parameter associated with a band having only one adjacent frequency band).

[0023] In some embodiments, the method may further include a reliability metric for determining the most recently determined value of a given reconstruction parameter. The method may also further include, based on the reliability metric, determining whether to estimate the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter or based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter (or, in special cases, a single reconstruction parameter).

[0024] In some embodiments, the method may further include: if the number of frames for which the value of a given reconstruction parameter cannot be determined exceeds a third threshold, then estimating the given reconstruction parameter of the lost frame based on the most recently determined values ​​of two or more reconstruction parameters (in special cases, a single reconstruction parameter) in addition to the given reconstruction parameter. The method may further include: otherwise estimating the given reconstruction parameter of the lost frame based on the most recently determined value of the given reconstruction parameter.

[0025] In some embodiments, each frame may contain reconstruction parameters associated with the corresponding frequency band. Then, the given reconstruction parameters of the lost frame may be estimated based on one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameters are associated.

[0026] In some embodiments, a given reconstruction parameter can be estimated by interpolation between a reconstruction parameter associated with a frequency band that is different from the frequency band to which the given reconstruction parameter is associated.

[0027] In some embodiments, a given reconstruction parameter can be estimated by interpolation between reconstruction parameters associated with a frequency band adjacent to the frequency band to which the given reconstruction parameter is associated. Alternatively, if the given reconstruction parameter has only one adjacent frequency band, then the given reconstruction parameter can be estimated by extrapolation from the reconstruction parameters associated with that adjacent frequency band.

[0028] According to another aspect of this disclosure, a method for processing an audio signal is provided. For example, the method may be performed at a receiver / decoder. The audio signal may comprise a sequence of frames. Each frame may contain representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. Each reconstruction parameter may be explicitly encoded once every given number of frames in the frame sequence and differentially encoded between the remaining frames. The method may comprise receiving an audio signal. The method may further comprise generating a reconstructed audio signal based on the received audio signal in a predefined channel format. Generating the reconstructed audio signal may include: for a given frame of the audio signal, identifying correctly decoded reconstruction parameters and reconstruction parameters that cannot be correctly decoded due to a lack of differential bases. The generation may further comprise: for a given frame, estimating reconstruction parameters that cannot be correctly decoded based on the correctly decoded reconstruction parameters of the given frame and / or the correctly decoded reconstruction parameters of one or more previous frames. The generation may further comprise: for a given frame, generating the reconstructed audio signal of the given frame using the correctly decoded reconstruction parameters and the estimated reconstruction parameters.

[0029] In some embodiments, estimating a given reconstruction parameter that cannot be correctly decoded for a given frame may involve estimating the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter. Alternatively, the estimation may involve estimating the given reconstruction parameter based on the most recently correctly decoded values ​​of two or more reconstruction parameters other than the given reconstruction parameter. In special cases, the given reconstruction parameter of a lost frame may be estimated based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (e.g., for a reconstruction parameter associated with a band having only one adjacent frequency band).

[0030] In some embodiments, the method may further include a reliability measure for determining the most recently correctly decoded value of a given reconstruction parameter. The method may further include determining, based on the reliability measure, whether to estimate the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter or based on the most recently correctly decoded values ​​of two or more reconstruction parameters (or, in special cases, a single reconstruction parameter) other than the given reconstruction parameter.

[0031] In some embodiments, the method may further include: if the most recently correctly decoded value of a given reconstruction parameter is older than a predetermined threshold in frames, then estimating the given reconstruction parameter based on the most recently correctly decoded values ​​of two or more reconstruction parameters (in special cases, a single reconstruction parameter) other than the given reconstruction parameter. The method may further include: otherwise estimating the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter.

[0032] In some embodiments, each frame may contain reconstruction parameters associated with a corresponding frequency band. Then, a given reconstruction parameter that cannot be correctly decoded for a given frame may be estimated based on the most recently correctly decoded values ​​of one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated.

[0033] In some embodiments, a given reconstruction parameter can be estimated by interpolation between a reconstruction parameter associated with a frequency band that is different from the frequency band to which the given reconstruction parameter is associated.

[0034] In some embodiments, a given reconstruction parameter can be estimated by interpolation between reconstruction parameters associated with a frequency band adjacent to the frequency band to which the given reconstruction parameter is associated. Alternatively, if the given reconstruction parameter has only one adjacent frequency band, then the given reconstruction parameter can be estimated by extrapolation from the reconstruction parameters associated with that adjacent frequency band.

[0035] According to another aspect of this disclosure, a method for encoding an audio signal is provided. For example, the method may be performed at an encoder. The encoded audio signal may comprise a sequence of frames. Each frame may contain representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. The method may include: for each reconstruction parameter, explicitly encoding the reconstruction parameter once every given number of frames in the frame sequence. The method may further include inter-frame (temporal) differential encoding of the reconstruction parameters in the remaining frames. Each frame may contain at least one explicitly encoded reconstruction parameter and at least one differentially encoded reconstruction parameter referenced to a previous frame. The set of explicitly encoded and differentially encoded reconstruction parameters may vary from frame to frame. Furthermore, the contents of these sets may be repeated after a predetermined frame period.

[0036] According to another aspect, a computer program is provided. The computer program may contain instructions that, when executed by a processor, cause the processor to perform all the steps of the methods described throughout this disclosure.

[0037] According to another aspect, a computer-readable storage medium is provided. This computer-readable storage medium can store the aforementioned computer program.

[0038] According to another aspect, an apparatus is provided that includes a processor and memory coupled to the processor. The processor can be adapted to perform all steps of the methods described throughout this disclosure. This apparatus can be associated with a receiver / decoder (decoder device) or an encoder (encoder device).

[0039] It should be understood that the features of the equipment and the steps of the method can be interchanged in many ways. Specifically, as a person skilled in the art should understand, the details of the disclosed method can be implemented by the corresponding equipment, and vice versa. Furthermore, it should be understood that any of the statements made above regarding the method (and, for example, its steps) also apply to the corresponding equipment (and, for example, its blocks, stages, units), and vice versa. Attached Figure Description

[0040] The following explanation of exemplary embodiments of this disclosure is based on the accompanying drawings, wherein...

[0041] Figure 1 This is a flowchart illustrating an example stream in the case of packet loss and good frames according to embodiments of the present disclosure.

[0042] Figure 2 This is a block diagram illustrating an example encoder and decoder according to embodiments of the present disclosure.

[0043] Figure 3 and Figure 4 This is a flowchart illustrating an example process of a PLC according to an embodiment of the present disclosure.

[0044] Figure 5 Showing the implementation Figures 1 to 4 Examples of mobile device architectures that describe the features and processes described in the text.

[0045] Figures 6 to 9 This is a flowchart illustrating an additional example of a method for processing (e.g., decoding) an audio signal according to embodiments of the present disclosure, and

[0046] Figure 10 This is a flowchart illustrating an example of a method for encoding audio signals according to embodiments of the present disclosure. Detailed Implementation

[0047] The figures and the following description relate to preferred embodiments for illustrative purposes only. It should be noted that, from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily regarded as feasible alternatives that can be employed without departing from the principles claimed.

[0048] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that, where feasible, similar or analogous reference numerals may be used in the figures and may indicate similar or analogous functionality. The figures depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods described herein may be employed without departing from the principles described herein.

[0049] Overview

[0050] In a broad sense, the technology according to this disclosure may include:

[0051] 1. Preservation of reconstruction parameters (e.g., SPAR parameters) during packet loss from the last correct frame.

[0052] 2. Mute or manipulate spatial images after prolonged periods of packet loss to mitigate inconsistent hidden signals (e.g., EVS hidden signals), and

[0053] 3. Perform reconstruction parameter estimation in the case of temporal differential coding after packet loss.

[0054] IVAS system

[0055] First, possible implementations of the IVAS system will be described as non-limiting examples of systems to which the technology of this disclosure is applicable.

[0056] IVAS provides a spatial audio experience for communication and entertainment applications. The basic spatial audio format is first-order stereo reverberation (FOA). For example, four signals (W, Y, Z, X) are encoded, allowing rendering to any desired output format, such as immersive speaker playback or binaural reproduction via headphones. Depending on the total bit rate, one, two, three, or four audio signals (downmix channels) are transmitted via an EVS (Enhanced Voice Services) codec operating in parallel with low latency. At the decoder, the four FOA signals are reconstructed by processing the downmix channels and their decorrelation versions using the transmitted parameters. This process is also referred to herein as upmixing, and the parameters are called Spatial Reconstruction (SPAR) parameters. The IVAS decoding process consists of EVS (core) decoding and SPAR upmixing. The EVS decoded signal is converted by a complex-valued low-latency filter bank. The SPAR parameters are encoded by perceptual excitation bands, and the number of bands is typically 12. Except for the W channel, the encoded downmix channels are the residual signals after (cross-channel) prediction using the SPAR parameters. The W channel is transmitted unmodified or modified (effective W), making better predictions for the remaining channels possible. After SPAR upmixing in the frequency domain, the FOA time-domain signal is combined using filters to generate the signal. An audio frame typically has a duration of 20ms.

[0057] In summary, the IVAS decoding process consists of EVS core decoding of the downmixing channel, filter bank analysis, parameter reconstruction of the four FOA signals (upmixing), and filter combination.

[0058] Especially at low bit rates such as 32kb / s or 64kb / s, the SPAR parameter can, for example, depend on the previously decoded frames being time-differential encoded to achieve a lower SPAR bit rate.

[0059] Generally, the techniques (e.g., methods and apparatus) according to embodiments of this disclosure are applicable to frame-based (or packet-based) multi-channel audio signals, i.e., (encoded) audio signals comprising a sequence of frames (or packets). Each frame contains a representation of multiple audio channels and reconstruction parameters (e.g., SPAR parameters) for upmixing the multiple audio channels to a predetermined channel format (e.g., a FOA having W, X, Y, and Z audio channels (components)). The multiple audio channels of the (encoded) audio signal can be associated with downmixed channels obtained by downmixing audio channels of a predefined channel format (e.g., W, X, Y, and Z).

[0060] IVAS system constraints

[0061] EVS and SPAR-DTX

[0062] If no voice activity (VAD) is detected and the background level is low, the EVS encoder can switch to Discontinuous Transmission (DTX) mode, operating at a very low bit rate. Typically, a small number of DTX parameters (Silence Indicator Frame SID) are transmitted every 8 frames, controlling Comfort Noise Generation (CNG) at the decoder. Similarly, the dedicated SPAR parameter for the SID frame is transmitted, allowing for faithful spatial reconstruction of the original spatial environment characteristics. The SID frame is followed by 7 frames with no data (NO_DATA), and the SPAR parameter remains constant until the next SID frame or an active audio frame is received.

[0063] EVS-PLC

[0064] If the EVS decoder detects a lost frame, a hidden signal is generated. The generation of the hidden signal can be guided by the signal classification parameters sent by the encoder in a previous correct frame (without a hidden signal), and depends on the codec mode (MDCT-based transformational codec or predictive speech codec) and other parameters using various techniques. EVS hiding can lead to an infinite amount of comfort noise generation. Because multiple instances of EVS (one per downmix channel) run in parallel in different configurations for IVAS, EVS hiding can span downmix channels and is inconsistent for different content.

[0065] It should be noted that EVS-PLC does not support metadata such as SPAR parameters.

[0066] Temporal differential coding of reconstructed parameters

[0067] The techniques according to embodiments of this disclosure are applicable to codecs employing temporal differential encoding that includes metadata containing reconstruction parameters (e.g., SPAR parameters). Unless otherwise indicated, differential encoding in the context of this disclosure should mean temporal differential encoding.

[0068] For example, each reconstruction parameter may be explicitly (i.e., non-differentially) encoded once every given number of frames in a frame sequence and differentially encoded between the remaining frames. Temporal differential encoding may follow a (interleaved) differential encoding scheme, according to which each frame contains at least one explicitly encoded reconstruction parameter and at least one differentially encoded reconstruction parameter referenced from a previous frame. The sets of explicitly encoded and differentially encoded reconstruction parameters may vary from frame to frame. The contents of these sets may be repeated after a predetermined frame period. For example, the contents of the aforementioned sets may be given by a group of (interleaved) encoding schemes that can be cyclically passed sequentially. Non-limiting examples of such encoding schemes applicable to, for example, IVAS contexts are given below.

[0069] For effective encoding of SPAR parameters, temporal differential encoding can be applied, for example, according to the following scheme:

[0070] Encoding scheme Time differential coding, with 1 to 12 cardinality 0 0 0 0 0 0 0 0 0 0 0 0 4a 0 1 1 1 0 1 1 1 0 1 1 1 4b 1 0 1 1 1 0 1 1 1 0 1 1 4c 1 1 0 1 1 1 0 1 1 1 0 1 4d 1 1 1 0 1 1 1 0 1 1 1 0

[0071] Table 1 shows the SPAR coding scheme for the time-differential encoded band with an indication of 1.

[0072] encoding scheme of previous frames Temporal differential coding scheme for the current frame cardinality 4a 4a 4b 4b 4c 4c 4d 4d 4a

[0073] Table 2 Application sequence of the time-difference SPAR coding scheme

[0074] Here, time differential coding always cycles through 4a, 4b, 4c, 4d and returns to start again from 4a. Depending on the payload and total bit rate requirements of the basic scheme, time differential coding may or may not be applied.

[0075] This encoding method ensures that, after packet loss, the parameters for the three bands (for a 12-parameter band configuration; other schemes can be applied similarly to other parameters) can always be correctly decoded, instead of temporal differential encoding for all bands. Changing the encoding scheme as shown in Table 2 ensures that the parameters for all bands can be correctly decoded within four consecutive (no packet loss) frames. However, depending on the type of packet loss, the parameters for some bands may not be correctly decoded beyond four frames.

[0076] Example Techniques

[0077] Prerequisites

[0078] 1. The logic in the decoder keeps track of frame types (e.g., NO_DATA, SID, and valid frames) so that DTX and lost / bad frames can be handled differently.

[0079] 2. The logic in the decoder, which is used to keep track of a consecutive number of lost packets.

[0080] 3. Logic, which is used to keep track of the reconstructed parameters (e.g., SPAR parameters) bands that are time-differential encoded and the number of frames since the last cardinality after packet loss (e.g., without coded differential cardinality).

[0081] An example of the above logic is illustrated below in pseudocode to decode a frame with SPAR parameters covering 12 frequency bands.

[0082]

[0083]

[0084] List 1. Logic for controlling the IVAS decoding process based on packet loss

[0085] The proposed processing

[0086] Generally, it should be understood that the methods according to embodiments of this disclosure are applicable to (encoded) audio signals comprising frame sequences (groups), each frame containing representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. Typically, such methods include receiving the audio signal and generating a reconstructed audio signal in a predefined channel format based on the received audio signal.

[0087] Next, examples of processing steps that can be used to generate reconstructed audio signals in the context of IVAS will be described. However, it should be understood that these processing steps are not limited to IVAS and are generally applicable to PLCs that use frame-based (packet-based) audio codec reconstruction parameters.

[0088] 1. Mute: If the number of consecutively lost frames exceeds a threshold (the second threshold in claim (e.g., 8), then the decoded output (e.g., FOA output) is (gradually) muted, for example, muted by 3dB per (lost frame). Otherwise, no mute is applied. Mute can be accomplished by modifying the overmixing matrix (e.g., the SPAR overmixing matrix) accordingly. Mute makes the PLC more consistent in terms of bit span rate and content for long-duration packet loss. Due to the above logic, mute can also be applied in the case of a CNG with DTX if needed.

[0089] Generally, if the number of consecutively lost frames exceeds a threshold (the second threshold in claims), the reconstructed audio signal can be gradually faded out (muteed). Fading out (muteing) the reconstructed audio signal can be achieved by applying a gradually decreasing gain to the reconstructed audio signal, by applying a gradually decreasing gain to multiple audio channels of the audio signal, or by applying a gradually decreasing gain to any upmixing coefficients used to generate the reconstructed audio signal. The fade-out can be performed according to a predetermined fade-out time (time constant). For example, as described above, the reconstructed audio signal can be muted by 3dB per (lost) frame. For example, the second threshold can be 8 frames.

[0090] 2. Spatial Fade-out: If the number of consecutively lost frames exceeds a threshold (the first threshold in claim 4 or 8, for example), then the decoded output (e.g., the FOA output) is spatially faded in and out toward a predefined number of frames toward a spatial target (i.e., faded in and out to a predefined spatial configuration). Otherwise, spatial fade-out is not applied. Spatial fade-out can be accomplished by linear interpolation between an identity matrix (e.g., 4x4) and a spatial target matrix according to a pre-defined fade-out time. As an example, orientation-independent spatial images (e.g., mute all channels except W) can reduce spatial discontinuities after group loss (if not completely mute). That is, for FOA, the predefined spatial configuration can only contain the W audio channel. Alternatively, the predefined spatial configuration can be associated with a predefined orientation. For example, another useful spatial target for FOA is a frontal image (X = W sqrt(2), Y = Z = 0). That is, one of the X, Y, and Z components (e.g., X) can be faded in and out to a scaled version of W, and the other two of the X, Y, and Z components (e.g., Y and Z) can be faded in and out to zero. In any case, the resulting matrix is ​​then applied to the SPAR upmixing matrix for all bands. Thus, the (SPAR) upmixing matrix used for audio reconstruction can be determined (e.g., generated) based on the matrix product of the silence upmixing matrix and the interpolated matrix, where the silence upmixing matrix can be derived from the reconstruction parameters. Spatial fade-out makes the PLC more consistent in terms of packet loss rate and content for long durations. Due to the above logic, spatial fade-in and fade-out can also be applied in the case of CNG with DTX if needed. The FOA format is used as a non-limiting example. Other formats can also be used, such as channel-based spatial formats that include stereo. It should be understood that a particular format may use a specific corresponding spatial fade-in and fade-out matrix.

[0091] Generally, generating a reconstructed audio signal may include: if the number of consecutively lost frames exceeds a threshold (the first threshold in claims), then fading the reconstructed audio signal into and out of a predefined spatial configuration. As stated above, this predefined spatial configuration may correspond to a spatially uniform audio signal or a predefined direction (e.g., the reconstructed audio signal is rendered to its predefined direction). It should be understood that the (first) threshold for spatial fading in and out may be less than or equal to the (second) threshold for fading out (mute). Therefore, if the above processing steps are combined, the reconstructed audio signal may first be faded in and out of the predefined spatial configuration, followed by mute, or in conjunction with mute.

[0092] 3. Parameter Estimation / Recovery from Packet Loss Using Time Differential Coding: Due to the logic described above, parameter bands that have not yet been correctly decoded due to the lack of a time differential cardinality can be identified. These parameter bands can be allocated from previous frame data, as in the case of hidden packet loss. As an alternative strategy, a cross-band linear (or nearest neighbor) interpolation is proposed when the last received cardinality (or generally, the last correctly decoded parameter for a particular parameter) is considered too old. For bands at the boundaries of the covered frequency range, this can be equivalent to extrapolation from their corresponding adjacent (or nearest) bands. The proposed method is advantageous because interpolation within correctly decoded bands is likely to yield better parameter estimates than using old previous frame data along with new correctly decoded data.

[0093] It is evident that the proposed method can be applied to both cases involving PLCs with infrequent packet loss (e.g., before or during space fade-in and / or mute, until the reconstructed audio signal has fully faded in and out) and cases involving recovery after sudden packet loss.

[0094] Generally, when at least one frame of an audio signal has been lost, the reconstruction parameters of at least one lost frame can be estimated based on the reconstruction parameters of previous frames. These estimates can then be used to generate a reconstructed audio signal of at least one lost frame.

[0095] For example, a given reconstruction parameter of a lost frame can be extrapolated across time or interpolated / extrapolated across frequency (generally, interpolated / extrapolated across other reconstruction parameters). In the former case, the given reconstruction parameter of the lost frame can be estimated based on the most recently determined value of the given reconstruction parameter. In the latter case, the given reconstruction parameter of the lost frame can be estimated based on the most recently determined values ​​of one (in the case of a frequency band at the boundary of the covered frequency range), two, or more reconstruction parameters other than the given reconstruction parameter.

[0096] Whether to use interpolation across time or interpolation / extrapolation across other reconstruction parameters can be determined based on a reliability metric of the most recently determined value of the given reconstruction parameter. That is, the reliability metric can be used to determine whether the given reconstruction parameter of the lost frame is estimated based on the most recently determined value of the given reconstruction parameter or on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. This reliability metric can be determined based on the years (e.g., in frames) of the most recently determined value of the given reconstruction parameter and / or the years (e.g., in frames) of the most recently determined values ​​of reconstruction parameters other than the given reconstruction parameter. In one implementation, if the number of frames for which the value of the given reconstruction parameter cannot be determined exceeds a third threshold, then the given reconstruction parameter of the lost frame can be estimated based on the most recently determined values ​​of one, two, or more reconstruction parameters other than the given reconstruction parameter. Otherwise, the given reconstruction parameter of the lost frame can be estimated based on the most recently determined value of the given reconstruction parameter.

[0097] As described above, each frame may contain reconstruction parameters associated with a corresponding frequency band, and a given reconstruction parameter of a lost frame may be estimated based on one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated. For example, a given reconstruction parameter may be estimated by interpolation (or extrapolation from) one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated. More specifically, in some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters associated with a frequency band adjacent to the frequency band to which the given reconstruction parameter is associated, or, if the frequency band to which the given reconstruction parameter is associated has a unique adjacent (or nearest) frequency band (which is for the highest and lowest frequency bands), then it may be estimated by extrapolation from the reconstruction parameters associated with that adjacent (or nearest) frequency band.

[0098] It should be understood that, in general, the above processing steps can be used alone or in combination. That is, the method according to this disclosure may involve any one, two, or all of the aforementioned processing steps 1 to 3.

[0099] Summary of key aspects of this disclosure

[0100] -This disclosure proposes the concept of a PLC and a spatial fade-out target, potentially in conjunction with silence.

[0101] - This disclosure proposes the concept of frames with a hybrid of hidden and conventional decoding in the temporal differential coding recovery stage.

[0102] This may involve

[0103] In the case of temporal differential coding, parameters are determined by interpolation based on previously correct frame data and / or currently correctly decoded parameters after packet loss.

[0104] The decision is based on how many new metrics are present between the previous correct frame data and / or the current interpolated data.

[0105] Example process and system

[0106] Figure 1 This is a flowchart illustrating the instance flow in cases of packet loss (left path) and correct frames (right path). The flowchart preceding the "Generate Upmixing Matrix" block is detailed in pseudocode in Listing 1 and described in item 3 of the process presented in the previous section. The process in "Modify Upmixing Matrix" is described in items 1 and 2 of the process presented in the previous section.

[0107] Figure 2 This is a block diagram illustrating an example IVAS SPAR encoder and decoder. The IVAS upmixing matrix includes the decoded downmixed channel, the decorrelation version with parameters (C, P1, ..., PD), the inverse remixing matrix, and the inverse prediction, all processed into a single upmixing matrix. The upmixing matrix can be modified via PLC processing.

[0108] Figure 3 and Figure 4 This is a flowchart illustrating an example process of a PLC.

[0109] Example System Architecture

[0110] Figure 5 This is a reference for implementation based on the embodiments. Figures 1 to 4 The described features and processes pertain to a mobile device architecture. Architecture 800 can be implemented in any electronic device, including (but not limited to): desktop computers, consumer audio / video (AV) equipment, radio broadcasting equipment, and mobile devices (e.g., smartphones, tablets, laptops, wearable devices). In the illustrated example embodiment, architecture 800 is used in a smartphone and includes a processor 801, a peripheral interface 802, an audio subsystem 803, a speaker 804, a microphone 805, sensors 806 (e.g., accelerometers, gyroscopes, barometers, magnetometers, cameras), a positioning processor 807 (e.g., a GNSS receiver), a wireless communication subsystem 808 (e.g., Wi-Fi, Bluetooth, cellular), and an I / O subsystem 809, which includes a touch controller 810 and other input controllers 811, a touch surface 812, and other input / control devices 813. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

[0111] Memory interface 814 is coupled to processor 801, peripheral device interface 802, and memory 815 (e.g., flash memory, RAM, ROM). Memory 815 stores computer program instructions and data, including (but not limited to): operating system instructions 816, communication instructions 817, GUI instructions 818, sensor processing instructions 819, telephone instructions 820, electronic messaging instructions 821, web browsing instructions 822, audio processing instructions 823, GNSS / navigation instructions 824, and application programs / data 825. Audio processing instructions 823 include instructions for executing reference... Figures 1 to 2 The audio processing instructions described in [the document].

[0112] Audio processing and PLC technology for parameter reconstruction

[0113] An example of a PLC in the context of IVAS has been described above. It should be understood that the concepts provided in that context are generally applicable to PLCs that handle frame-based (group-based) audio signal reconstruction parameters. References will now be made. Figures 6 to 10 Describe additional examples of methods that employ these concepts.

[0114] An overview of the general approach to processing audio signals 600 is as follows: Figure 6 As described above, the (encoded) audio signal includes a sequence of frames, each frame containing representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. Method 600 includes steps S610 and S620, which may include additional sub-steps, which will be referenced below. Figures 7 to 9 This will be described in detail. Furthermore, for example, method 600 can be performed at the receiver / decoder.

[0115] exist Step S610 At this location, (encoded) audio signals are received. For example, the audio signal can be received as a (blocked) bitstream.

[0116] exist Step S620 At the location, a reconstructed audio signal is generated based on the received audio signal in a predefined channel format. The reconstructed audio signal may be generated based on the received audio signal and reconstruction parameters (and / or estimates of the reconstruction parameters, as detailed below). Furthermore, generating the reconstructed audio signal may involve upmixing audio channels of the audio signal to a predefined channel format. Upmixing audio channels to a predefined channel format may involve reconstructing audio channels of the audio signal based on its audio channels and their decorrelated versions in a predefined channel format. The decorrelated versions may be generated based on at least some of the audio channels of the audio signal and the reconstruction parameters.

[0117] Figure 7A method 700 for generating a reconstructed audio signal at step S620 is illustrated, including example (sub)steps S710, S720, and S730. It should be understood that steps S720 and S730 are related to possible embodiments of step S620, which may be used individually or in combination. That is, step S620 may (other than step S710) not include steps S720 and S730, or include any or both of steps S720 and S730.

[0118] exist Step S710 At this point, determine whether at least one frame of the audio signal has been lost. This can be done according to the description above in the prerequisites section.

[0119] If so, Step S720 Furthermore, if the number of consecutively lost frames exceeds a first threshold, the reconstructed audio signal is faded in and out to a predefined spatial configuration. This can be accomplished according to step 2 of the processing described in the previous section.

[0120] Alternatively or in Step S730 If the number of consecutively lost frames exceeds a second threshold that is greater than or equal to the first threshold, then the reconstructed audio signal is gradually faded out (muted). This can be accomplished according to step 1 of the processing described in the previous section.

[0121] Figure 8 A method 800 for generating a reconstructed audio signal at step S620 is illustrated, including example (sub)steps S810, S820, and S830. It should be understood that steps S810 to S830 can be performed individually or in conjunction with... Figure 7 The possible implementation schemes are related to the steps of S620 used in the combination of possible implementation schemes.

[0122] exist Step S810 At this point, determine whether at least one frame of the audio signal has been lost. This can be done according to the description above in the prerequisites section.

[0123] Next, in Step S820 If at least one frame of the audio signal has been lost, then estimates of the reconstruction parameters for at least one lost frame are generated based on one or more reconstruction parameters of the previous frames. This can be accomplished according to step 3 of the processing described in the preceding section.

[0124] exist Step S830 At this point, a reconstructed audio signal for at least one lost frame is generated using an estimate of the reconstruction parameters for at least one lost frame. This can be done as discussed above with respect to step S620, for example, via upmixing. It should be understood that if the actual audio channel has also been lost, its estimate can be used instead. The EVS hidden signal is an example of such an estimate.

[0125] Method 800 can be applied as long as the number of lost frames is less than a predetermined number (e.g., less than a first threshold or a second threshold). Alternatively, Method 800 can be applied until the reconstructed audio signal has completely spatially faded in and / or completely faded out. Thus, in the case of continuous packet loss, Method 800 can be used to mitigate packet loss before or until silence / spatial fade-in / fade-out takes effect. However, it should be noted that the concept of Method 800 can also be used to recover from burst packet loss when temporal differential coding of the reconstruction parameters is present.

[0126] Now refer to Figure 9 Examples of this method for processing audio signals to recover from burst packet loss are described, such as those that can be performed at a receiver / decoder. As previously assumed, the audio signal comprises a sequence of frames, each containing representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. Furthermore, it is assumed that each reconstruction parameter is explicitly encoded once every given number of frames in the frame sequence and differentially encoded between the remaining frames. This can be accomplished using temporal differential encoding of the reconstruction parameters according to the preceding section. Similar to method 600, the method for processing audio signals to recover from burst packet loss includes receiving the audio signal (similar to step S610) and generating a reconstructed audio signal based on the received audio signal in a predefined channel format (similar to step S620). Figure 9 The method 900 shown includes steps S910, S920, and S930, which are sub-steps for generating a reconstructed audio signal based on the audio signal received for a given frame in a predefined channel format. It should be understood that the method for recovering from burst packet loss can be applied to correctly received frames (e.g., the first few frames) following several lost frames.

[0127] exist Step S910 At this point, the system identifies correctly decoded reconstruction parameters and those that cannot be correctly decoded due to a lack of temporal differential cardinality. If several frames (packets) have been lost in the past, this is expected to result in a lack of temporal differential cardinality.

[0128] exist Step S920 At this point, the reconstruction parameters that cannot be correctly decoded are estimated based on the correctly decoded reconstruction parameters of a given frame and / or the correctly decoded reconstruction parameters of one or more previous frames. This can be accomplished according to the processing step 3 proposed in the above section.

[0129] For example, estimating a given reconstruction parameter that cannot be correctly decoded for a given frame (due to a lack of temporal differential cardinality) could involve estimating the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter (e.g., the last correctly decoded value before the (burst) packet loss) or based on the most recently correctly decoded values ​​of one or more other reconstruction parameters besides the given reconstruction parameter. It is obvious that the most recently correctly decoded values ​​of one or more other reconstruction parameters besides the given reconstruction parameter may have already been decoded for / from the (current) given frame. Which of the two methods can be based on a reliability metric of the most recently correctly decoded value of the given reconstruction parameter should be determined. For example, this metric could be the age of the most recently correctly decoded value of the given reconstruction parameter. For instance, if the most recently correctly decoded value of the given reconstruction parameter is older than a predetermined threshold (e.g., in frames), then the given reconstruction parameter can be estimated based on the most recently correctly decoded values ​​of one or more other reconstruction parameters besides the given reconstruction parameter. Otherwise, the given reconstruction parameter can be estimated based on the most recently correctly decoded value of the given reconstruction parameter. However, it should be understood that other reliability metrics are also feasible.

[0130] Depending on the applicable codec (e.g., IVAS, for example), each frame may contain reconstruction parameters associated with corresponding bands in multiple frequency bands. A given reconstruction parameter that cannot be correctly decoded for a given frame can then be estimated based on the most recently correctly decoded value of one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated. For example, a given reconstruction parameter can be estimated by interpolation among reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated. In some cases, a given reconstruction parameter can be extrapolated from a single reconstruction parameter associated with a frequency band different from the frequency band to which the given reconstruction parameter is associated. Specifically, a given reconstruction parameter can be estimated by interpolation among reconstruction parameters associated with frequency bands adjacent to the frequency band to which the given reconstruction parameter is associated. If the frequency band to which the given reconstruction parameter is associated has only one adjacent (or nearest) frequency band (for example, in the case of the highest and lowest frequency bands), then the given reconstruction parameter can be estimated by extrapolation from the reconstruction parameters associated with that adjacent (or nearest) frequency band.

[0131] exist Step S930 At this point, the reconstructed audio signal for a given frame is generated using the correctly decoded reconstruction parameters and the estimated reconstruction parameters. This can be done as described above for step S620, for example, via upmixing.

[0132] The scheme for time-difference coding of reconstructed parameters has been described in the previous section on time-difference coding of reconstructed parameters. It should be understood that this disclosure also relates to a method for encoding audio signals to which this time-difference coding is applied. Examples of this method 1000 for encoding audio signals are provided in... Figure 10 The diagram is schematically illustrated. It is assumed that the encoded audio signal comprises a sequence of frames, where each frame contains representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. Therefore, method 1000 generates an encoded audio signal that can be decoded, for example, by any of the methods described above. Method 1000 includes steps S1010 and S1020 that can be performed for each reconstruction parameter (e.g., SPAR parameter) to be encoded.

[0133] exist Step S1010 At each given number of frames in the frame sequence, the parameters are reconstructed once by explicit encoding (e.g., non-differential encoding or plaintext encoding).

[0134] exist Step S1020 At this point, the parameters are reconstructed by differential coding between (time) frames of the remaining frames.

[0135] A choice can be made between differential and non-differential encoding of the corresponding reconstruction parameters for a given frame, such that each frame contains at least one explicitly encoded reconstruction parameter and at least one (temporally) differentially encoded reconstruction parameter referencing a previous frame. Furthermore, to ensure recoverability in the event of packet loss, the set of explicitly and differentially encoded reconstruction parameters varies from frame to frame. For example, the set of explicitly and differentially encoded reconstruction parameters can be selected according to a grouping scheme, which is periodically cyclically circulated. That is, the contents of the aforementioned set of reconstruction parameters can be repeated after a predetermined frame period. It should be understood that each reconstruction parameter is explicitly encoded once every given number of frames. Preferably, this given number of frames is the same for all reconstruction parameters.

[0136] advantage

[0137] As partially outlined in the preceding sections, the techniques described in this disclosure can provide PLCs with the following technical advantages over conventional techniques.

[0138] 1. Provide reasonable reconstruction parameters (e.g., SPAR parameters) in the event of packet loss to provide a consistent spatial experience based on, for example, EVS hidden signals.

[0139] 2. Mitigating inconsistencies in lost audio data due to prolonged periods of lost packets (e.g., EVS hiding).

[0140] 3. Provides optimal reconstruction parameters (e.g., SPAR parameters) after packet loss when temporal differential coding is applied.

[0141] explain

[0142] Aspects of the system described herein can be implemented in a suitable computer-based audio processing network environment to process digital or digitized audio files. A portion of the adaptive audio system may comprise one or more networks, including any desired number of individual machines, and one or more routers (not shown) for buffering and routing data transmitted between computers. This network can be established on various network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0143] One or more of the components, blocks, processes, or other functional components may be implemented by a computer program executed by a processor-based computing device controlling the system. It should also be noted that the various functions disclosed herein may be described and / or described in terms of their behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware and firmware, and embodied in data and / or instructions in various machine-readable or computer-readable media. Such computer-readable media embodying formatted data and / or instructions may include (but are not limited to) various forms of physical (non-transitory) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0144] While one or more embodiments have been described by way of example and according to specific embodiments, it should be understood that one or more embodiments are not limited to the disclosed embodiments. On the contrary, as will be appreciated by those skilled in the art, it is desirable to cover various modifications and similar arrangements. Therefore, the scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and similar arrangements.

[0145] Enumeration Examples

[0146] Various aspects and implementations of this disclosure can also be understood from the exemplary embodiments (EEEs) enumerated below, which are not claims.

[0147] EEE1. A method for processing audio, comprising: determining whether the number of consecutively lost frames satisfies a threshold; and in response to determining that the number satisfies the threshold, outputting a spatial fade-in / fade-out via decoded first-order stereo reverb (FOA).

[0148] EEE2. The method according to EEE1, wherein the threshold is 4 or 8.

[0149] EEE3. The method according to EEE1 or EEE2, wherein the spatial fade-in / fade-out of the decoded FOA output includes linear interpolation between the identity matrix and the spatial target matrix based on the assumed fade-out time.

[0150] EEE4. The method according to any one of EEE1 to EEE3, wherein the spatial fade-in / fade-out has a fade-in / fade-out level based on a time threshold.

[0151] EEE5. A method for processing audio, comprising: identifying correctly decoded parameters; identifying parameter bands that have not been correctly decoded due to a lack of time difference cardinality; and allocating the parameter bands that have not been correctly decoded based at least in part on the correctly decoded parameters.

[0152] EEE6. The method according to EEE5, wherein the allocation of the parameter band that has not yet been correctly decoded is performed using previous frame data.

[0153] EEE7. The method described in EEE5 or EEE6, wherein the allocation of the parameter band that has not yet been correctly decoded is performed using interpolation.

[0154] EEE8. The method according to EEE7, wherein the interpolation includes performing linear interpolation across frequency bands in response to determining that the last correctly decoded value of a particular parameter is older than a threshold.

[0155] EEE9. The method according to EEE7 or EEE8, wherein the interpolation includes interpolation between nearest neighbors.

[0156] EEE10. The method according to any of EEE5 to EEE9, wherein allocating the identified parameter band comprises: determining previous frame data that is considered correct; determining current interpolated data; and determining whether to allocate the identified parameter band using the previous correct frame data or the current interpolated data based on a metric about how new the previous correct frame data is.

[0157] EEE11. A system comprising: one or more processors; and a non-transitory computer-readable medium storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations according to any one of EEE1 to EEE10.

[0158] EEE12. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of EEE1 to EEE10.

Claims

1. A method for processing an audio signal, wherein the audio signal comprises a sequence of frames, each frame containing representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predefined channel format, the method comprising: Receive the audio signal; and Based on the received audio signal, a reconstructed audio signal is generated in the predefined channel format. in Generating the reconstructed audio signal includes: Determine whether at least one frame of the audio signal has been lost; and If the number of consecutively lost frames exceeds a first threshold, then the reconstructed audio signal is faded in and out to a predefined spatial configuration. The process of fading the reconstructed audio signal into the predefined spatial configuration involves linear interpolation between the identity matrix and the target matrix that indicate the predefined spatial configuration, based on the predefined fade-out time.

2. The method according to claim 1, further comprising: If the number of consecutively lost frames exceeds a second threshold that is greater than or equal to the first threshold, then the reconstructed audio signal is gradually faded out.

3. An apparatus for processing audio signals, comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all the steps of the method according to claim 1 or claim 2.

4. A non-transitory computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all the steps of the method according to claim 1 or claim 2.

Images