Packet loss concealment

Abstract

To provide a method for processing audio signals to conceal packet loss.SOLUTION: An audio signal includes a frame sequence, and each frame includes representations of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels to a predetermined channel format. A packet loss concealment (PLC) method includes the steps of receiving an audio signal, and generating a reconstructed audio signal in a predetermined channel format based on the received audio signal. The step of generating the reconstructed audio signal includes the steps of determining whether at least one frame of audio is lost, and fading the reconstructed audio signal to a predetermined spatial configuration, if the number of consecutive lost frames exceeds a first threshold.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] [Related Applications] This application claims priority to the following priority applications: U.S. Provisional Application 63 / 049,323 (Reference Number: D20068USP1) filed on July 8, 2020 and U.S. Provisional Application 63 / 208,896 (Reference Number: D20068USP2) filed on June 9, 2021.

[0002] [Technical Field] This disclosure relates to methods and devices for processing audio signals. This disclosure further describes decoder processing in coders such as immersive voice and audio system (IVAS) coders in the case of packet (frame) loss to achieve the best possible audio experience. This principle is known as Packet Loss Concealment (PLC).

Background Art

[0003] Audio coders for coding spatial audio such as IVAS include metadata that includes reconstruction parameters (e.g., spatial reconstruction parameters) that enable the exact spatial configuration of the encoded audio. Although packet loss concealment may be performed on the actual audio signal, if this metadata is lost, it may result in a recognizable incorrect spatial reconstruction of the audio and thus audible artifacts.

[0004] Therefore, it is necessary to improve packet loss concealment of metadata including reconstruction parameters such as spatial reconstruction parameters.

Summary of the Invention

[0005] From the above perspective, this disclosure provides a method for processing an audio signal, a method for encoding an audio signal, and corresponding equipment, a computer program, and a computer-readable storage medium, each having the features of an independent claim.

[0006] Aspects of this disclosure provide a method for processing an audio signal. The method may be performed in a receiver / decoder. The audio signal may include a frame sequence. Each frame includes a representation of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels into a predetermined (or predefined) channel format. The audio signal may be a multi-channel audio signal. The predefined channel format may be first-order Ambisonics (FOA), such as W, X, Y, and Z audio channels (components). In this case, the audio signal may include up to four audio channels. Multiple audio channels in the audio signal may relate to downmixed channels obtained by downmixing the audio channels of the predefined channel format. The reconstruction parameters may be spatial reconstruction (SPAR) parameters. The method may include the step of receiving an audio signal. The method further includes the step of generating a reconstructed audio signal in a predetermined channel format based on the received audio signal. In this case, the generation of the reconstructed audio signal may be based on the received audio signal and the reconstruction parameters (and / or estimation of the reconstruction parameters). Furthermore, generating a reconstructed audio signal may involve upmixing the audio channels (of multiple) of the audio signal. Upmixing multiple audio channels into a predefined channel format may involve reconstructing the audio channels of a predefined channel format based on the multiple audio channels and their uncorrelated versions. The uncorrelated versions may be generated based on the multiple audio channels of the audio signal and (at least some of) the reconstruction parameters. For this purpose, the upmix matrix may be determined based on the reconstruction parameters. Generating a reconstructed audio signal may also involve determining whether at least one frame of the audio signal is missing.Subsequently, if the number of consecutive loss frames exceeds a first threshold, generation may involve fading the reconstructed audio signal into a predetermined (or predefined) spatial configuration. In one example, the predefined spatial configuration may relate to an omnidirectional audio signal. For a reconstructed FOA audio signal, this means that only the W audio channel is retained. The first threshold is, for example, 4 or 8 frames. The duration of the frames is, for example, 20 ms.

[0007] By configuring it as defined above, the proposed method can mitigate inconsistent audio in the event of packet loss, especially prolonged packet loss, and provide a consistent spatial experience for the user. This may be particularly relevant in Enhanced Voice Service (EVS) frameworks where, in the event of packet loss, the EVS hijacking signals of individual audio channels may not be consistent with one another.

[0008] In some embodiments, a predefined spatial configuration may correspond to a spatially uniform audio signal. For example, in the case of FOA, the reconstructed audio signal faded out to a predefined spatial configuration may contain only the W audio channel. Alternatively, a predefined spatial configuration may correspond to a predefined direction of the reconstructed audio signal. In this case, for example, in the case of FOA, one of the X, Y, and Z components may fade out to a scaled version of W, while the remaining two X, Y, and Z components fade out to 0.

[0009] In some embodiments, fading a reconstructed audio signal to a predefined spatial configuration may involve linear interpolation between an identity matrix and a target matrix representing the predefined spatial configuration, according to a predetermined fade-out time. In this case, an upmix matrix for audio reconstruction may be determined (e.g., generated) based on the matrix product of a prominent upmix matrix and the interpolated matrix. For this purpose, the prominent upmix matrix may be derived based on reconstruction parameters.

[0010] In some embodiments, the method may further include the step of gradually fading out the reconstructed audio signal if the number of consecutive loss frames exceeds a second threshold greater than or equal to a first threshold. Gradually fading out (i.e., muting) the reconstructed audio signal can be achieved by applying a gradually decreasing gain to the reconstructed audio signal, multiple audio channels of the audio signal, or any upmix coefficient used in generating the reconstructed audio signal. The gradual fade-out can be performed according to a (second) predetermined fade-out time (time constant). For example, the reconstructed audio signal may be muted by 3 dB per (loss) frame. The second threshold is, for example, 8 frames.

[0011] This further enhances the ability to provide a consistent user experience, especially in cases of packet loss over very long periods.

[0012] In some implementations, this method may further include generating an estimate of the reconstruction parameters for at least one lost frame based on one or more reconstruction parameters of previous frames if at least one frame of the audio signal is lost. This method may further include using the estimate of the reconstruction parameters for at least one lost frame to generate a reconstructed audio signal for at least one lost frame. This may apply if fewer than a predetermined number of frames (e.g., fewer than a first threshold) are lost. Alternatively, it may apply until the reconstructed audio signal is completely spatially faded out and / or completely faded out (muted).

[0013] In some embodiments, each reconstruction parameter may be explicitly coded once for each given number of frames in a frame sequence, and then (time)-differentially coded between frames for the remaining frames. Furthermore, estimating a given reconstruction parameter for a loss frame may include estimating the given reconstruction parameter for the loss frame based on the most recently determined value of the given reconstruction parameter. Alternatively, the estimation may include estimating the given reconstruction parameter for the loss frame based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. Exceptionally, the estimation may include estimating the given reconstruction parameter for the loss frame based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (for example, for a reconstruction parameter relating to a frequency band with only one adjacent frequency band). Thus, a given reconstruction parameter may be extrapolated over time, interpolated over reconstruction parameters, or, for example, extrapolated from a single adjacent frequency band in the case of a reconstruction parameter relating to the lowest / highest frequency band. Differential coding can follow an (interleaved) differential coding scheme in which each frame includes at least one explicitly coded reconstruction parameter and at least one differentially coded reconstruction parameter referencing a previous frame, and the set of explicitly coded and differentially coded reconstruction parameters differs from frame to frame. The contents of these sets can be repeated after a given frame period. It is understood that the values ​​of the reconstruction parameters can be determined by correctly decoding their values.

[0014] This allows for the provision of reasonable reconstruction parameters (e.g., SPAR parameters) in the event of packet loss, and provides a consistent spatial experience based, for example, on EVS-hidden signals. Furthermore, this allows for the provision of the best reconstruction parameters (e.g., SPAR parameters) after packet loss with time-difference coding applied.

[0015] In some embodiments, the method may further include the step of determining a reliability index for the most recently determined value of a given reconstruction parameter. The method may further include the step of determining a reliability index for the most recently determined value of a given reconstruction parameter, and the step of determining, based on the reliability index, whether to estimate a given reconstruction parameter of a loss frame based on the most recently determined value of the given reconstruction parameter or based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter (exceptionally, a single reconstruction parameter). The reliability index may be determined based on the elapsed time (e.g., in frames) of the most recently determined value of the given reconstruction parameter and / or the elapsed time (e.g., in frames) of the most recently determined values ​​of the reconstruction parameters other than the given reconstruction parameter.

[0016] In some embodiments, the method may further include the step of estimating a given reconstruction parameter for a loss frame based on recently determined values ​​of reconstruction parameters other than the given reconstruction parameter, if the number of frames for which a value of a given reconstruction parameter could not be determined exceeds a third threshold. The method may further include, in other cases, estimating a given reconstruction parameter for a loss frame based on recently determined values ​​of a given reconstruction parameter.

[0017] In some embodiments, each frame may include reconstruction parameters associated with its respective frequency band. A given reconstruction parameter for a loss frame may be estimated based on (one or more) reconstruction parameters associated with frequency bands different from the frequency band to which the given reconstruction parameter is associated.

[0018] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to frequency bands different from the frequency band to which the given reconstruction parameter relates. Exceptionally, for frequency bands at the boundary of the covered frequency range (i.e., the highest or lowest frequency band), a given reconstruction parameter for a loss frame may be estimated by extrapolating from reconstruction parameters relating to the frequency band adjacent to (or closest to) the highest or lowest frequency band.

[0019] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to neighboring frequency bands of the frequency band to which the given reconstruction parameter relates. Alternatively, if the frequency band to which the given reconstruction parameter relates has only one neighboring frequency band, the reconstruction parameter can also be estimated by extrapolation from the reconstruction parameter relating to that neighboring frequency band.

[0020] Another aspect of the present disclosure provides a method for processing an audio signal. The method may be performed, for example, in a receiver / decoder. The audio signal may include a frame sequence. Each frame includes a representation of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels into a predetermined channel format. The method may include the step of receiving the audio signal. The method further includes the step of generating a reconstructed audio signal in a predetermined channel format based on the received audio signal, wherein the generation of the reconstructed audio signal may also include determining whether at least one frame of the audio signal has been lost. If at least one frame of the audio signal has been lost, the generation may further include generating an estimate of the reconstruction parameters for at least one lost frame based on the reconstruction parameters of previous frames. The generation may further include using the estimate of the reconstruction parameters for at least one lost frame to generate a reconstructed audio signal for at least one lost frame.

[0021] In some embodiments, each reconstruction parameter may be explicitly coded once for each given number of frames in a frame sequence, and then (time)-differentially coded between frames for the remaining frames. Next, estimating a given reconstruction parameter for a loss frame may include estimating the given reconstruction parameter for the loss frame based on the most recently determined value of the given reconstruction parameter. Alternatively, the estimation may include estimating the given reconstruction parameter for the loss frame based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. Exceptionally, the estimation may include estimating the given reconstruction parameter for the loss frame based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (for example, for a reconstruction parameter relating to a frequency band with only one adjacent frequency band).

[0022] In some embodiments, the method may further include the step of determining a reliability index for the most recently determined value of a given reconstruction parameter. The method may further include the step of determining a reliability index for the most recently determined value of a given reconstruction parameter, and the step of determining, based on the reliability index, whether to estimate a given reconstruction parameter of a loss frame based on the most recently determined value of the given reconstruction parameter or based on the most recently determined values ​​of two or more other reconstruction parameters (exceptionally, a single reconstruction parameter).

[0023] In some embodiments, the method may further include the step of estimating a given reconstruction parameter for a loss frame based on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter (exceptionally, a single reconstruction parameter) if the number of frames for which the value of a given reconstruction parameter could not be determined exceeds a third threshold. The method may further include, in other cases, estimating a given reconstruction parameter for a loss frame based on the most recently determined value of a given reconstruction parameter.

[0024] In some embodiments, each frame may include reconstruction parameters associated with its respective frequency band. Then, a given reconstruction parameter of a loss frame may be estimated based on (one or more) reconstruction parameters associated with frequency bands different from the frequency band to which the given reconstruction parameter is associated.

[0025] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to frequency bands different from the frequency band to which the given reconstruction parameter relates.

[0026] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to neighboring frequency bands of the frequency band to which the given reconstruction parameter relates. Alternatively, if the frequency band to which the given reconstruction parameter relates has only one neighboring frequency band, the given reconstruction parameter can also be estimated by extrapolation from the reconstruction parameter relating to that neighboring frequency band.

[0027] According to another aspect of the present disclosure, a method for processing an audio signal is provided. The method may be performed, for example, in a receiver / decoder. The audio signal can include a frame sequence. Each frame includes a representation of a plurality of audio channels and reconstruction parameters for upmixing the plurality of audio channels into a predetermined channel format. Each reconstruction parameter is explicitly coded once per given number of frames in the frame sequence and may be differentially coded between the remaining frames. The method may include receiving the audio signal. The method further includes generating a reconstructed audio signal in a predetermined channel format based on the received audio signal. Here, generating the reconstructed audio signal may include identifying correctly decoded reconstruction parameters and reconstruction parameters that cannot be correctly decoded due to a missing differential basis for a given frame of the audio signal. The said generation may further include, for a given frame, estimating reconstruction parameters that could not be correctly decoded based on the correctly decoded reconstruction parameters of the given frame and / or the correctly decoded reconstruction parameters of one or more previous frames. The said generation may further include, for a given frame, generating a reconstructed audio signal for the given frame using the correctly decoded reconstruction parameters and the estimated reconstruction parameters.

[0028] In some embodiments, the step of estimating a given reconstruction parameter that could not be correctly decoded for a given frame may include the step of estimating the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter. Alternatively, the said estimation may include estimating the given reconstruction parameter based on the most recently correctly decoded values of two or more reconstruction parameters other than the given reconstruction parameter. Exceptionally, a given reconstruction parameter of a lost frame can be estimated based on the most recently determined value of one reconstruction parameter other than the given reconstruction parameter (for example, in the case of a reconstruction parameter for a frequency band with only one adjacent frequency band).

[0029] In some embodiments, the method may further include the step of determining an indicator of the reliability of the most recently correctly decoded value of a given reconstruction parameter. The method may further include the step of determining whether to estimate the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter or based on the most recently correctly decoded values of two or more reconstruction parameters other than the given reconstruction parameter (exceptionally, a single reconstruction parameter), based on the indicator of reliability.

[0030] In some embodiments, the method may further include the step of estimating a given reconstruction parameter based on the most recently correctly decoded values of two or more reconstruction parameters other than the given reconstruction parameter (exceptionally, a single reconstruction parameter), if the most recently correctly decoded value of the given reconstruction parameter is older than a predetermined threshold on a frame-by-frame basis. The method may further include, in other cases, the step of estimating the given reconstruction parameter based on the most recently correctly decoded value of the given reconstruction parameter.

[0031] In some embodiments, each frame may contain a reconstruction parameter associated with its respective frequency band. Then, a given reconstruction parameter that could not be correctly decoded may be estimated based on the most recent correctly decoded values ​​of one or more reconstruction parameters associated with a frequency band different from the frequency band to which the given reconstruction parameter relates.

[0032] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to frequency bands different from the frequency band to which the given reconstruction parameter relates.

[0033] In some embodiments, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to neighboring frequency bands of the frequency band to which the given reconstruction parameter relates. Alternatively, if the frequency band to which the given reconstruction parameter relates has only one neighboring frequency band, the given reconstruction parameter can also be estimated by extrapolation from the reconstruction parameter relating to that neighboring frequency band.

[0034] Another aspect of this disclosure provides a method for encoding an audio signal. The method may be performed, for example, in an encoder. The encoded audio signal may include a frame sequence. Each frame includes a representation of multiple audio channels and a reconfiguration parameter for upmixing the multiple audio channels into a predetermined channel format. The method may include a step of explicitly encoding the reconfiguration parameter once for each given number of frames in the frame sequence. The method may further include a step of (time) differential encoding the reconfiguration parameter between frames of the remaining frames. Here, each frame may include at least one reconfiguration parameter that is explicitly encoded and at least one reconfiguration parameter that is differentially encoded with reference to a previous frame. The set of explicitly encoded and differentially encoded reconfiguration parameters may differ from frame to frame. Furthermore, the contents of these sets may be repeated after a predetermined frame period.

[0035] In another embodiment, a computer program is provided. The computer program may include instructions that, when executed by a processor, cause the processor to perform all the steps of the method described throughout this disclosure.

[0036] In another embodiment, a computer-readable storage medium is provided. The computer-readable storage medium can store the aforementioned computer program.

[0037] In another aspect, a device is provided that includes a processor and memory coupled to the processor. The processor can be adapted to perform all the steps of the method described throughout the disclosure. This device may be related to a receiver / decoder (decoder device) or an encoder (encoder device).

[0038] It is understood that the characteristics of the equipment and the steps of the method may be interchangeable in many ways. In particular, the details of the disclosed method can be implemented by the corresponding equipment, as those skilled in the art will understand, and vice versa. Furthermore, it is understood that the above descriptions made with respect to the method (and, for example, those steps) are also applicable to the corresponding apparatus (and, for example, its blocks, stages, units), and vice versa. [Brief explanation of the drawing]

[0039] Embodiments of the disclosure are described below with reference to the attached drawings. [Figure 1] This flowchart shows an example of the flow in the case of packet loss and good frames according to the disclosed embodiment. [Figure 2] This is a block diagram showing an exemplary encoder and decoder according to embodiments of the present disclosure. [Figure 3] This is a flowchart illustrating an example of the processing performed by a PLC according to an embodiment of the present disclosure. [Figure 4] This is a flowchart illustrating an example of the processing performed by a PLC according to an embodiment of the present disclosure. [Figure 5] Figures 1 to 4 show examples of mobile device architectures that implement the features and processing described. [Figure 6] This is a flowchart illustrating an example of an embodiment of the method shown in Figure 3 for processing (e.g., decoding) an audio signal according to an embodiment of the present disclosure. [Figure 7] This is a flowchart illustrating an example of an embodiment of the method shown in Figure 3 for processing (e.g., decoding) an audio signal according to an embodiment of the present disclosure. [Figure 8] This is a flowchart illustrating an example of an embodiment of the method shown in Figure 3 for processing (e.g., decoding) an audio signal according to an embodiment of the present disclosure. [Figure 9] This is a flowchart illustrating an example of an embodiment of the method shown in Figure 3 for processing (e.g., decoding) an audio signal according to an embodiment of the present disclosure. [Figure 10]This flowchart shows an example of a method for encoding an audio signal according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0040] Figures (FIG) and the following description relate to preferred embodiments by illustration only. It should be noted from the following discussion that alternative embodiments of the structures and methods disclosed herein are readily recognizable as viable alternatives that can be adopted without departing from the principles of the claims.

[0041] Several embodiments are described below in detail. Examples of embodiments are shown in the accompanying drawings. Note that, where practicable, similar or similar reference numerals may be used in the drawings to indicate similar or similar functions. The drawings illustrate embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods described herein may be adopted without departing from the principles set forth herein.

[0042] overview Broadly speaking, the technology described herein may include the following: 1. Preservation of reconstruction parameters (e.g., SPAR parameters) during packet loss from the last good frame. 2. Mute and spatial image manipulation after prolonged packet loss to mitigate inconsistent hijacking signals (e.g., EVS hijacking signals). 3. Estimation of reconstruction parameters after packet loss in the case of time-difference coding.

[0043] IVAS system First, as a non-limiting example of a system to which the technology of this disclosure can be applied, we will describe possible implementations of an IVAS system.

[0044] IVAS provides spatial audio experiences for communication and entertainment applications. The underlying spatial audio format is First Order Ambisonics (FOA). For example, four signals (W, Y, Z, X) are coded and can be rendered to any desired output format, such as immersive speaker playback or binaural playback with headphones. Depending on the total bitrate, one, two, three, or four audio signals (downmix channels) are transmitted with low latency via an Enhanced Voice Service (EVS) codec running in parallel. In the decoder, the four FOA signals are reconstructed by processing the downmix channels and their uncorrelated versions using the transmitted parameters. This process is also called upmixing, and the parameters are called Spatial Reconstruction (SPAR) parameters. The IVAS decoding process consists of EVS (core) decoding and SPAR upmixing. The EVS-decoded signal is transformed by a complex-valued low-latency filter bank. SPAR parameters are coded per perceptually motivated frequency bands, with the number of bands typically being 12. The coded downmix channels, excluding the W channel, are the residual signals after (cross-channel) prediction using SPAR parameters. The W channel is transmitted either uncorrected or corrected (active W) to allow for better prediction of the remaining channels. After upmixing the SPAR in the frequency domain, the FOA time-domain signal is generated by filter bank synthesis. Typically, the duration of one audio frame is 20 ms.

[0045] In summary, the IVAS decoding process consists of EVS core decoding of the downmix channel, filter bank analysis, parametric reconstruction (upmix) of the four FOA signals, and filter bank synthesis.

[0046] In particular, at low bitrates such as 32kb / s or 64kb / s, SPAR parameters may be time-diffcoded, for example, depending on previously decoded frames to reduce the SPAR bitrate.

[0047] In general, the techniques (e.g., methods and apparatus) according to embodiments of the present disclosure may be applicable to frame-based (or packet-based) multichannel audio signals, i.e., audio signals (encoded) that constitute a sequence of frames (or packets). Each frame contains a representation of multiple audio channels and reconstruction parameters (e.g., SPAR parameters) for upmixing the multiple audio channels into a predetermined channel format, such as an FOA having W, X, Y, and Z audio channels (components). The multiple audio channels of the (encoded) audio signal may relate to a predefined channel format, for example, downmixed channels obtained by downmixing the W, X, Y, and Z audio channels.

[0048] Limitations of the IVAS system EVS-DTX and SPAR-DTX If no audio activity is detected (VAD) and the background level is low, the EVS encoder may switch to a discontinuous transmission (DTX) mode, which operates at a very low bitrate. Typically, every 8 frames, a small number of DTX parameters (Silence Indicator frame, SID) are sent to control comfort noise generation (CNG) in the decoder. Similarly, dedicated SPAR parameters are sent for the SID frame, which allows for faithful spatial reconstruction of the original spatial environment characteristics. The SID frame is followed by 7 frames with no data (NO_DATA), and the SPAR parameters remain constant until the next SID frame or ACTIVE audio frame is received.

[0049] EVS-PLC When the EVS decoder detects a lost frame, a concealment signal is generated. The generation of the concealment signal may be guided by signal classification parameters transmitted by the encoder in previous good frames without concealment, using various techniques depending on the codec mode (MDCT-based conversion codecs or predictive speech codecs), and other parameters. EVS concealment may generate infinite comfort noise. In IVAS, multiple instances of EVS (one for each downmix channel) run in parallel with different configurations, so EVS concealment may be inconsistent across downmix channels and per content.

[0050] Note that EVS-PLC does not apply to metadata such as SPAR parameters.

[0051] Time difference coding of reconstruction parameters The techniques according to embodiments of this disclosure can be applied to codecs that employ time-difference coding of metadata, including reconstruction parameters (e.g., PSAR parameters). Unless otherwise specified, difference coding in the context of this disclosure means time-difference coding.

[0052] For example, each reconstruction parameter may be explicitly coded (i.e., non-time-differentially) once for every given number of frames in a frame sequence, and then differentially coded between frames for the remaining frames. Here, the differential coding can follow an (interleaved) differential coding scheme in which each frame contains at least one explicitly coded reconstruction parameter and at least one differentially coded reconstruction parameter by reference to a previous frame. The set of explicitly coded and differentially coded reconstruction parameters may differ from frame to frame. The contents of these sets can be repeated after a given frame period. For example, the contents of the aforementioned sets may be given by a group of (interleaved) coding schemes that can cycle sequentially. A non-restrictive example of such a coding scheme applicable in the context of IVAS is shown below.

[0053] For efficient coding of SPAR parameters, time-difference coding can be applied according to the following scheme, for example. [Table 1] SPAR coding scheme where time-differential coded bandwidth is shown as 1 [Table 1] [Table 2] Application order of time-differing SPAR coding schemes [Table 2]

[0054] Here, time-shifted coding always cycles through 4a, 4b, 4c, and 4d, returning to 4a and resuming. Whether or not time-shifted coding is applied may depend on the payload and total bitrate requirements of the basic scheme.

[0055] This coding method ensures that, in contrast to time-difference coding of all bandwidths after packet loss, the parameters of three bandwidths (in the case of a 12-parameter bandwidth configuration, other schemes may be applied to other parameter bandwidth configurations in a similar manner) can always be correctly decoded. Changing the coding scheme as shown in Table 2 ensures that the parameters of all bandwidths can be correctly decoded within four consecutive (lossless) frames. However, depending on the packet loss pattern, some bandwidth parameters may not be correctly decoded beyond four frames.

[0056] Exemplary technology Prerequisites 1. Logic within the decoder to track frame type (e.g., NO_DATA, SID, ACTIVE frames) so that DTX and lost / bad frames can be handled separately. 2. Logic within the decoder to track the number of consecutive lost packets. 3. Logic to track the time-difference coding reconstruction parameters (e.g., SPAR parameters) after packet loss (e.g., without a base for coded differences), bandwidth, and the number of frames since the last base.

[0057] An example of the logic described above is shown in the following pseudocode for decoding a single frame with SPAR parameters covering 12 frequency bands. [List 1] Logic to control the IVAS decoding process while avoiding packet loss

number

number

[0058] Proposed process Generally, methods according to embodiments of the disclosure are applicable to (encode) audio signals that constitute a sequence of frames (packets), where each frame is understood to include a representation of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels into a predetermined channel format. Typically, such a method includes the steps of receiving an audio signal and generating a reconstructed audio signal in a predetermined channel format based on the received audio signal.

[0059] Next, we will describe examples of processing steps in the context of IVAS that can be used to generate reconstructed audio signals. However, it is understood that these processing steps are not limited to IVAS and can be generally applied to PLCs of reconstructed parameters of frame-based (packet-based) audio codecs.

[0060] 1. Mute: If the number of consecutive lost frames exceeds a threshold (the second threshold in the claim range, e.g., 8), the decoded output (e.g., FOA output) is muted (gradually) by, for example, 3dB per (lost) frame. Otherwise, no mute is applied. Muting can be achieved by appropriately modifying the upmix matrix (e.g., the SPAR upmix matrix). Muting increases PLC consistency across the bitrate and content, and lengthens the duration of packet loss. The above logic provides a means to apply mute in the case of CNG with DTX as needed.

[0061] Generally, if the number of consecutive loss frames exceeds a threshold (the second threshold in the claim), the reconstructed audio signal may be gradually faded out (muted). Gradually fading out (muting) the reconstructed audio signal is achieved by applying a gradually decreasing gain to the reconstructed audio signal, by applying a gradually decreasing gain to multiple audio channels of the audio signal, or by applying a gradually decreasing gain to the upmix coefficients used to generate the reconstructed audio signal. Stepped fade-out can be performed according to a predetermined fade-out time (time constant). For example, as described above, the reconstructed audio signal may be muted by 3 dB per (loss) frame. The second threshold is, for example, 8 frames.

[0062] 2. Spatial Fade-Out: When the number of consecutive lost frames exceeds a threshold (the first threshold in the claim, e.g., 4 or 8), the decoded output (e.g., FOA output) is spatially faded out toward a spatial target (i.e., a predefined spatial configuration) within a predefined number of frames. Otherwise, spatial fade-out is not applied. Spatial fading can be achieved by linearly interpolating between an identity matrix (e.g., 4x4) and a spatial target matrix according to the expected fade-out time. For example, a direction-independent spatial image (e.g., muting all channels except W) can reduce spatial discontinuity after packet loss (if not completely muted). That is, in the case of FOA, the predefined spatial configuration may include only the W audio channel. Alternatively, the predefined spatial configuration may be related to a predefined direction. For example, another useful spatial target for FOA is the front image (X=Wsqrt(2), Y=Z=0). In other words, one of the X, Y, and Z components (e.g., X) may fade out to a scaled version of W, and the remaining two X, Y, and Z components (e.g., Y and Z) may fade out to 0. In either case, the generated matrix is ​​applied to the SPAR upmix matrix for all bandwidths. Thus, the (SPAR) upmix matrix for audio reconstruction may be determined (e.g., generated) based on the matrix product of a prominent upmix matrix and an interpolation matrix from which the prominent upmix matrix can be derived from the reconstruction parameters. Spatial fading increases PLC consistency across bitrate and content and lengthens the duration of packet loss. The logic above provides a means to apply spatial fading to CNG with DTX as well, if necessary. FOA is used as a non-limiting example. Other formats, such as channel-based spatial formats including stereo, can be used similarly. It is understood that certain formats can use certain corresponding spatial fade matrices.

[0063] Generally, the generation of a reconstructed audio signal may include fading the reconstructed audio signal into a predefined spatial configuration if the number of consecutive loss frames exceeds a threshold (the first threshold in the claim). Accordingly, this predefined spatial configuration may correspond to a spatially uniform audio signal or a predefined direction (e.g., a predefined direction in which the reconstructed audio signal is rendered). It is understood that the (first) threshold for spatial fading may be less than or equal to the (second) threshold for fading out (muting). Therefore, when the above processing steps are combined, the reconstructed audio signal may first fade out into a predefined spatial configuration and then be muted, or in conjunction with it.

[0064] 3. Parameter Estimation / Recovery from Packet Loss via Time-Difference Coding: The logic above allows us to identify parameter bands that have not yet been correctly decoded since the time-difference base was lost. These parameter bands can be allocated by previous frame data, as in the case of packet loss concealment. As an alternative strategy, linear (or nearest neighbor) interpolation across frequency bands is proposed when the last received base (or generally the last correctly decoded parameter for a particular parameter) is considered too old. In frequency bands at the boundaries of the covered frequency range, this may correspond to extrapolation from each neighboring (or nearest neighbor) frequency band. The proposed approach is beneficial because interpolation on correctly decoded bands is likely to give better parameter estimations than using older previous frame data in combination with new, correctly decoded data.

[0065] In particular, the proposed approach could be used in both the case of PLC for some lost packets (e.g., before or during spatial fade-out and / or muting, until the reconstructed audio signal is spatially completely faded out or completely faded out) and the case of recovery after burst packet loss.

[0066] Generally, if at least one frame of an audio signal is lost, estimations of the reconstruction parameters for at least one lost frame may be generated based on the reconstruction parameters of previous frames. These estimations can then be used to generate a reconstructed audio signal for at least one lost frame.

[0067] For example, a given reconstruction parameter of a loss frame can be extrapolated over time or interpolated / extrapolated over frequency (generally, interpolated / extrapolated between other reconstruction parameters). In the former case, a given reconstruction parameter of a loss frame can be estimated based on recently determined values ​​of a given reconstruction parameter. In the latter case, a given reconstruction parameter of a loss frame can be estimated based on recently determined values ​​of one (in the case of frequency bands at the boundaries of the covered frequency range), two, or more reconstruction parameters other than the given reconstruction parameter.

[0068] Whether to use extrapolation across time or interpolation / extrapolation across other reconstruction parameters can be determined based on a reliability metric for the most recently determined value of a given reconstruction parameter. That is, based on the reliability metric, it can be determined whether to estimate a given reconstruction parameter of a loss frame based on the last determined value of a particular reconstruction parameter, or on the most recently determined values ​​of two or more reconstruction parameters other than the given reconstruction parameter. This reliability metric may be determined based on the elapsed time (e.g., in frames) of the most recently determined value of the given reconstruction parameter and / or the elapsed time (e.g., in frames) of the most recently determined values ​​of the other reconstruction parameters. In one implementation, the method may further estimate a given reconstruction parameter of a loss frame based on the most recently determined values ​​of one or more reconstruction parameters other than the given reconstruction parameter if the number of frames for which the value of the given reconstruction parameter could not be determined exceeds a third threshold. Otherwise, a given reconstruction parameter of a loss frame can be estimated based on the most recently determined value of the given reconstruction parameter.

[0069] As described above, each frame contains reconstruction parameters associated with its respective frequency band, and a given reconstruction parameter of a loss frame may be estimated based on one or more reconstruction parameters associated with frequency bands different from the frequency band to which the given reconstruction parameter is associated. For example, a given reconstruction parameter may be estimated by interpolation (or extrapolation therefrom) between one or more reconstruction parameters relating to frequency bands different from the frequency band to which the given reconstruction parameter is associated. More specifically, in some implementations, a given reconstruction parameter can be estimated by interpolating between reconstruction parameters associated with frequency bands adjacent to the frequency band to which the given reconstruction parameter is associated, or, if there is only one frequency band adjacent to (or closest to) the frequency band to which the given reconstruction parameter is associated (in the case of the highest and lowest frequency bands), by extrapolating from the reconstruction parameters associated with that adjacent (or closest) frequency band.

[0070] It is understood that the processing steps described above can generally be used individually or in combination. That is, a method according to this disclosure may include one, two, or all of the processing steps 1 through 3 described above.

[0071] Summary of Key Points in This Disclosure This disclosure proposes the concept of spatial targets for PLC and spatial fade-out, potentially in relation to muting. This disclosure proposes the concept of having a frame in which concealment and normal decoding coexist during the time-difference coding recovery phase. This includes: - Determination of parameters after packet loss in the case of time-difference coding based on previous good frame data and / or interpolation of currently correctly decoded parameters, and, - Determine whether to use the previous good frame data or / or the current interpolated data, based on a measure of how recent the previous good frame data is.

[0072] Examples of processes and systems Figure 1 is a flowchart illustrating an example of the flow when packet loss (left path) and good frames (right path) occur. The flowchart leading up to the "Generate Upmix Matrix" box is explained in detail in the form of pseudocode in List 1, and is described in item 3 of the "Proposed Process" section above. The "Modify Upmix Matrix" process is described in items 1 and 2 of the "Proposed Process" section above.

[0073] Figure 2 is a block diagram showing an example of an IVAS SPAR encoder and decoder. The IVAS upmix matrix involves the process of combining the decoded downmix channel and uncorrelated version with parameters (C, P1, ..., PD), the inverse remix matrix, and the inverse prediction into a single upmix matrix. The upmix matrix may be modified by PLC processing.

[0074] Figures 3 and 4 are flowcharts illustrating exemplary processes in a PLC.

[0075] Exemplary system architecture Figure 5 shows a mobile device architecture for implementing the features and processes described with reference to Figures 1-4 according to an embodiment. Architecture 800 can be implemented in any electronic device, including, but not limited to, desktop computers, consumer audio / visual (AV) equipment, radio broadcasting equipment, and mobile devices (e.g., smartphones, tablet computers, laptop computers, wearable devices). In the exemplary embodiment shown, Architecture 800 is for a smartphone and includes a processor 801, a peripheral interface 802, an audio subsystem 803, a speaker 804, a microphone 805, sensors 806 (e.g., accelerometer, gyroscope, barometer, magnetometer, camera), a position processor 807 (e.g., GNSS receiver), a wireless communication subsystem 808 (e.g., Wi-Fi, Bluetooth, cellular), and an I / O subsystem 809 including a touch controller 810 and other input controllers 811, a touch surface 812, and other input / controllers 813. Other architectures with more or fewer components can also be used to implement embodiments of the disclosure.

[0076] The memory interface 814 is coupled to the processor 801, the peripheral device interface 802, and the memory 815 (e.g., flash, RAM, ROM). The memory 815 stores computer program instructions and data, including, but not limited to, operating system instructions 816, communication instructions 817, GUI instructions 818, sensor processing instructions 819, telephone instructions 820, electronic messaging instructions 821, web browsing instructions 822, audio processing instructions 823, GNSS / navigation instructions 824, and application / data 825. The audio processing instructions 823 include instructions for performing audio processing as described in this specification with reference to Figures 1-2.

[0077] Audio processing and PLC technology for reconstruction parameters An example of PLC in the context of IVAS was previously described. It is understood that the concepts provided in this context can generally be applied to PLC for the reconstruction parameters of frame-based (packet-based) audio signals. Next, additional examples of how to adopt these concepts are described with reference to Figures 6-10.

[0078] Figure 6 shows an overview of the entire method 600 for processing an audio signal. As previously mentioned, the (encoded) audio signal includes a sequence of frames, each frame containing a representation of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels into a predetermined channel format. Method 600 includes steps S610 and S620, which may further include substeps, and will be described in detail below with reference to Figures 7-9. Furthermore, Method 600 may be performed, for example, in a receiver / decoder.

[0079] In step S610, the (encoded) audio signal is received. The audio signal can be received, for example, as a (packetized) bitstream.

[0080] In step S620, a reconstructed audio signal in a predefined channel format is generated based on the received audio signal. Here, the reconstructed audio signal can be generated based on the received audio signal and reconstruction parameters (and / or estimation of reconstruction parameters as detailed below). Furthermore, generating the reconstructed audio signal may involve upmixing the audio channels of the audio signal into a predefined channel format. Upmixing the audio channels into a predefined channel format may involve reconstructing the audio channels in a predefined channel format based on the audio channels of the audio signal and their uncorrelated versions. The uncorrelated versions may be generated based on the audio channels of the audio signal and (at least some of) the reconstruction parameters.

[0081] Figure 7 shows a method 700 including exemplary (sub)steps S710, S720, and S730 for generating a reconstructed audio signal in step S620. It is understood that steps S720 and S730 relate to possible implementations of step S620, which can be used individually or in combination. That is, step S620 may not include either step S720 or S730 (in addition to step S710), or it may include either or both.

[0082] In step S710, it is determined whether at least one frame of the audio signal has been lost. This can be done according to the description above in the Prerequisites section.

[0083] In that case, if the number of consecutive loss frames exceeds the first threshold in step S720, the reconstructed audio signal is faded out into a predefined spatial configuration. This can be done according to item / step 2 of the section "Proposed Process" above.

[0084] As an addition or alternative, in step S730, if the number of consecutive lost frames exceeds a second threshold greater than or equal to a first threshold, the reconstructed audio signal is gradually faded out (muted). This can be done according to item / step 1 in the section "Proposed Process" above.

[0085] Figure 8 shows Method 800, which includes exemplary (sub)steps S810, S820, and S830 for generating a reconstructed audio signal in step S620. It is understood that steps S810-S830 relate to a possible implementation of step S620, which can be used alone or in combination with the possible implementation shown in Figure 7.

[0086] In step S810, it is determined whether at least one frame of the audio signal has been lost. This can be done according to the above description in the Prerequisites section.

[0087] Next, in step S820, if at least one frame of the audio signal is lost, an estimate of the reconstruction parameters for at least one lost frame is generated based on one or more reconstruction parameters of the previous frames. This can be done according to item / step 3 of the section "Proposed Process" above.

[0088] In step S830, estimations of the reconstruction parameters for at least one lost frame are used to generate a reconstructed audio signal for at least one lost frame. This can be done, for example, via upmixing, as described in step S620. It is understood that if the actual audio channels are also lost, their estimations may be used instead. The EVS hijacked signal is an example of such an estimation.

[0089] Method 800 can be applied as long as fewer than a predetermined number of frames (e.g., fewer than the first or second threshold) are lost. Alternatively, Method 800 may be applied until the reconstructed audio signal is completely spatially faded out and / or completely faded out. Therefore, in the case of persistent packet loss, Method 800 can be used to mitigate packet loss before or until mute / spatial fading is enabled. However, it should be noted that the concept of Method 800 can also be used to recover from burst packet loss when time-difference coding of reconstruction parameters is present.

[0090] Here, an example of such a method for processing an audio signal for recovery from burst packet loss, such as one performed in a receiver / decoder, is described with reference to Figure 9. As previously stated, the audio signal is assumed to consist of a sequence of frames, each frame containing a representation of multiple audio channels and a reconstruction parameter for upmixing the multiple audio channels into a predetermined channel format. Furthermore, each reconstruction parameter is assumed to be explicitly coded once for every given number of frames in the frame sequence and then differentially coded between the remaining frames. This can be done according to the section "Time-Differential Coding of Reconstruction Parameters" above. Similar to method 600, a method for processing an audio signal for recovery from burst packet loss includes the steps of receiving an audio signal (similar to step S610) and generating a reconstructed audio signal in a predefined channel format based on the received audio signal (similar to step S620). Method 900 shown in Figure 9 includes steps S910, S920, and S930, which are substeps for generating a reconstructed audio signal in a predefined channel format based on a received audio signal of a given frame. It is understood that recovery methods from burst packet loss can be applied to correctly received frames (e.g., the first few frames) following a large number of lost frames.

[0091] In step S910, the reconstructed parameters that were correctly decoded and those that could not be correctly decoded due to the missing differential base are identified. If a large number of frames (packets) have been lost in the past, it is expected that the time differential base will be missing.

[0092] In step S920, the reconstruction parameters that could not be correctly decoded are estimated based on the correctly decoded reconstruction parameters of a given frame and / or the correctly decoded reconstruction parameters of one or more previous frames. This can be done according to item 3 of the section "Proposed Process" above.

[0093] For example, the step of estimating a given reconstruction parameter that cannot be correctly decoded for a given frame (due to time-difference-based loss) includes the step of estimating the given reconstruction parameter based on the most recent correctly decoded value of the given reconstruction parameter (e.g., the last correctly decoded value before (burst) packet loss), or the step of estimating the given reconstruction parameter based on the most recent correctly decoded value of one or more reconstruction parameters other than the given reconstruction parameter. In particular, the most recent correctly decoded value of one or more reconstruction parameters other than the given reconstruction parameter may have been decoded for / from the (current) given frame. Which of the two approaches to follow can be determined based on an indicator of the reliability of the most recent correctly decoded value of the given reconstruction parameter. This indicator may be, for example, the elapsed time since the most recent correctly decoded value of the given reconstruction parameter. For example, if the most recent correctly decoded value of the given reconstruction parameter is older than a given threshold (e.g., on a frame-by-frame basis), the given reconstruction parameter may be estimated based on the most recent correctly decoded value of one or more reconstruction parameters other than the given reconstruction parameter. In other cases, a given reconstruction parameter can be estimated based on recent correctly decoded values ​​of that given reconstruction parameter. However, it is understood that other metrics of reliability are also feasible.

[0094] Depending on the applicable codec (e.g., IVAS), each frame may contain reconstruction parameters associated with each of several frequency bands. Then, a given reconstruction parameter that could not be correctly decoded may be estimated based on the most recent correctly decoded values ​​of one or more reconstruction parameters associated with frequency bands different from the frequency band to which the given reconstruction parameter relates. For example, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to frequency bands different from the frequency band to which the given reconstruction parameter relates. In some cases, a given reconstruction parameter may be extrapolated from a single reconstruction parameter relating to a frequency band different from the frequency band to which the given reconstruction parameter relates. Specifically, a given reconstruction parameter may be estimated by interpolation between reconstruction parameters relating to neighboring frequency bands of the frequency band to which the given reconstruction parameter relates. If the frequency band to which a given reconstruction parameter relates has only one neighboring (or nearest) frequency band (for example, the highest and lowest frequency bands), the given reconstruction parameter can also be estimated by extrapolating from the reconstruction parameters relating to that neighboring (or nearest) frequency band.

[0095] In step S930, the reconstructed audio signal for a given frame is generated using the correctly decoded reconstruction parameters and the estimated reconstruction parameters. This can be done, for example, via upmixing, as described in step S620.

[0096] The scheme for time-difference coding of reconstruction parameters was described above in the section “Time-Difference Coding of Reconstruction Parameters”. It is understood that this disclosure also relates to methods for encoding audio signals to which such time-difference coding is applied. An example of such a method 1000 for encoding an audio signal is schematically shown in Figure 10. Assume that the encoded audio signal includes a sequence of frames, each frame containing a representation of multiple audio channels and reconstruction parameters for upmixing the multiple audio channels into a predetermined channel format. Thus, method 1000 generates an encoded audio signal that can be decoded by, for example, one of the methods described above. Method 1000 includes steps S1010 and S1020 that can be performed for each reconstruction parameter to be coded (e.g., SPAR parameters).

[0097] In step S1010, the reconstruction parameters are explicitly encoded (e.g., non-differentially encoded, or explicitly encoded) once for each given number of frames in the frame sequence.

[0098] In step S1020, the reconstruction parameters are encoded (time-)differentially between the remaining frames.

[0099] For a given frame, the choice of whether to differentially or non-differentially encode each reconstruction parameter can be made such that each frame contains at least one explicitly encoded reconstruction parameter and at least one time-differentially encoded reconstruction parameter with reference to a previous frame. Furthermore, to ensure resilience in case of packet loss, the set of explicitly encoded and differentially encoded reconstruction parameters differs from frame to frame. For example, the set of explicitly encoded and differentially encoded reconstruction parameters is selected according to a group of schemes in which the scheme cycles periodically. That is, the contents of the aforementioned set of reconstruction parameters may be repeated after a given frame period. It is understood that each reconstruction parameter is explicitly encoded once for a given number of frames. It is desirable that this given number of frames is the same for all reconstruction parameters.

[0100] advantage As partially outlined in the section above, the technology described in this disclosure can provide the following technical advantages to PLCs compared to conventional technologies. 1. Provide reasonable reconstruction parameters (e.g., SPAR parameters) in the event of packet loss, and provide a consistent spatial experience based on, for example, EVS-hidden signals. 2. Mitigate inconsistencies in lost audio data over long periods of time in lost packets (e.g., EVS concealment). 3. Provides the best reconstruction parameters (e.g., SPAR parameters) after packet loss with time-difference coding applied.

[0101] interpretation The configurations of the systems described herein may be implemented in an audio processing network environment based on a suitable computer for processing digital or digitized audio files. The adaptive audio system portion may include one or more networks containing any desired number of individual machines, including one or more routers (not shown) that function to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0102] One or more components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a computing device based on the system's processor. It should also be noted that the various functions disclosed herein may be described in terms of their operation as data and / or instructions embodied in hardware, firmware, and / or various machine-readable or computer-readable media, using any number of combinations of register transfers, logical components, and / or other characteristics. Computer-readable media on which such formatted data and / or instructions are embodied include, but are not limited to, various forms of physical (non-temporary) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0103] While one or more implementations have been described as examples in terms of specific embodiments, it should be understood that one or more implementations are not limited to the disclosed embodiments. Rather, the implementations are intended to cover various modifications and similar configurations, as will be apparent to those skilled in the art. Accordingly, the appended claims should be interpreted most broadly to encompass all such modifications and similar configurations.

[0104] <Examples of listed embodiments> Various aspects and implementations of this disclosure may also be apparent from the enumerated example embodiments (EEE) listed below, which are not claimed. (EEE1) A method for processing audio, A step of determining whether the number of consecutive loss frames meets a threshold, In response to determining that the number satisfies the threshold, the steps include spatially fading the decoded first-order ambisonics (FOA) output, A method that includes this. (EEE2) The method of EEE1, wherein the value is 4 or 8. (EEE3) The step of spatially fading the decoded FOA output includes the method of EEE1 or EEE2, wherein the step includes linear interpolation between the identity matrix and the spatial target matrix according to the assumed fade-out time. (EEE4) The method according to any one of EEE1 to EEE3, wherein the spatial fading has a fade level based on a time threshold. A method for processing (EEE5) audio, A step to identify parameters that have not yet been correctly decoded due to time-difference-based loss, A step of allocating a parameter band that has not yet been correctly decoded based on at least some of the correctly decoded parameters, A method that includes this. (EEE6) The method of EEE5, wherein the step of allocating parameter bandwidths that have not yet been correctly decoded is performed using previous frame data. (EEE7) The method of EEE5 or EEE6, wherein the step of allocating parameter bands that have not yet been correctly decoded is performed using interpolation. (EEE8) The method of EEE7, wherein linear interpolation across a frequency band is included in the interpolation in response to determining that the last correctly decoded value of a particular parameter is older than a threshold. (EEE9) A method of EEE7 or EEE8 in which the interpolation includes interpolation between the nearest neighbors. (EEE10) The step of allocating the identified parameter band is: A step to determine previous frame data that is considered good, The current steps for determining interpolated data, The steps include determining whether to allocate the identified parameter bandwidth using the previous good frame data or the current interpolated data, based on a metric relating to the relevance of the previous good frame data, The method described in any one of the items EEE5 to EEE9, including the method described in item EEE5 to EEE9. (EEE11) system, One or more processors, A non-temporary computer-readable storage medium for storing instructions, wherein, when the instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the operations described in any one of the EEE1 to 10 items, A system that includes this. A non-temporary computer-readable medium for storing (EEE12) instructions, wherein, when the instructions are executed by one or more processors, the instructions cause the one or more processors to perform the operations described in any one of EEE1 to 10.

Claims

1. A method for processing an audio signal, wherein the audio signal comprises a frame sequence, each frame comprising a representation of a plurality of audio channels and a reconstruction parameter for upmixing the plurality of audio channels into a predefined channel format, each reconstruction parameter being explicitly coded as a difference-based once for each given number of frames in the frame sequence, and then difference-coded between the remaining frames, the method is The steps include receiving the aforementioned audio signal, The steps include generating a reconstructed audio signal in the predefined channel format based on the received audio signal, Includes, The step of generating the reconstructed audio signal involves, for a given frame of the audio signal, The steps include identifying correctly decoded reconstruction parameters and reconstruction parameters that cannot be correctly decoded due to the absence of the difference base, A step of estimating a reconstruction parameter that cannot be correctly decoded based on the correctly decoded reconstruction parameter of a given frame and / or the correctly decoded reconstruction parameters of one or more previous frames, wherein each frame includes a reconstruction parameter associated with its respective frequency band, and if the most recent correctly decoded value of the undecoded reconstruction parameter is older than a predetermined threshold on a frame-by-frame basis, the undecoded reconstruction parameter is estimated based on the most recent correctly decoded value of one or more reconstruction parameters associated with a frequency band different from the frequency band to which the undecoded reconstruction parameter relates. The steps include using the correctly decoded reconstruction parameters and the estimated reconstruction parameters to generate a reconstructed audio signal of the given frame, Methods that include...

2. The step of estimating reconstruction parameters that cannot be correctly decoded for the given frame is performed if the most recent correctly decoded value is not older than the predetermined threshold on a frame-by-frame basis. A step of estimating the reconstruction parameters that cannot be correctly decoded based on the latest correctly decoded values, or A step of estimating the reconstruction parameter that cannot be correctly decoded based on the latest correctly decoded values ​​of one or more reconstruction parameters other than the reconstruction parameter that cannot be correctly decoded. The method according to claim 1, including the method described in claim 1.

3. A device comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 2.

4. A computer program that causes a computing device to execute the method described in any one of claims 1 to 2.

Images