Method and apparatus for controlling multi-channel audio frame loss concealment
By mixing error hidden frames under generation and processing in multi-channel audio signals, combined with energy adjustment of residual signal spectrum, the naturalness and correlation problems of missing or damaged frame hiding in multi-channel audio codec are solved, and the problem of missing or damaged frame hiding in multi-channel audio codec is realized. Stable and natural output of the signal.
Patent Information
- Application Number
- CN202411283537.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-20
- Filing Date
- 2019-05-16
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, in multi-channel audio encoding and decoding, lost or damaged data frame hiding methods are difficult to effectively control the spatial characteristics of signals, resulting in unnatural or too wide output, especially in the error hiding of residual signal components. question.
By generating downmix error hidden frames and transforming them into the frequency domain to decorrelate them, combining the residual signal spectrum to generate energy-adjusted decorrelation residual signal hidden frames, providing transformed frames and parameters to the multi-channel audio synthesis component. , to perform inverse frequency domain transformation to generate replacement frames of lost or damaged frames, phase adjustment of phase ECU avoids discontinuity of periodic signal components.
Effectively maintain the energy level of the residual signal, avoid periodic discontinuity, ensure the naturalness and correlation of the output signal, and adapt to the loss of different signal types.
Smart Images

Figure CN119068883A_ABST
Abstract
Description
[0001] This application is a divisional application of the Chinese patent application “Method and device for controlling multi-channel audio frame loss concealment” with application number 201980084864.7 (filing date May 16, 2019). Technical Field
[0002] The present application relates to methods and apparatus for controlling packet loss concealment for stereo or multi-channel audio encoding and decoding. Background Art
[0003] Although the capacity of telecommunication networks is constantly increasing, there is still great interest in limiting the required bandwidth per communication channel. In mobile networks, the less transmission bandwidth is used for each call, the lower the power consumption of mobile devices and base stations. This translates into energy and cost savings for mobile operators, while end users will experience longer battery life and longer talk time. Furthermore, mobile networks can serve a larger number of users in parallel, as each user consumes less bandwidth.
[0004] Most listeners are accustomed to high-quality immersive audio through modern music playback systems and cinemas. In mobile telecommunication services, constraints on radio resources and processing delays have kept the quality low and most voice services still deliver only mono sound. Recently, stereo and multi-channel sound for communication services has gained momentum in the context of virtual / mixed / augmented reality where immersive sound reproduction beyond mono is required. Rendering high-quality spatial sound within the bandwidth constraints of telecommunication networks remains a challenge. Additionally, sound reproduction also needs to cope with varying channel conditions where occasional data packets may be lost, for example due to network congestion or poor cell coverage.
[0005] In a typical stereo recording, the channel pairs exhibit a high degree of similarity or correlation. Some embodiments of stereo coding schemes [1] can exploit this correlation by employing parametric coding, where the individual channels are encoded with high quality and supplemented with parametric descriptions that allow reconstruction of the full stereo image. The process of reducing a channel pair to a single channel is often referred to as downmixing, and the resulting channels are often referred to as downmix channels. The downmixing process often attempts to maintain energy by aligning the inter-channel time difference (ITD) and inter-channel phase difference (IPD) before mixing the channels. In order to maintain the energy balance of the input signal, the inter-channel level difference (ILD) may also be measured. The ITD, IPD and ILD are then encoded and can be used in the reverse upmixing process when reconstructing the stereo channel pair at the decoder. The ITD, IPD and ILD parameters describe the correlated components of the channel pair, while the stereo channel pair may also include uncorrelated components that cannot be reconstructed from the downmix. This uncorrelated component can be represented by the inter-channel coherence parameter (ICC). The non-correlated components can be synthesized at the stereo decoder by passing the decoded downmix channels through a decorrelation filter, which outputs a signal with low correlation with the decoded downmix. The strength of the decorrelated components can be controlled with ICC parameters.
[0006] Although parametric stereo reproduction gives good quality at low bitrates, the quality tends to saturate as the bitrate increases due to limitations of the parametric model. To overcome this problem, the non-correlated components can be encoded. This encoding is achieved by simulating stereo reconstruction in the encoder and subtracting the reconstructed signal from the input channels, resulting in a residual signal. If the downmix transform is reversible, the residual signal can be represented by only a single channel for the case of stereo channels. Typically, the residual signal encoding targets lower frequencies that are more psychoacoustically correlated, while higher frequencies can be synthesized using a decorrelator method. Figure 2 is a block diagram depicting an embodiment of a general arrangement for a parametric stereo codec including a residual encoder. Figure 2 In the embodiment of the present invention, the encoder receives an input signal, performs the above-mentioned processing in the stereo processing and downmixing block 210, encodes the mono output via the mono encoder 220, encodes the residual signal via the residual encoder 230, and encodes the ITD, IPD, ILD and ICC parameters. The decoder receives the encoded mono output, the encoded residual signal and the encoded parameters. The decoder decodes the residual signal via the residual decoder 250 and decodes the mono signal via the mono decoder 260. The parameter synthesis block 270 receives the decoded mono signal and the decoded residual signal, and outputs the stereo channels CH1 and CH2 based on the parameters.
[0007] Similar principles apply to multi-channel audio such as 5.1 and 7.1.4 and spatial audio representations such as Ambisonics or spatial audio object coding. The number of channels can be reduced by exploiting the correlation between channels and bundling the reduced set of channels with metadata or parameters for channel reconstruction or spatial audio rendering at the decoder.
[0008] In order to overcome the problem of transmission errors and lost data packets, telecommunication services use packet loss concealment (PLC) techniques. In the event that data packets are lost or damaged due to poor connectivity, network congestion, etc., the loss information of the lost or damaged data packets at the receiver side can be replaced by a synthetic signal by the decoder to conceal the lost or damaged data packets. Some embodiments of the PLC technique are usually closely related to the decoder, where internal states can be used to generate signal continuations or extrapolations to mask packet losses. For multi-mode codecs with several operating modes for different signal types, there are usually several PLC techniques that can be implemented to handle the concealment of lost or damaged data packets.
[0009] For speech coding modes based on linear prediction (LP), a technique that can be used is based on adjusting the glottal pulse position using the estimated frame end pitch information and a copy of the pitch period of the previous frame [2]. Depending on the number of consecutive lost frames and the stability of the last good frame, the gain of the long-term predictor (LTP) converges to zero with speed [2]. Coding modes based on frequency domain (FD) are generally designed to process general or complex signals, such as music. For such signals, different techniques can be used depending on the characteristics of the last received frame. Such analysis can include the number of detected tonal components and the periodicity of the signal. If the frame loss occurs during a highly periodic signal (such as active speech or a single instrumental), a time domain PLC similar to the LP-based PLC may be suitable for implementation. In this case, the FD PLC can simulate the LP decoder by estimating the LP parameters and the excitation signal based on the last received frame [2]. If the lost frame occurs during a non-periodic or noise-like signal, the last received frame can be repeated in the spectral domain, where the coefficients are multiplied with a random symbol signal to reduce the metallic sound of the repeated signal. For stationary tonal signals, it has been found in some embodiments to be advantageous to use a method based on prediction and extrapolation of detected tonal components. More details on the above techniques can be found in [2].
[0010] One hiding method that operates in the frequency domain is the Phase ECU [3]. It can be implemented as a standalone tool that operates on a buffer of previously decoded and reconstructed time signals. Its framework is based on the sinusoidal analysis and synthesis paradigm. In this technique, the sinusoidal components of the last good frame are extracted and phase shifted. When a frame is lost, the sinusoidal frequencies are obtained from the past decoded synthesis in the DFT domain. First, the corresponding frequency bin is identified by finding the peak of the amplitude spectrum plane. Then, the peak frequency bin is used to estimate the fractional frequency of the peak. The peak frequency bin and the corresponding fractional frequency can be stored for creating a replacement for the lost frame. The frequency bin corresponding to the peak along with the neighboring values is phase shifted using the fractional frequency. For the remaining frequency bins of the frame, the amplitude of the past synthesis is retained, while the phase can be randomized. Burst errors can also be handled so that the estimated signal can be smoothly muted by making it converge to zero. More details of the Phase ECU can be found in [3].
[0011] There are many different terms used for packet loss concealment techniques, including frame error concealment (FEC), frame loss concealment (FLC), and error concealment unit (ECU).
[0012] The PLC techniques described above are techniques designed for mono audio codecs. For stereo or multi-channel decoders, one solution for error concealment may be to apply any of the PLC techniques described above on each channel. However, this solution does not provide any control over the spatial characteristics of the signal. Using this solution will most likely create non-correlated signals, which can give a stereo or multi-channel output that sounds unnatural or too wide. Figure 2 For the stereo case depicted in , this translates into using mono PLC for the downmix signal and for the residual signal components separately.
[0013] Error concealment of residual signal components may be particularly sensitive, since the residual components may be added to a spatially unmasked side signal. Discontinuities cause large changes in the characteristics of the side signal and are therefore easily detected and found to be disturbing when heard. Summary of the invention
[0014] According to some embodiments of the present invention, a method for approximating a lost or damaged multi-channel audio frame of a received multi-channel audio signal in a decoding device is provided. The method includes: generating a downmix error concealment frame, and transforming the downmix error concealment frame into the frequency domain to generate a transformed downmix error concealment frame. The method also includes decorrelating the transformed downmix concealment frame to generate a decorrelated concealment frame. The method also includes obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal. The method also includes: using the residual signal spectrum to generate an energy-adjusted decorrelated residual signal concealment frame, and providing the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame. The method also includes performing an inverse frequency domain transform of the synthesized multi-channel audio frame to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0015] A potential advantage of combining the phase evolution error concealment method for the peaks of the spectrum with the noise spectrum from the error concealed downmix signal after passing through the decorrelator is that the operation avoids discontinuities in the periodic signal components by phasing the peaks. In addition, the noise spectrum maintains a desired relationship with the downmix signal, such as a desired correlation level. Another potential advantage is that the operation maintains the energy level of the residual signal at a stable level during frame loss.
[0016] According to other embodiments of the present invention, a device is configured to approximate a lost or damaged multi-channel audio frame of a received multi-channel audio signal. The device includes at least one processor and a memory communicatively coupled to the processor, the memory including instructions executable by the processor, which cause the processor to perform operations. The operations include generating a downmix error concealment frame and transforming the downmix error concealment frame into a frequency domain to generate a transformed downmix error concealment frame. The operations also include decorrelating the transformed downmix concealment frame to generate a decorrelated concealment frame. The operations also include obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal. The operations also include generating an energy-adjusted decorrelated residual signal concealment frame using the residual signal spectrum, and providing the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame. The operations also include performing an inverse frequency domain transform of the synthesized multi-channel audio frame to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0017] According to other embodiments of the present invention, a decoder is configured to perform operations. The operations include: generating a downmix error concealment frame, and transforming the downmix error concealment frame into the frequency domain to generate a transformed downmix error concealment frame. The operation also includes decorrelating the transformed downmix concealment frame to generate a decorrelated concealment frame. The operation also includes obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal. The operation also includes using the residual signal spectrum to generate an energy-adjusted decorrelated residual signal concealment frame, and providing the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame. The operation also includes performing an inverse frequency domain transform of the synthesized multi-channel audio frame to generate a replacement frame for a lost or damaged multi-channel audio frame.
[0018] According to other embodiments of the present invention, a computer program product includes a non-transitory computer-readable medium storing computer program code, which, when executed by at least one processor, causes the at least one processor to: generate a downmix error concealment frame; transform the downmix error concealment frame into the frequency domain to generate a transformed downmix error concealment frame; decorrelate the transformed downmix error concealment frame to generate a decorrelated concealment frame; obtain a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal; use the residual signal spectrum to generate an energy-adjusted decorrelated residual signal concealment frame; provide the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and perform an inverse frequency domain transform of the synthesized multi-channel audio frame to generate a replacement frame for a lost or damaged multi-channel audio frame.
[0019] According to some other embodiments of the present invention, a method for approximating a lost or damaged multi-channel audio frame of a received multi-channel audio signal in a decoding device including a processor is provided, the method comprising the following operations performed by the processor. The operations include: generating a downmix error concealment frame, and transforming the downmix error concealment frame into a frequency domain to generate a transformed downmix error concealment frame. The operations also include decorrelating the transformed downmix concealment frame to generate a decorrelated concealment frame. The operations also include obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal. The operations also include generating an energy-adjusted decorrelated residual signal concealment frame using the residual signal spectrum. The operations also include obtaining a multi-channel audio replacement parameter set. The operations also include performing an inverse frequency domain transform of the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and the multi-channel audio parameters from the previously received multi-channel audio signal frame to generate a transformed downmix error concealment time domain frame, an energy-adjusted decorrelated residual concealment time domain frame, and multi-channel audio time domain parameters. The operations also include providing the transformed downmix error concealment time domain frame, the energy adjusted decorrelated residual concealment time domain frame, and the multi-channel audio time domain parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame.
[0020] According to some other embodiments of the present invention, a computer program product includes a non-transitory computer-readable medium storing computer program code, which, when executed by at least one processor, causes the at least one processor to: generate a downmix error concealment frame; transform the downmix error concealment frame into the frequency domain to generate a transformed downmix error concealment frame; decorrelate the transformed downmix error concealment frame to generate a decorrelated concealment frame; obtain a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame; use the residual signal spectrum to generate an energy-adjusted decorrelated residual signal concealment frame; obtain a multi-channel audio time domain replacement parameter set; perform an inverse frequency domain transform of the transformed downmix error concealment frame and the energy-adjusted decorrelated residual concealment frame to generate a transformed downmix error concealment time domain frame and an energy-adjusted decorrelated residual concealment time domain frame; and provide the transformed downmix error concealment time domain frame, the energy-adjusted decorrelated residual concealment time domain frame, and the multi-channel audio time domain replacement parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame.
[0021] According to some other embodiments of the present inventive concept, a device is provided that is configured to approximate a lost or damaged multi-channel audio frame of a received multi-channel audio signal. The device includes at least one processor and a memory communicatively coupled to the processor, the memory including instructions executable by the processor, which cause the processor to perform operations. The operations include: generating a downmix error concealment frame, and transforming the downmix error concealment frame into a frequency domain to generate a transformed downmix error concealment frame. The operations also include decorrelating the transformed downmix concealment frame to generate a decorrelated concealment frame. The operations also include obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal. The operations also include generating an energy-adjusted decorrelated residual signal concealment frame using the residual signal spectrum. The operations also include obtaining a set of multi-channel audio replacement parameters. The operations also include performing an inverse frequency domain transform of the transformed downmix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and the multi-channel audio parameters from the previously received multi-channel audio signal frame to generate a transformed downmix error concealment time domain frame, an energy-adjusted decorrelated residual concealment time domain frame, and multi-channel audio time domain parameters. The operations also include providing the transformed downmix error concealment time domain frame, the energy adjusted decorrelated residual concealment time domain frame, and the multi-channel audio time domain parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application, and illustrate certain non-limiting embodiments of the inventive concept. In the drawings:
[0023] Figure 1 is a block diagram illustrating an example of an environment for a loss concealment system according to some embodiments;
[0024] Figure 2 is a block diagram illustrating components of a parametric stereo codec according to some embodiments;
[0025] Figure 3 is a diagram showing a combined sinusoidal component and noise spectrum according to some embodiments;
[0026] Figure 4 is a block diagram illustrating a stereo parametric encoder according to some embodiments;
[0027] Figure 5 is a block diagram illustrating a stereo parametric decoder according to some embodiments;
[0028] Figure 6 is a block diagram illustrating the operation of generating a residual signal according to some embodiments of the inventive concept;
[0029] Figure 7is a block diagram illustrating operations of generating a replacement multi-channel audio frame according to some embodiments of the inventive concept;
[0030] Figure 8 is a flowchart illustrating the operation of a decoder according to some embodiments of the inventive concept;
[0031] Fig. 9 is a flowchart illustrating the operation of a decoder generating a residual signal according to some embodiments of the inventive concept;
[0032] Fig. 10A and Fig. 10B is a diagram of a generated spectrum of a generated residual signal according to some embodiments of the inventive concept;
[0033] Fig.11 is a block diagram illustrating a decoder according to some embodiments of the inventive concept;
[0034] Figure 12-Figure 18 is a flowchart illustrating the operation of a decoder according to some embodiments of the inventive concept;
[0035] Fig.19 is a block diagram illustrating approximate phase adjustment according to some embodiments of the inventive concept. DETAILED DESCRIPTION
[0036] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. However, the inventive concept can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be assumed to exist / use in another embodiment by default.
[0037] The following description presents various embodiments of the disclosed subject matter. These embodiments are presented as teaching examples and should not be interpreted as limiting the scope of the disclosed subject matter. For example, certain details of the described embodiments may be modified, omitted or expanded without departing from the scope of the described subject matter.
[0038] Figure 1An example of an operating environment of a decoder 100 is shown, and the decoder 100 can be used to decode a multi-channel bitstream as described herein. The decoder 100 can be a part of a media player, a mobile device, a set-top device, a desktop computer, etc. The decoder 100 receives a coded bitstream. The bitstream can be sent from an encoder, from a storage device 104, from a cloud device via a network 102, etc. During operation, the decoder 100 receives and processes a frame of a bitstream as described herein. The decoder 100 outputs a multi-channel audio signal and sends the multi-channel audio signal to a multi-channel audio player 106, which has at least one speaker for playing the multi-channel audio signal. The storage device 104 can be a part of a repository of a multi-channel audio signal such as a store or a storage repository of a streaming music service, a separate storage component, a component of a mobile device, etc. The multi-channel audio player can be a Bluetooth speaker, a device with at least one speaker, a mobile device, a streaming music service, etc.
[0039] Fig.11 1 is a block diagram illustrating elements of a decoder 100 according to some embodiments of the present invention, the decoder 100 being configured to decode multi-channel audio frames and provide concealment for lost or damaged frames. As shown, the decoder 100 may include a network interface circuit 1105 (also referred to as a network interface), which is configured to provide communication with other devices / entities / functions / etc. The decoder 100 may also include a processor circuit 1101 (also referred to as a processor) coupled to the network interface circuit 1105 and a memory circuit 1103 (also referred to as a memory) coupled to the processor circuit. The memory circuit 1103 may include a computer-readable program code that, when executed by the processor circuit 1101, causes the processor circuit to perform operations according to the embodiments disclosed herein.
[0040] According to other embodiments, the processor circuit 1101 can be defined to include a memory so that a separate memory circuit is not required. As discussed herein, the operation of the decoder 100 can be performed by the processor 1101 and / or the network interface 1105. For example, the processor 1101 can control the network interface 1105 to send communications to the multi-channel audio player 106 and / or receive communications from one or more other network nodes / entities / servers (such as encoder nodes, repository servers, etc.) through the network interface 1105. In addition, modules can be stored in the memory 1103, and these modules can provide instructions so that when the instructions of the modules are executed by the processor 1101, the processor 1101 performs the corresponding operations.
[0041] In one embodiment, the Figure 2For more details, please refer to Figure 4 The encoder is described below. In the following description, embodiments will be described using two channels. The embodiments can be used with more than two channels. A multi-channel encoder processes the input left and right channels in segments called frames (in Figure 2 In CH1 and CH2, Figure 4 For a given frame m, the two input channels can be written as
[0042]
[0043] Wherein, l represents the left channel, r represents the right channel, n=0,1,2,…,N represents the number of samples in frame m, and N is the length of the frame. In an embodiment, frames can be extracted in an overlapping manner in the encoder so that the decoder can use an overlap-add strategy to reconstruct the multi-channel audio signal. The input channels are windowed with a suitable window function w(n) and transformed to the discrete Fourier transform (DFT) domain. Note that other frequency domain representations can be used here, such as a quadrature mirror filter (QMF) filter bank, a hybrid QMF filter bank, or an odd DFT (ODFT) representation consisting of MDCT and MDST transform components.
[0044]
[0045] Then, in parameter analysis box 410, analyze signal to extract ITD, IPD and ILD parameters. In addition, channel coherence can be analyzed, and ICC parameters can be derived. The multi-channel audio parameter set for frame m can be represented as P (m), which includes the complete set of ITD, IPD, ILD and ICC parameters used in parameter representation. These parameters are encoded by parameter encoder 430, and are added to bit stream to be stored and / or sent to decoder.
[0046] Before generating the downmix channels, in one embodiment, it can be beneficial to compensate for ITD and IPD to reduce cancellation and maximize the energy of the downmix. ITD compensation can be implemented in the time domain or in the frequency domain before frequency transformation, but in essence it is to perform a time shift on one or both channels to eliminate ITD. Phase alignment can be implemented in different ways, but the goal is to align the phases so that cancellation is minimized. This ensures maximum energy in the downmix. ITD and IPD adjustments can be made in frequency bands, or over the full spectrum, and preferably, should be made using quantized ITD and IPD parameters to ensure that the modifications can be reversed at the decoder level.
[0047] The embodiments described below are independent of the implementation of IPD and ITD parameter analysis and compensation. In other words, these embodiments do not depend on how IPD and ITP are analyzed or compensated. In such embodiments, the channels for which ITD and IPD are adjusted are indicated by asterisks:
[0048]
[0049] The input channels with ITD and IPD adjusted are then downmixed by the parametric analysis and downmix block 410 to produce an intermediate / side representation, also referred to as downmix / side representation. One way to perform downmixing is to use sums and differences of the signals.
[0050]
[0051] Downmix signalX M (m, k) is encoded by the downmix encoder 420 to be stored and / or sent to the decoder. The encoding can be performed in the frequency domain, but it can also be performed in the time domain. In that case, a DFT synthesis stage is required to generate a time domain version of the downmix signal, which is then provided to the downmix encoder 420. However, the transformation to the time domain may introduce a delay misalignment with the multi-channel audio parameters, which may require additional processing. In one embodiment, this is solved by introducing an additional delay or by interpolating the parameters to ensure that the decoder synthesis of the downmix is aligned with the multi-channel audio parameters.
[0052] Complementary secondary signal X S (m, k) can be generated by the local parameter synthesis block 440 according to the downmix and the obtained multi-channel audio parameters. The downmixed signal can be exported using:
[0053]
[0054] Where p(·) is the prediction function and can be implemented as a single scaling factor α that minimizes the mean square error (MSE) between the side signal and the predicted side signal. Further, the prediction can be applied over frequency bands and involves prediction parameters for each frequency band b.
[0055]
[0056] If the coefficients for band b are specified as a column vector and X M,b (m), then the minimum MSE predictor can be derived as
[0057]
[0058] However, this expression can be simplified to produce more stable prediction parameters.b can be used as an alternative implementation of the ILD parameter. More details are described in the prediction model in reference [4].
[0059] Given the predicted side signal, the prediction residual X can be created R (m,k)[4].
[0060]
[0061] The prediction residual can be input into the residual encoder 450. The encoding can be performed directly in the DFT domain or can be performed in the time domain. Similarly, for a downmix encoder, the time domain encoder will require DFT synthesis, which may require alignment of the signal in the decoder. The residual signal represents a diffuse component that is unrelated to the downmix signal. If the residual signal is not sent, a solution in one embodiment can be to replace the signal for the residual signal in the decoder in a stereo synthesis state with a signal from a decorrelated version of the decoded downmix signal. This replacement is typically used for low bit rates, where the bit budget is too low to represent the residual signal with any useful resolution. For intermediate bit rates, a portion of the residual is typically encoded. In this case, lower frequencies are typically encoded because they are more perceptually relevant. For the remainder of the spectrum, the decorrelator signal is used as a replacement for the residual signal in the decoder. This method is typically referred to as a hybrid coding mode [4]. More details are provided below in the decoder description.
[0062] The representation of the encoded downmix, the encoded multi-channel audio parameters, and the encoded residual signal are multiplexed into a bitstream 360 which may be sent to a decoder or stored in a medium for future decoding.
[0063] In one embodiment, a multi-channel decoder is used in the DFT domain, such as in Figures 5 to 7 Shown in. Figure 5 An embodiment of a decoder is shown, wherein Figure 6 The block generates a residual signal in case of frame loss. Figure 7 Shows Figure 5 and Figure 6 In the following description, the block combination should be used. Figure 7 However, it should be noted that Figure 7 The demultiplexer 710 provides at least Figure 5 The same function of the demultiplexer 510, Figure 7 The down-mix decoder 715 provides at least Figure 5 The down-mix decoder 520 has the same function as Figure 7 The stereo parameter decoder 725 provides at least Figure 5The same function as the stereo parameter decoder 530, Figure 7 The decorrelator 730 provides at least Figure 5 The decorrelator 540 has the same function as Figure 7 The residual decoder 735 provides at least Figure 5 The residual decoder 550 has the same function as Figure 7 The parameter synthesis block 760 provides at least Figure 5 The parameter synthesis block 560 has the same function. Similarly, Figure 7 The down-mixing PLC 720 provides at least Figure 6 The same function of down-mixing PLC 610, Figure 7 The decorrelator 730 provides at least Figure 6 The decorrelator 620 has the same function as Figure 7 The memory 740 provides at least Figure 6 The same function as the memory 630, Figure 7 The spectrum shaper 745 provides at least Figure 6 The spectrum shaper 640 has the same function, Figure 7 The Phase ECU 750 provides at least Figure 6 The same functions as the Phase ecu 650, Figure 7 The signal combiner 755 provides at least Figure 6 The same function as the signal combiner 660, and Figure 7 The parameter synthesis block 760 provides at least Figure 6 The parameter synthesis block 670 has the same function.
[0064] Now go to Figure 7 , the down-mix decoder 715 provides a reconstructed down-mix signal The signal is segmented into DFT analysis frames m, where n = 0, 1, 2, ..., N-1 represents the number of samples in frame m. The analysis frames are usually extracted in an overlapping manner, which allows the use of an overlap-add strategy in the DFT synthesis stage. The corresponding DFT spectrum can be obtained through DFT transformation:
[0065]
[0066] Where w(n) represents a suitable window function. The shape of the window function can be designed using a trade-off between frequency characteristics and algorithmic delay due to the length of the overlap region. Similarly, the residual decoder 735 performs a scalar decoder for frame m and time instance n=0, 1, 2, ... N. R Generate the reconstructed residual signal Note that the frame length N Rcan be different from N, since the residual signal can be generated at a different sampling rate. Since the residual encoding can only be for a lower frequency range, it can be beneficial to represent it with a lower sampling rate to save memory and computational complexity. Obtain the DFT representation of the residual signal Note that if the residual signal is upsampled in the DFT domain to the same sampling rate as the reconstructed downmix, the DFT coefficients will need to be multiplied by N / N R to scale, and will be zero-padded to match the length N. To simplify the notation, and because the present embodiment is not affected by using different sampling rates, in order to better understand the method, the sampling rates shall be equal in the following description, and N R = N. Therefore, no scaling or zero padding should be shown.
[0067] It should be noted that in case the downmix and / or residual signal is encoded in the DFT domain, no frequency transformation by means of the DFT is required. In this case, decoding of the downmix and / or residual signal provides the DFT spectrum required for further processing.
[0068] In error-free frames (often referred to as good frames), the multi-channel audio decoder uses the decoded downmix signal together with the decoded multi-channel audio parameters and combines the decoded residual signal to produce a multi-channel synthesis. is stored in memory 740 so that the variable The residual signal spectrum of the last received frame is always maintained.
[0069] In some embodiments, relevant sub-parts of the spectrum can be stored to save memory, for example, only lower frequency regions are stored. In other embodiments, the residual signal can be stored in the time domain, and the DFT spectrum can be obtained only when an error occurs. This can reduce the peak calculation complexity because the error concealment operation usually has a lower complexity than the decoding of the correctly received frame. In the following description, the residual signal has been transformed to the DFT domain during normal operation, and the residual signal is stored as a DFT spectrum. In other embodiments, the residual signal is stored in the time domain. In these embodiments, the residual signal spectrum is obtained by transforming the residual signal to the DFT domain.
[0070] Decoded downmix is fed to the decorrelator 730 to synthesize the uncorrelated signal components D(m,n), and the resulting signal is transformed into the DFT domain X D (m,k). Note that decorrelation can also be performed in the frequency domain. Decoded downmix Decorrelation component X D (m,k) and the residual signal Together with the multi-channel audio parameters P(m), they are fed to the parametric multi-channel synthesis block 760 to generate a reconstructed multi-channel audio signal. After multi-channel synthesis has been applied in the DFT domain, the left and right channels are transformed into the time domain and output from the stereo decoder.
[0071] Go to Fig.12 , when the decoder 100 detects a lost or damaged multi-channel audio frame (i.e., a bad frame) of the encoded multi-channel audio signal, the decoder 100 may perform operations. Figure 7 In operation 1201, the PLC of the downmix decoder 715 is activated and generates an error concealment frame for the downmix. In operation 1203, the downmix error concealment frame is frequency transformed to generate a corresponding DFT spectrum In operation 1205, the transformed downmix error concealed frame may be input into the same decorrelator function 730 that was used for downmixing to generate a decorrelated concealed frame D ECU (m,n), or is input to different decorrelator functions and then frequency transformed to produce a decorrelated down-mix hidden frame X D,ECU (m,k).
[0072] The decorrelator function can be done in the time domain before the transform, in the form of an all-pass filter, delay, or a combination thereof. It can also be done in the frequency domain after the frequency transform, in which case it operates on frames, possibly including past frames.
[0073] In operation 1207, a residual signal spectrum is obtained. The residual signal spectrum may be retrieved from a memory where it has been previously stored. In the case where the residual signal is stored prior to the DFT transform operation, the residual signal spectrum is obtained by performing a DFT operation on the stored residual signal. In order to generate a hidden frame of the residual signal, in operation 1209, an energy adjusted decorrelated residual signal is generated. In operation 1209, the phase ECU 750 performs a phase extrapolation or phase evolution strategy on the residual signal from the past synthesis stored in the memory 740 as previously described. See also [3].
[0074] Go to Fig.13 In operation 1301, a phase extrapolation or phase evolution strategy phase shifts the peak sinusoid of the residual signal spectrum (see Figure 3 sine component), and in operation 1303, the noise spectrum of the non-peak sinusoidal curve (see Figure 3 More details on these operations are given in Fig.14 is provided.
[0075] Go to Fig.14 In operation 1401, the residual signal spectrum (may also be referred to as the "prototype signal") is first input to a peak detector circuit, which detects peak frequencies on a fractional frequency scale. A set of peaks can be detected:
[0076] F={f i},i=1,2,…N peaks ,
[0077] These peaks are given by their estimated fractional frequencies f i Indicates that, and where N peaks is the number of detected peaks. Here, fractional frequencies are represented as fractions of DFT bins, such that, for example, the Nyquist frequency is found at f=N / 2+1. Then in operation 1403, each detected peak is associated with a number of frequency bins representing the detected peak. The number of frequency bins can be found by rounding the fractional frequency to the nearest integer and including neighboring bins, e.g., N on each side. peaks Peaks:
[0078]
[0079] Among them, [·] represents the rounding operation, G i It means that at frequency f i The number of regions with peak values at peaks is a fine tuning constant determined when designing the system. peaks Provides greater accuracy in each peak representation, but also introduces greater distances between peaks that can be modeled. peaks Suitable values of can be 1 or 2. The hidden spectrum of the residual signal X R,ECU (m, k) is formed by inserting region groups, including a phase adjustment operation 1405, which is based on the fractional frequency and the number of samples between the analysis frame of the previous frame and where the current frame will begin.
[0080] N step =NN overlap
[0081] According to the following phase adjustment, for each peak frequency f i The phase adjustment is applied to each corresponding zone group G i :
[0082] Δφ i =2πN step f i / N,
[0083] This phase adjustment is applied to the corresponding region of the hidden spectrum of the residual signal:
[0084]
[0085] In operation 1407, decorrelation concealment frame X is used D,ECU The spectral coefficients of (m,k) are used to fill X R,ECU The unpeaked area G of (m,k) i The remaining area occupied by ω (which can be called the noise spectrum or the noise component of the spectrum). To ensure that the coefficients have the appropriate energy levels and overall spectral shape, the energy can be adjusted to match the residual spectrum memory The energy of the noise spectrum. This can be done by adding all the peak regions G in the calculation buffer i This is done by setting it to zero and matching the energy of the remaining noise spectrum region. Fig. 10A As shown in , energy matching can be performed on a frequency band basis.
[0086] Go to Fig.15 In operation 1501, a spanning region range k is specified. start(b) …k end(b) In operation 1503, the energy matching gain factor g b can be calculated as:
[0087]
[0088] In operation 1505, the noisy spectral region k is filled with the energy adjusted decorrelated residual concealment frame using the energy matching gain factor:
[0089] For frequency band b, Note that scaling can also be applied to a wide or narrow band or even to each frequency region. In the case of scaling per region, the residual memory The amplitude spectrum of the hidden frame X is maintained while applying the decorrelation D,ECU The phase of the spectrum of (m,k). For example, it can be obtained by X D,ECU The magnitude of (m,k) is adjusted to match The magnitude or The phase is adjusted to match X D,ECU Scaling can be achieved by changing the phase of (m,k). However, performing the scaling on a frequency band basis preserves some of the spectral fine structure that may be desired.
[0090] In the embodiment where scaling is performed for each frequency bin, the decorrelation concealment frame X is applied. D,ECU The phase of the spectrum of (m,k) can be approximated using the phase. This can reduce the complexity of scaling. The energy matching gain factor g kcan be calculated as:
[0091]
[0092] The energy-adjusted decorrelated residual hidden frame is used to fill the noisy spectral region k using the energy matching gain factor:
[0093]
[0094] g k The calculation of involves square roots and divisions, which can be computationally complex. In an embodiment, an approximate phase adjustment is used that matches the sign and order of the absolute values of the real and imaginary components of the phase target so that the phase moves within π / 4 of the phase target. This embodiment can skip the energy matching gain factor g k Gain scaling performed. X R,ECU (m,k) can be written as:
[0095] X R,ECU (m,k)=a+jb
[0096]
[0097] Among them, when the absolute values of the real and imaginary components are of the same order, that is,
[0098]
[0099] (c,d) is
[0100]
[0101] otherwise,
[0102]
[0103] Approximate phase adjustment at Fig.19 is shown in Fig.19 The phase target is represented by X at 1900 D,ECU (m,k) is given. ECU synthesis without phase adjustment is illustrated at 1904. The ECU synthesis X after the approximate phase adjustment has been applied D,ECU (m, k) is illustrated at 1902. Approximate phase adjustments may be used on a frequency band basis and / or on a per frequency bin basis.
[0104] Note that if no tonal component is found, i.e. no peak is detected, the entire hidden frame will be replaced by the decorrelated hidden frame X with spectral shaping applied. R,ECU (m,k). Fig.17 Go to Fig.17 , in operation 1701, the decoder 100 detects whether there is a peak signal in the residual signal spectrum on the scale of fractional frequency. If there is a peak signal, operations 1703 to 1707 are performed. Specifically, in operation 1703, each peak frequency is associated with a plurality of peak frequency regions. Operation 1703 is similar in operation to operation 1403. In operation 1705, a phase adjustment is applied to each of the plurality of peak frequency regions. Operation 1705 is similar in operation to operation 1405. In operation 1707, the spectral coefficients of the decorrelated hidden frame are used to fill the remaining regions, and the energy levels of the remaining regions are adjusted to match the energy levels of the noise spectrum of the residual spectrum memory. Operation 1707 is similar in operation to operation 1407. If there is no peak signal, operation 1709 is performed, which fills all regions with the spectral coefficients of the decorrelated hidden frame, and adjusts the energy levels of the regions to match the energy levels of the noise spectrum of the residual spectrum memory.
[0105] In order to perform stereo synthesis of error concealed frames, it is necessary to estimate the multi-channel parameters for the lost frames. This concealment can be done in various ways, but one method that has been found to give reasonable results is to simply repeat the stereo parameters from the last received frame to produce the multi-channel audio replacement parameters
[0106] The final spectrum of the hidden residual spectrum is found by combining the spectrum peaks with the energy adjusted noise spectrum in the signal combiner 755. An example of the combination is shown in Fig. 10B is shown in the figure.
[0107] return Fig.12 In operation 1213, the downmix error concealment frame Decorrelation downmix hidden frame X D,ECU (m,k) and the energy adjusted decorrelated residual hidden frame X R,ECU (m,k) and multi-channel audio parameters are fed together to the parametric synthesis block 760 to produce the reconstructed signal. After the synthesis has been applied in the DFT domain, in operation 1215 the multi-channel signal is transformed into the time domain (eg, left and right channels) and output from the stereo decoder.
[0108] For example, in Fig.16 In operation 1601, a multi-channel audio signal is generated based on the reconstructed signal (ie, the replacement frame). In operation 1603, the multi-channel audio signal is output to at least one speaker for playback.
[0109] Go to Figures 5 to 7, showing the DFT and IDFT. The IDFT is used to decouple the downmix decoding and residual decoding from the DFT analysis stage. In other embodiments, the IDFT is not used. In other embodiments where the signal processing described above is performed in the time domain, the DFT is only used to provide a decorrelated downmix concealed frame X D,ECU (m,k) and residual signal spectrum The IDFT is used to provide their time domain counterparts.
[0110] Go to Figure 8 and Fig. 9 , depicting Fig.12 Flowchart of how the hiding operation of the residual signal can be performed serially or in parallel. In the case of an error-free frame, the DFT spectrum of the residual signal is stored in a memory and updated in each error-free frame in operation 810. The memory is then used for concealment of "lost frames". When the decoder detects frame loss / damage or is notified of frame loss / damage, in operation 820, the PLC algorithm designed for the downmix part is activated and a downmix signal is generated. The PLC algorithm used for down-mixing can be selected from the above-mentioned techniques. Then, in operation 830, can be fed to a decorrelator to extract the uncorrelated signal X D,ECU (m, k). Decorrelation can also be performed in the time domain. In addition, a downmix memory storing downmix signals of past frames can be included in the input of the decorrelator. Then in operation 840, the residual from the last good frame is Note that operations 830 and 840 are independent of each other and may be performed in other ways. In order to keep the shape of the residual signal close to the residual of the last good frame, in operation 850, the spectrum of the decorrelator signal is reshaped based on the residual signal of the last good frame. In operation 860, the phase-shifted sinusoidal component of the residual signal of the last good frame and the reshaped decorrelated signal are combined to generate a hidden frame X of the residual signal. R,ECU (m, k). In another embodiment, the decoder may process operations 820 and 830 in parallel with operation 840. Fig. 9 is shown in the figure.
[0111] Fig. 10A and Fig. 10B An example showing how the decorrelator signal can be shaped. Fig. 10A The residual signal spectrum (labeled as prototype) and the decorrelator output are shown. Fig. 10B shows the hidden frame X for the residual signal derived as described above. R,ECU (m,k).
[0112] As previously mentioned, the input to the parameter synthesis block 670 may alternatively be in the time domain. Fig.18 1 shows the operation of the decoder 100 when the input of the parameter synthesis block 670 is in the time domain and the parameter synthesis block synthesizes the signal in the time domain. Operations 1801 to 1811 are similar to those described above. Fig.12 The same operations as operations 1201 to 1211 of the decoder 100 are performed. In operation 1813, the decoder 100 performs an inverse frequency domain (IFD) transform on the decorrelated hidden frame and the hidden frame of the residual signal. In operation 1815, the generated IFD transform signal and the parametric multi-channel audio time domain replacement parameters are provided to the multi-channel audio synthesis component, which generates output channels in the time domain.
[0113] List of Examples:
[0114] 1. A method for approximating a lost or damaged multi-channel audio frame of a received multi-channel audio signal in a decoding device comprising a processor, the method comprising the following operations performed by the processor:
[0115] Generate a downmix error concealment frame (610, 720, 820, 1201);
[0116] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1203);
[0117] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (620, 730, 830, 1205);
[0118] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1207);
[0119] Generating an energy adjusted decorrelated residual signal concealment frame (640-660, 745-755, 850-860, 1209) using the residual signal spectrum;
[0120] Get a multi-channel audio replacement parameter set;
[0121] providing (1213) the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and the multi-channel audio replacement parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and
[0122] An inverse frequency domain transform of the synthesized multi-channel audio frame is performed (1215) to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0123] 2. The method of embodiment 1, wherein the multi-channel audio replacement parameter set is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0124] 3. The method according to any one of embodiments 1-2, further comprising:
[0125] generating (1601) a multi-channel audio signal based on the replacement frame; and
[0126] The multi-channel audio signal is output (1603) to at least one speaker for playback.
[0127] 4. The method according to any one of embodiments 1-3, wherein obtaining the residual signal spectrum includes retrieving the residual signal spectrum from a storage device.
[0128] 5. The method according to any one of embodiments 1-4, wherein generating an energy adjusted decorrelated residual signal hidden frame comprises:
[0129] Phase shifting (650, 750, 840, 1301) the peak sinusoidal component of the residual signal spectrum; and
[0130] The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted (640, 745, 850, 1303).
[0131] 6. The method of any one of embodiments 1-4, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0132] Detecting the peak frequency of the residual signal spectrum of the stored residual signal on a fractional frequency scale (1401, 1701);
[0133] Associating each peak frequency with a plurality of peak frequency regions representing the peak frequency (1403, 1703);
[0134] Applying a phase adjustment (650, 750, 840, 1405, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0135] The remaining region (1407, 1707) of the residual signal hidden spectrum is filled using the spectral coefficients of the decorrelated hidden frame, and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0136] 7. The method according to any one of embodiments 1-4, wherein generating an energy adjusted decorrelated residual signal hidden frame comprises:
[0137] Detecting whether there is a peak frequency (650, 750, 840, 1701) in the residual signal spectrum of the stored residual signal on a fractional frequency scale;
[0138] In response to detecting that there is no peak frequency in the residual signal spectrum:
[0139] filling (1709) each bin of the residual signal hidden spectrum using spectral coefficients of the decorrelated hidden frame and adjusting the energy level of the bin to match the energy level of the noise spectrum of the residual signal spectrum;
[0140] In response to detecting a peak frequency in the residual signal spectrum:
[0141] Associating each peak frequency with a plurality of peak frequency bins representing the peak frequency (1703);
[0142] Applying a phase adjustment (650, 750, 840, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0143] The remaining region of the residual signal hidden spectrum is filled with the spectral coefficients of the decorrelated hidden frame (1707), and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0144] 8. The method of any one of embodiments 6-7, wherein adjusting the energy level of the remaining region to match the energy level of the noise spectrum of the residual signal spectrum comprises matching the energy level on a frequency band basis.
[0145] 9. The method of embodiment 7, wherein the frequency band b spans (1501) a region range k start(b) …k end(b) , and the matching energy levels include:
[0146] Calculate (1503) the energy matching gain factor g b for
[0147]
[0148] and filling (1505) the remaining area with the energy adjusted decorrelated residual hidden frame,
[0149] For frequency band b,
[0150] 10. The method according to any one of embodiments 1-9, wherein the generation of the energy adjusted decorrelated residual signal concealed frame is performed in parallel with the transformation of the downmix error concealed frame into the frequency domain and the decorrelation of the transformed downmix concealed frame.
[0151] 11. A method according to any one of embodiments 1-10, wherein one of transforming the downmix error concealment frame into the frequency domain and decorrelating the transformed downmix error concealment frame is performed before the other of transforming the downmix error concealment frame into the frequency domain and decorrelating the transformed downmix error concealment frame.
[0152] 12. A decoder (100) for a communication network, the decoder (100) comprising:
[0153] A processor (1101); and
[0154] A memory (1103) coupled to a processor, wherein the memory includes instructions, which when executed by the processor, cause the processor to perform operations according to any one of embodiments 1-11.
[0155] 13. A computer program comprising computer executable instructions, which are configured to, when executed on a processor (1101) included in a device, cause the device to perform the method according to any one of embodiments 1-11.
[0156] 14. A computer program product, comprising a computer-readable storage medium having computer-executable instructions, the computer-executable instructions being configured to, when executed on a processor (1101) included in a device, cause the device to perform a method according to any one of embodiments 1-11.
[0157] 15. An apparatus configured to approximate lost or damaged multi-channel audio frames of a received multi-channel audio signal, the apparatus comprising:
[0158] at least one processor (1101);
[0159] A memory (1103) communicatively coupled to the processor, the memory comprising instructions executable by the processor, the instructions causing the processor to perform operations comprising:
[0160] Generate a downmix error concealment frame (610, 720, 820, 1201);
[0161] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1203);
[0162] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (620, 730, 830, 1205);
[0163] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1207);
[0164] Generating an energy adjusted decorrelated residual signal concealment frame (640-660, 745-755, 850-860, 1209) using the residual signal spectrum;
[0165] Obtain (1211) a multi-channel audio replacement parameter set;
[0166] providing (1213) the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and the multi-channel audio replacement parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and
[0167] An inverse frequency domain transform of the synthesized multi-channel audio frame is performed (1215) to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0168] 16. The apparatus of embodiment 15, wherein the set of multi-channel audio replacement parameters is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0169] 17. The apparatus of any one of embodiments 15-16, further comprising:
[0170] generating (1601) a multi-channel audio signal based on the replacement frame; and
[0171] The multi-channel audio signal is output (1603) to at least one speaker for playback.
[0172] 18. The apparatus of any one of embodiments 15-17, wherein obtaining the residual signal spectrum comprises retrieving the residual signal spectrum from a storage device.
[0173] 19. The apparatus of any one of embodiments 15-18, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0174] Phase shifting (650, 750, 840, 1301) the peak sinusoidal component of the residual signal spectrum; and
[0175] The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted (640, 745, 850, 1303).
[0176] 20. The apparatus of any one of embodiments 15-18, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0177] Detecting the peak frequency of the residual signal spectrum (1401, 1701) of the stored residual signal on a fractional frequency scale;
[0178] Associating each peak frequency with a plurality of peak frequency regions representing the peak frequency (1403, 1703);
[0179] Applying a phase adjustment (650, 750, 840, 1405, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0180] The remaining region (1407, 1707) of the residual signal hidden spectrum is filled using the spectral coefficients of the decorrelated hidden frame, and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0181] 21. The apparatus of any one of embodiments 15-18, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0182] Detecting whether there is a peak frequency (650, 750, 840, 1701) in the residual signal spectrum of the stored residual signal on a fractional frequency scale;
[0183] In response to detecting that there is no peak frequency in the residual signal spectrum:
[0184] Filling (1709) each region of the residual signal hidden spectrum using spectral coefficients of the decorrelated hidden frame and adjusting the energy level of the region to match the energy level of the noise spectrum of the residual signal spectrum; In response to detecting that there is a peak frequency in the residual signal spectrum:
[0185] Associating each peak frequency with a plurality of peak frequency bins representing the peak frequency (1703);
[0186] Applying a phase adjustment (650, 750, 840, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0187] The remaining region of the residual signal hidden spectrum is filled with the spectral coefficients of the decorrelated hidden frame (1707), and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0188] 22. The apparatus of any one of embodiments 20-21, wherein adjusting the energy level of the residual region to match the energy level of the noise spectrum of the residual signal spectrum comprises matching the energy level on a frequency band basis.
[0189] 23. The apparatus of embodiment 22, wherein the frequency band b spans (1501) a region range k start(b) …k end(b) , and the matching energy levels include:
[0190] Calculate (1503) the energy matching gain factor g b for
[0191]
[0192] and filling (1505) the remaining area with the energy adjusted decorrelated residual hidden frame,
[0193] For frequency band b,
[0194] 24. An audio decoder comprising the apparatus according to any one of embodiments 14-21.
[0195] 25. A decoder configured to perform operations comprising:
[0196] Generate a downmix error concealment frame (610, 720, 820, 1201);
[0197] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1203);
[0198] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (620, 730, 830, 1205);
[0199] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1207);
[0200] Generating an energy adjusted decorrelated residual signal concealment frame (640-660, 745-755, 850-860, 1209) using the residual signal spectrum;
[0201] Obtain (1211) a multi-channel audio replacement parameter set;
[0202] providing (1213) the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and the multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and
[0203] An inverse frequency domain transform of the synthesized multi-channel audio frame is performed (1215) to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0204] 26. The decoder of embodiment 25, wherein the multi-channel audio replacement parameter set is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0205] 27. A computer program product comprising a non-transitory computer readable medium storing computer program code which, when executed by at least one processor, causes the at least one processor to:
[0206] Generate a downmix error concealment frame (610, 720, 820, 1201);
[0207] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1203);
[0208] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (620, 730, 830, 1205);
[0209] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1207);
[0210] Generating an energy adjusted decorrelated residual signal concealment frame (640-660, 745-755, 850-860, 1209) using the residual signal spectrum;
[0211] Obtain (1211) a multi-channel audio replacement parameter set;
[0212] providing (1213) the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and the multi-channel audio parameters from a previously received multi-channel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and
[0213] An inverse frequency domain transform of the synthesized multi-channel audio frame is performed (1215) to generate a replacement frame for the lost or damaged multi-channel audio frame.
[0214] 28. The computer program product of embodiment 27, wherein the set of multi-channel audio replacement parameters is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0215] 29. The computer program product of any one of embodiments 27-28, wherein the non-transitory computer readable medium stores other computer program code that, when executed, causes at least one processor to:
[0216] generating (1601) a multi-channel audio signal based on the replacement frame; and
[0217] The multi-channel audio signal is output (1603) to at least one speaker for playback.
[0218] 30. The computer program product of any one of embodiments 27-29, wherein obtaining the residual signal spectrum comprises retrieving the residual signal spectrum from a storage device.
[0219] 31. The computer program product of any one of embodiments 27-20, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0220] Phase shifting (650, 750, 840, 1301) the peak sinusoidal component of the residual signal spectrum; and
[0221] The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted (640, 745, 850, 1303).
[0222] 32. The computer program product of any one of embodiments 27-30, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0223] Detecting the peak frequency of the residual signal spectrum (1401, 1701) of the stored residual signal on a fractional frequency scale;
[0224] Associating each peak frequency with a plurality of peak frequency regions representing the peak frequency (1403, 1703);
[0225] Applying a phase adjustment (650, 750, 840, 1405, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0226] The remaining region (1407, 1707) of the residual signal hidden spectrum is filled using the spectral coefficients of the decorrelated hidden frame, and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0227] 33. The computer program product of any one of embodiments 27-30, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0228] Detecting whether there is a peak frequency (650, 750, 840, 1701) in the residual signal spectrum of the stored residual signal on a fractional frequency scale;
[0229] In response to detecting that there is no peak frequency in the residual signal spectrum:
[0230] Each bin of the residual signal hidden spectrum is filled (1709) using the spectral coefficients of the decorrelated hidden frame and the energy level of the bin is adjusted to match the energy level of the noise spectrum of the residual signal spectrum (X R,CUR (m,k)=gX D,ECU (m,k)); In response to detecting a peak frequency in the residual signal spectrum:
[0231] Associating each peak frequency with a plurality of peak frequency bins representing the peak frequency (1703);
[0232] Applying a phase adjustment (650, 750, 840, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0233] The remaining region of the residual signal hidden spectrum is filled with the spectral coefficients of the decorrelated hidden frame (1707), and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0234] 34. The computer program product of any one of embodiments 32-33, wherein adjusting the energy level of the residual region to match the energy level of the noise spectrum of the residual signal spectrum comprises matching the energy level on a frequency band basis.
[0235] 35. The computer program product of embodiment 34, wherein frequency band b spans (1501) region range k start(b) …k end(b) , and the matching energy levels include:
[0236] Calculate (1503) the energy matching gain factor g b for
[0237]
[0238] and filling (1505) the remaining area with the energy adjusted decorrelated residual hidden frame,
[0239] For frequency band b,
[0240] 36. A method of approximating a lost or damaged multi-channel audio frame of a received multi-channel audio signal in a decoding device comprising a processor, the method comprising the following operations performed by the processor:
[0241] Generate downmix error concealment frame (610, 720, 820, 1801);
[0242] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1803);
[0243] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (620, 730, 830, 1805);
[0244] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (810, 1807);
[0245] Generating an energy adjusted decorrelated residual signal concealment frame (640-660, 745-755, 850-860, 1809) using the residual signal spectrum;
[0246] Obtain (1811) a multi-channel audio replacement parameter set;
[0247] performing (1813) an inverse frequency domain transform of the transformed downmix error concealed frame, the energy adjusted decorrelated residual concealed frame, and the multi-channel audio parameters from a previously received multi-channel audio signal frame to generate a transformed downmix error concealed time domain frame, an energy adjusted decorrelated residual concealed time domain frame, and the multi-channel audio time domain parameters; and
[0248] The transformed downmix error concealed time domain frame, the energy adjusted decorrelated residual concealed time domain frame and the multi-channel audio time domain parameters are provided (1815) to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame.
[0249] 37. The method of embodiment 36, wherein the multi-channel audio replacement parameter set is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0250] 38. The method of any one of embodiments 36-37, further comprising:
[0251] generating (1601) a multi-channel audio signal based on the synthesized multi-channel audio replacement frame; and
[0252] The multi-channel audio signal is output (1603) to at least one speaker for playback.
[0253] 39. The method of any one of embodiments 36-38, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0254] Phase shifting (650, 750, 840, 1301) the peak sinusoidal component of the residual signal spectrum; and
[0255] The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted (640, 745, 850, 1303).
[0256] 40. The method of any one of embodiments 36-38, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0257] Detecting the peak frequency of the residual signal spectrum of the stored residual signal on a fractional frequency scale (1401, 1701);
[0258] Associating each peak frequency with a plurality of peak frequency regions representing the peak frequency (1403, 1703);
[0259] Applying a phase adjustment (650, 750, 840, 1405, 1705) to each of the plurality of peak frequency regions according to the phase adjustment to form a residual signal hidden spectrum; and
[0260] The remaining region (1407, 1707) of the residual signal hidden spectrum is filled using the spectral coefficients of the decorrelated hidden frame, and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0261] 41. The method of any one of embodiments 36-38, wherein generating an energy adjusted decorrelated residual signal concealment frame comprises:
[0262] Detecting whether there is a peak frequency (650, 750, 840, 1701) in the residual signal spectrum of the stored residual signal on a fractional frequency scale;
[0263] In response to detecting that there is no peak frequency in the residual signal spectrum:
[0264] filling (1709) each bin of the residual signal hidden spectrum using spectral coefficients of the decorrelated hidden frame and adjusting the energy level of the bin to match the energy level of the noise spectrum of the residual signal spectrum;
[0265] In response to detecting a peak frequency in the residual signal spectrum:
[0266] Associating each peak frequency with a plurality of peak frequency bins representing the peak frequency (1703);
[0267] Applying a phase adjustment to each of the plurality of peak frequency regions according to the phase adjustment (650, 750, 840, 1705)
[0268] to form a residual signal to hide the spectrum; and
[0269] The remaining region of the residual signal hidden spectrum is filled with the spectral coefficients of the decorrelated hidden frame (1707), and the energy level of the remaining region is adjusted to match the energy level of the noise spectrum of the residual signal spectrum.
[0270] 42. The method of any one of embodiments 40-41, wherein adjusting the energy level of the residual region to match the energy level of the noise spectrum of the residual signal spectrum comprises matching the energy level on a frequency band basis by:
[0271] Designate (1501) frequency band b across zone range k start(b) …k end(b) ;
[0272] Calculate (1503) the energy matching gain factor g b for
[0273]
[0274] and filling (1507) the remaining area with the energy adjusted decorrelated residual hidden frame,
[0275] For frequency band b,
[0276] 43. A computer program product comprising a non-transitory computer readable medium storing computer program code which, when executed by at least one processor, causes the at least one processor to:
[0277] Generate a downmix error concealment frame (1801);
[0278] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1803);
[0279] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (1805);
[0280] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1807);
[0281] Generating an energy adjusted decorrelated residual signal hidden frame using the residual signal spectrum (1809);
[0282] Obtain a multi-channel audio time domain replacement parameter set;
[0283] performing (1813) an inverse frequency domain transform of the transformed down-mix error concealed frame and the energy adjusted decorrelated residual concealed frame to generate a transformed down-mix error concealed time domain frame and an energy adjusted decorrelated residual concealed time domain frame; and
[0284] The transformed downmix error concealed time domain frame, the energy adjusted decorrelated residual concealed time domain frame and the multi-channel audio time domain replacement parameters are provided (1815) to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame.
[0285] 44. The computer program product of embodiment 38, wherein the set of multi-channel audio time domain replacement parameters is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0286] 45. An apparatus configured to approximate lost or damaged multi-channel audio frames of a received multi-channel audio signal, the apparatus comprising:
[0287] at least one processor (1101);
[0288] A memory (1103) communicatively coupled to the processor, the memory comprising instructions executable by the processor, the instructions causing the processor to perform operations comprising:
[0289] Generate a downmix error concealment frame (1801);
[0290] transforming the downmix error concealed frame into the frequency domain to generate a transformed downmix error concealed frame (1803);
[0291] decorrelating the transformed down-mix hidden frame to generate a decorrelated hidden frame (1805);
[0292] Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame (1807);
[0293] Generating an energy adjusted decorrelated residual signal hidden frame using the residual signal spectrum (1809);
[0294] Obtain (1811) a multi-channel audio time domain replacement parameter set;
[0295] performing (1813) an inverse frequency domain transform of the transformed down-mix error concealed frame and the energy adjusted decorrelated residual concealed frame to generate a transformed down-mix error concealed time domain frame and an energy adjusted decorrelated residual concealed time domain frame; and
[0296] The transformed downmix error concealed time domain frame, the energy adjusted decorrelated residual concealed time domain frame and the multi-channel audio time domain replacement parameters are provided (1815) to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio replacement frame.
[0297] 46. The apparatus of embodiment 39, wherein the set of multi-channel audio time domain replacement parameters is obtained by repeating parameters from a previously received multi-channel audio signal frame.
[0298] A description of the abbreviations disclosed above is provided below.
[0299] Abbreviation
[0300] DFT Discrete Fourier Transform
[0301] LP Linear Prediction
[0302] PLC Packet Loss Concealment
[0303] ECU Error Hiding Unit
[0304] FEC Frame Error Correction / Concealment
[0305] MDCT Modified Discrete Cosine Transform
[0306] MDST Modified discrete sine transform
[0307] ODFT Odd Discrete Fourier Transform
[0308] LTP Long Term Predictor
[0309] ITD Inter-channel Time Difference
[0310] IPD Inter-channel Phase Difference
[0311] ILD Inter-channel level difference
[0312] ICC Inter-channel Coherence
[0313] FD Frequency Domain
[0314] TD Time Domain
[0315] FLC Frame Loss Concealment
[0316] BFI Bad Frame Indicator
[0317] QMF Quadrature Mirror Filter Bank
[0318] Citations to references in the above disclosures are provided below.
[0319] [1]. C. Faller, “Parametric multichannel audio coding: synthesis of coherence cues”, IEEE Journal on Audio, Speech and Language Processing, Vol. 14, No. 1, pp. 299-310, January 2006.
[0320] [2]. J. Lecomte et al., “Packet-loss concealment technology advances in EVS,” in 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Brisbane, Queensland, 2015, pp. 5708–5712.
[0321] [3]. S. Bruhn, E. Norvell, J. Svedberg, and S. Sverrisson, “A novel sinusoidal approach to audio signal frame loss concealment and its application in the new evs codec standard,” in 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Brisbane, Queensland, 2015, pp. 5142–5146.
[0322] [4]. Breebaart, J., Hotho, G., Koppens, J., Schuijers, E., “Background, Concept, and Architecture for the Recent MPEG Surround Standard on Multichannel Audio Compression”, J. Audio Eng, Soc., Vol. 55, No. 5, May 2007.
[0323] Further definitions and embodiments are discussed below.
[0324] In the above description of various embodiments of the inventive concept, it should be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the inventive concept. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the inventive concept belongs. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of this specification and the related art, and will not be idealized or over-interpreted unless clearly defined in this article.
[0325] When an element is referred to as being "connected", "coupled", "responsive" or a variation thereof to another element, it may be directly connected to, coupled to, or responsive to another element, or there may be an intermediate element. On the contrary, when an element is referred to as being "directly connected", "directly coupled", "directly responsive" or a variation thereof to another element, there is no intermediate element. The same number always refers to the same element. In addition, the "coupling", "connection", "responsiveness" or a variation thereof used herein may include wireless coupling, connection or response. As used herein, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. For the sake of brevity and / or clarity, well-known functions or configurations may not be described in detail. The term "and / or" includes any and all combinations of one or more related listed items.
[0326] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments. Throughout the specification, the same reference numerals or the same reference signs represent the same or similar elements.
[0327] As used herein, the terms "comprise," "comprising," "including," "comprising," "includes," "including," "having," "having," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. In addition, as used herein, the common abbreviation "eg (for example)" derived from the Latin phrase "exempli gratia" may be used to introduce or specify one or more general examples of previously mentioned items, and is not intended to limit such items. The general abbreviation "ie (that is)" derived from the Latin phrase "id est" may be used to specify a specific item from a more general statement.
[0328] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices), and / or computer program products. It should be understood that the blocks of the block diagrams and / or flowchart illustrations and the combination of blocks in the block diagrams and / or flowchart illustrations can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a general-purpose computer circuit, a processor circuit of a special-purpose computer circuit, and / or other programmable data processing circuits to produce a machine so that the instructions executed via the processor of the computer and / or other programmable data processing devices transform and control transistors, values stored in memory locations, and other hardware components in such circuit systems to implement the functions / actions specified in the block diagrams and / or one or more flowchart blocks, thereby creating a means (functionality) and / or structure for implementing the functions / actions specified in the block diagrams and / or (one or more) flowchart blocks.
[0329] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in the block diagram and / or one or more flowchart blocks. Therefore, embodiments of the present invention may be embodied in hardware and / or in software (including firmware, resident software, microcode, etc.) running on a processor such as a digital signal processor, which may be collectively referred to as "circuitry," "module," or variations thereof.
[0330] It should also be noted that in some alternative implementations, the function / action marked in the box may not occur in the order marked in the flow chart. For example, depending on the function / action involved, two boxes shown as continuous can actually be performed substantially in parallel, or boxes can sometimes be performed in reverse order. In addition, the function of a given box of a flow chart and / or block diagram can be divided into a plurality of boxes, and / or the function of two or more boxes of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the present invention, other boxes can be added / inserted between the illustrated boxes, and / or boxes / operations can be omitted. In addition, although some figures include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the depicted arrows.
[0331] In the case of not departing from the principle of the inventive concept in essence, various changes and modifications may be made to the embodiments. All these changes and modifications are intended to be included in the scope of the inventive concept. Therefore, the above disclosed subject matter is considered to be illustrative rather than restrictive, and the examples of the embodiments are intended to cover all such modifications, enhancements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, to the maximum extent permitted by law, the scope of the inventive concept will be determined by the broadest permissible interpretation of the disclosure including the embodiments and their equivalent examples, and should not be restricted or limited by the foregoing detailed description.
[0332] Additional instructions are provided below.
[0333] Generally, all terms used in this article should be interpreted according to their common meaning in the relevant technical field, unless different meanings are clearly given and / or different meanings are implied from the context of its use. Unless explicitly stated, otherwise, all references to one / an / the element, device, assembly, part, step, etc. should be openly interpreted as referring to at least one instance of the element, device, assembly, part, step, etc. The steps of any method disclosed herein are not necessarily performed in the exact order disclosed, unless a step is clearly described as after or before another step and / or it is implied that a step must be after or before another step. As long as it is appropriate, any feature of any embodiment disclosed herein can be applied to any other embodiment. Similarly, any advantage of any embodiment can be applied to any other embodiment, and vice versa. Other purposes, features and advantages of the disclosed embodiment will be apparent from the following description.
[0334] Any suitable steps, methods, features, functions or benefits disclosed herein may be performed by one or more functional units or modules of one or more virtual devices. Each virtual device may include multiple such functional units. These functional units may be implemented by processing circuits, which may include one or more microprocessors or microcontrollers and other digital hardware, and other digital hardware may include digital signal processors (DSPs), dedicated digital logic, etc. The processing circuit may be configured to execute program codes stored in a memory, which may include one or more types of memory, such as read-only memory (ROM), random access memory (RAM), cache memory, flash memory device, optical storage device, etc. The program code stored in the memory includes program instructions for executing one or more telecommunications and / or data communication protocols and instructions for executing one or more technologies described herein. In some embodiments, the processing circuit may be used to cause the corresponding functional unit to perform the corresponding function according to one or more embodiments of the present disclosure.
[0335] The term "unit" may have a conventional meaning in the field of electronics, electrical devices and / or electronic devices and may include, for example, electrical and / or electronic circuits, devices, modules, processors, memories, logical solid-state and / or discrete devices, computer programs or instructions for performing corresponding tasks, procedures, calculations, output and / or display functions, etc., such as described in this document.
Claims
1. A method for approximating a lost or damaged multi-channel audio frame of a received multi-channel audio signal in a decoding device, the method comprising: generating a downmix error concealment frame; transforming the downmix error concealment frame into a frequency domain to obtain a transformed downmix error concealment frame; decorrelating the transformed down-mix hidden frame to obtain a decorrelated hidden frame; Obtaining a residual signal spectrum of a previously decoded multi-channel audio signal frame; Using the residual signal spectrum to generate an energy-adjusted decorrelated residual signal hidden frame; Get a multi-channel audio replacement parameter set; generating a synthetic multi-channel audio frame using the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and the multi-channel audio replacement parameters; as well as An inverse frequency domain transform of the synthesized multi-channel audio frame is performed to generate a replacement frame for the lost or damaged multi-channel audio frame.
2. The method according to claim 1, wherein: The multi-channel audio replacement parameter set is obtained by repeating parameters from the previously received multi-channel audio signal frame.
3. The method according to claim 1, further comprising: generating a multi-channel audio signal based on the replacement frame; as well as The multi-channel audio signal is output to at least one speaker for playback.
4. The method according to claim 1, wherein: Obtaining the residual signal spectrum includes: retrieving the residual signal spectrum from a storage device.
5. The method according to claim 1, wherein: Generating the energy adjusted decorrelated residual signal hidden frame comprises: Phase shifting the peak sinusoidal component of the residual signal spectrum; and The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted.
6. The method according to claim 1, wherein: Generating the energy adjusted decorrelated residual signal hidden frame comprises: detecting a peak frequency of the residual signal spectrum of the stored residual signal on a fractional frequency scale; associating each detected peak frequency with a plurality of adjacent frequency bins forming a set of bins representing the peak frequency; applying a phase adjustment to each of the plurality of peak frequency regions; and The phase adjusted peak frequency region and the energy adjusted decorrelated residual signal concealment frame are used to form a residual signal concealment spectrum.
7. The method according to claim 1, wherein: Generating the energy adjusted decorrelated residual signal concealment frame includes detecting whether a peak frequency exists in the residual signal spectrum of the stored residual signal on a fractional frequency scale.
8. The method according to claim 7, wherein: In response to not detecting a peak frequency in the residual signal spectrum, each region of the residual signal concealment spectrum is populated with spectral coefficients of the energy adjusted decorrelated residual signal concealment frame.
9. The method according to claim 5, wherein: Adjusting the energy level includes combining a phase of a region of the decorrelation concealment frame with an amplitude of a region of the residual signal concealment spectrum.
10. The method according to claim 9, wherein: Combining the phases includes applying an approximate phase adjustment by matching the signs and orders of real and imaginary components of the residual signal hidden spectrum to the decorrelated hidden frame.
11. An audio decoder configured to approximate lost or damaged multi-channel audio frames of a received multi-channel audio signal, the audio decoder being configured to: generating a downmix error concealment frame; transforming the downmix error concealment frame into a frequency domain to obtain a transformed downmix error concealment frame; decorrelating the transformed down-mix hidden frame to obtain a decorrelated hidden frame; Obtaining a residual signal spectrum of a stored residual signal of a previously received multi-channel audio signal frame; Using the residual signal spectrum to generate an energy-adjusted decorrelated residual signal hidden frame; Get a multi-channel audio replacement parameter set; generating a synthesized multi-channel audio frame using the transformed downmix error concealment frame, the energy adjusted decorrelated residual concealment frame, and multi-channel audio replacement parameters from the previously received multi-channel audio signal frame; as well as An inverse frequency domain transform of the synthesized multi-channel audio frame is performed to generate a replacement frame for the lost or damaged multi-channel audio frame.
12. The audio decoder according to claim 11, wherein: The multi-channel audio replacement parameter set is obtained by repeating parameters from the previously received multi-channel audio signal frame.
13. The audio decoder according to claim 12, further configured to: generating a multi-channel audio signal based on the replacement frame; and The multi-channel audio signal is output to at least one speaker for playback.
14. The audio decoder according to claim 11, wherein: Obtaining the residual signal spectrum includes: retrieving the residual signal spectrum from a storage device.
15. The audio decoder according to claim 11, wherein: Generating the energy adjusted decorrelated residual signal hidden frame comprises: Phase shifting the peak sinusoidal component of the residual signal spectrum; and The energy of the noise spectrum of the non-peak sinusoidal component of the residual signal spectrum of the stored residual signal is adjusted.
16. The audio decoder according to claim 11, wherein: Generating the energy adjusted decorrelated residual signal hidden frame comprises: detecting a peak frequency of the residual signal spectrum of the stored residual signal on a fractional frequency scale; associating each detected peak frequency with a plurality of adjacent frequency bins forming a set of bins representing the peak frequency; applying a phase adjustment to each of the plurality of peak frequency regions; and The phase adjusted peak frequency region and the energy adjusted decorrelated residual signal concealment frame are used to form a residual signal concealment spectrum.
17. The audio decoder according to claim 11, wherein: Generating the energy adjusted decorrelated residual signal concealment frame includes detecting whether a peak frequency exists in the residual signal spectrum of the stored residual signal on a fractional frequency scale.
18. The audio decoder of claim 17, further configured to fill each region of the residual signal hidden spectrum with spectral coefficients of the energy adjusted decorrelated residual signal hidden frame in response to no peak frequency being detected in the residual signal spectrum.
19. The audio decoder of claim 15, wherein: Adjusting the energy level includes combining a phase of a region of the decorrelation concealment frame with an amplitude of a region of the residual signal concealment spectrum.
20. The audio decoder of claim 19, wherein: Combining the phases includes applying an approximate phase adjustment by matching the signs and orders of real and imaginary components of the residual signal hidden spectrum to the decorrelated hidden frame.
