Audo decoder and decoding method
The use of multi-tap convolution matrices with varying frequency resolution addresses the high computational complexity and bitrate issues in audio signal processing for headphones, enhancing efficiency and reducing power consumption on mobile devices.
Patent Information
- Application Number
- JP2024160571
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2015-10-08
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-05
AI Technical Summary
Existing audio signal processing methods for headphone playback, such as HRIR and BRIR convolution, face high computational complexity and power consumption issues, especially with object-based content, and conventional parametric techniques require high bitrates and complex filter banks, which are impractical for mobile devices.
A method using multi-tap convolution matrices with varying frequency resolution, where low frequencies use multi-tap matrices and higher frequencies use stateless matrices, reducing computational complexity and memory usage by compensating for decoder frequency resolution with multi-tap convolution matrices.
This approach reduces computational complexity, memory usage, and bitrate requirements, enabling efficient audio signal processing for various playback formats on mobile devices while maintaining audio quality.
Smart Images

Figure 2024169554000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 62 / 209,742, filed August 25, 2015, and European Patent Application No. 15189008.4, filed October 8, 2015, the contents of each of which are incorporated herein by reference in their entirety.
[0002] Technical Field The present invention relates to the field of signal processing, and in particular, discloses a system for efficient transmission of audio signals with spatialized components. [Background technology]
[0003] Any discussion of background art throughout the specification should in no way be considered an admission that such art is widely known or forms part of the common general knowledge in the art.
[0004] Audio content creation, encoding, distribution and playback has traditionally been performed in channel-based formats, i.e., one specific target playback system is considered for the content throughout the content ecosystem. Examples of such target playback system audio formats are mono, stereo, 5.1, 7.1, etc.
[0005] When content is played on a playback system different from the one for which it was intended, a downmix or upmix process can be applied. For example, 5.1 content can be played on a stereo playback system by using a specific downmix formula. Another example is playing stereo encoded content on a 7.1 speaker setup. This may involve a so-called upmix process, where the upmix may or may not be guided by information present in the stereo signal. One system with upmix capabilities is Dolby Pro Logic from Dolby Laboratories, Inc. (Non-Patent Document 1).
[0006] When stereo or multi-channel content is played over headphones, it is often desirable to simulate a multi-channel speaker setup with a head-related impulse response (HRIR) or binaural room impulse response (BRIR). HRIR and BRIR simulate the acoustic path from each loudspeaker to the eardrum in a (simulated) anechoic or reverberant environment, respectively. Specifically, the audio signal can be convolved with the HRIR or BRIR to recover the inter-aural level difference (ILD), inter-aural time difference (ITD) and spectral cues to allow the listener to discern the location of each individual channel. Simulation of the acoustic environment (reverberation) also helps to achieve a certain perceived distance.
[0007] Sound source localization and virtual speaker simulation When stereo, multi-channel or object-based content is played over headphones, it is often desirable to simulate a multi-channel speaker setup or a set of discrete virtual acoustic objects by means of Head-Related Impulse Responses (HRIRs) or Binaural Room Impulse Responses (BRIRs), which simulate the acoustic paths from each loudspeaker to the eardrum in a (simulated) anechoic or reverberant environment, respectively.
[0008] Specifically, the audio signal can be convolved with the HRIR or BRIR to recover the interaural level difference (ILD), interaural time difference (ITD) and spectral cues that allow the listener to discern the location of each individual channel or object. Simulation of the acoustic environment (early reflections and late reverberation) also helps to achieve a certain perceived distance.
[0009] Turning to FIG. 1, two object or channel signals x are read from content store 12 for processing by four HRIRs (e.g., 14). i A schematic overview 10 of the process flow for rendering 13, 11 is shown. The HRIR outputs are then summed (15, 16) for each channel signal to generate a headphone and speaker output for playback to the listener via headphones 18. The basic principles of HRIR are explained for example in non-patent document 2.
[0010] The HRIR / BRIR convolution approach has several drawbacks. One of them is the significant amount of processing required for headphone playback. HRIR or BRIR convolution needs to be applied separately for every input object or channel, so the computational complexity typically grows linearly with the number of channels or objects. Because headphones are typically used in conjunction with battery-powered portable devices, high computational complexity is undesirable as it substantially reduces battery life. Furthermore, with the introduction of object-based audio content, which may contain over 100 simultaneously active objects, the computational complexity of HRIR convolution can become substantially higher than for traditional channel-based content.
[0011] Parametric coding techniques Computational complexity is not the only issue for the delivery of channel or object-based content within the content authoring, distribution and playback ecosystem. In many practical situations, especially for mobile applications, the available data rates for content delivery are severely constrained. Consumers, broadcasters and content providers have delivered stereo (two-channel) audio content using lossy perceptual audio codecs with typical bitrates between 48 and 192 kbits / s. These conventional channel-based audio codecs, such as MPEG-1 Layer 3 (Non-Patent Document 6), MPEG AAC (Non-Patent Document 7) and Dolby Digital (Non-Patent Document 8), have bitrates that scale almost linearly with the number of channels. As a result, the delivery of tens or even hundreds of objects leads to bitrates that are impractical or even unavailable for consumer delivery purposes.
[0012] To allow the delivery of complex object-based content at bit rates comparable to those required for stereo content delivery using conventional perceptual audio codecs, so-called parametric methods have been the subject of research and development over the last decade. These parametric methods allow the reconstruction of a large number of channels or objects from a relatively small number of elementary signals. These elementary signals can be conveyed from the transmitter to the receiver using conventional audio codecs augmented with additional (parametric) information to allow the reconstruction of the original objects or channels. Examples of such techniques are Parametric Stereo (Non-Patent Document 3), MPEG Surround (Non-Patent Document 4) and MPEG Spatial Audio Object Coding (Non-Patent Document 5).
[0013] An important aspect of techniques such as parametric stereo and MPEG Surround is that these methods aim at the parametric reconstruction of a single pre-determined presentation (e.g. stereo loudspeakers for parametric stereo and 5.1 speakers for MPEG Surround). In the case of MPEG Surround, a headphone virtualizer can be integrated into the decoder, which generates a virtual 5.1 loudspeaker setup for headphones, in which the virtual 5.1 speakers correspond to the 5.1 loudspeaker setup for loudspeaker playback. As a consequence, these presentations are not independent in that the headphone presentation represents the same (virtual) loudspeaker layout as the loudspeaker presentation. On the other hand, MPEG spatial audio object coding aims at the reconstruction of objects, which requires subsequent rendering.
[0014] Turning now to Fig. 2, a parametric system 20 supporting channels and objects is depicted in overview. The system is divided into an encoder 21 and a decoder 22 part. The encoder 21 receives channels and objects 23 as input and generates a downmix 24 with a limited number of base signals. Furthermore, a set of object / channel reconstruction parameters 25 is calculated. A signal encoder 26 encodes the base signals from the downmixer 24 and includes the calculated parameters 25 and object metadata 27, which indicates how the objects should be rendered, in the resulting bitstream.
[0015] The decoder 22 first decodes (29) the elementary signals and then performs channel and / or object reconstruction 30 with the aid of the transmitted reconstruction parameters 31. The resulting signals can be played directly (if channels) or rendered 32 (if objects). For the latter, each reconstructed object signal is rendered according to its associated object metadata. One example of such metadata is a position vector (e.g. the x, y, z coordinates of an object in a three-dimensional coordinate system).
[0016] Matrix Processing in Decoder The object and / or channel reconstruction 30 can be achieved by time- and frequency-varying matrix operations. s The first stage typically involves transforming the fundamental signal through a transform or filter bank, denoted as [n], where s is the fundamental signal index and n is the sample index.
[0017] A wide variety of transforms and filter banks can be used, for example the Discrete Fourier Transform (DFT), the Modified Discrete Cosine Transform (MDCT) or a Quadrature Mirror Filter (QMF) bank. The output of such a transform or filter bank is denoted as Zs[k,b], where b is the subband or spectral index and k is the frame, slot or subband time or sample index.
[0018] In most cases, a subband or spectral index is mapped to a smaller set of parameter bands p that share common object / channel reconstruction parameters. This can be denoted by b∈B(p). In other words, B(p) denotes the set of consecutive subbands b that belong to parameter band index p. Conversely, p(b) denotes the parameter band index p to which subband b is mapped. Then, the reconstructed channel or object in the subband or transform domain [Y J 〕 is signal Z i is obtained by matrix multiplication with the matrix M[p(b)].
[0019]
number
[0020] The above process typically applies to a limited range of subband samples, slots or frames k. In other words, the matrix M[p(b)] is typically updated / modified over time. For simplicity of notation, these updates are not shown here, but it is conceivable that the processing of the set of samples k associated with the matrix M[p(b)] can be a time-varying process.
[0021] In some cases where the number of signals to be reconstructed J is significantly greater than the number of basic signals S, an optional decorrelator output D acting on one or more of the basic signals may be used. m It is often helpful to use [k,b], which can be included in the reconstructed output signal.
[0022]
number
[0023] Minimum Mean Square Error (MMSE) Prediction for Object / Channel Reconstruction Basic signal Z s Although various strategies and methods exist for reconstructing an object or channel from the set of [k,b], one specific method is often referred to as the minimum mean square error (MMSE) predictor. It uses correlation and covariance matrices to derive the matrix coefficients M that minimize the L2 norm between the desired signal and the reconstructed signal. For this method, the basis signals z s [n] is the input object or channel signal x i It is generated as a linear combination of [n].
[0024]
number
[0025]
number
[0026]
number
[0027] The basis for calculating the matrix coefficients M by the encoder is the decoder output [Y j 〕 and original input object / channel X j The goal is to minimize the mean squared error E, which represents the squared error between
[0028]
number
[0029] M=(Z * Z+εI) -1 Z * X where ε is a regularization constant and * is the complex conjugate transpose operator. This operation can be performed independently for each parameter band b, yielding the matrix M[p(b)].
[0030] Minimum Mean Square Error (MMSE) Prediction for Representation Transformation Besides object and / or channel reconstruction, parametric techniques can be used to transform one representation into another. An example of such a representation transformation can be used to convert a stereo mix intended for loudspeaker playback into a binaural representation for headphones, and vice versa.
[0031] Figure 4 shows the control flow for a method 50 for one such representation transformation. The object or channel audio is first processed in an encoder 52 by a hybrid quadrature mirror filter decomposition bank 54. A loudspeaker rendering matrix G is calculated based on the object metadata using amplitude panning techniques to produce an object signal X stored in a storage medium 51. i Applied to stereo loudspeaker presentation Z55 s This loudspeaker presentation can be encoded using an audio encoder 57.
[0032] Furthermore, a binaural rendering matrix H is generated using an HRTF database 59 and applied (58). This matrix H is then applied to the binaural signal Y j This allows the reconstruction of a binaural mix using the stereo loudspeaker mix as input. The matrix coefficients M are encoded by the audio encoder 57.
[0033] The information to be transmitted is transmitted from the encoder 52 to the decoder 53, where it is divided into components M and Z s If loudspeakers are used as the playback system, the loudspeaker presentation is s and thus the matrix coefficients M are discarded. On the other hand, for headphone playback, the loudspeaker presentation is first transformed 62 to a binaural presentation by applying the time- and frequency-varying matrix M before hybrid QMF synthesis and playback 60.
[0034] The desired binaural output from matrix processing element 62 is expressed in matrix notation as Y=XH In this case, the matrix coefficient M is expressed as follows in the encoder 52: M=(G * X * XG+εI) -1 G * X * XH can be obtained by
[0035] In this application, the coefficients of the encoder matrix H applied in 58 are typically complex-valued, e.g. with delay or phase correction elements, allowing the restoration of interaural time differences that are perceptually very important for sound source localization in headphones. In other words, the binaural rendering matrix H is complex-valued, and thus the transformation matrix M is complex-valued. For a perceptually transparent restoration of sound source localization cues, it has been shown that a frequency resolution that mimics the frequency resolution of the human auditory system is desirable (Non-Patent Document 11).
[0036] In the above sections, the minimum mean square error criterion is used to determine the matrix coefficients M. Without loss of generality, other well-known criteria or methods for calculating the matrix coefficients M can be used as well to replace or augment the minimum mean square error principle. For example, the matrix coefficients M can be calculated using higher order error terms or by minimizing the L1 norm (e.g., the minimum absolute deviation criterion). Furthermore, various methods can be used, including non-negative factorization or optimization techniques, non-parametric estimators, maximum likelihood estimators, etc. Furthermore, the matrix coefficients can be calculated using iterative or gradient descent processes, interpolation methods, heuristic methods, dynamic programming, machine learning, fuzzy optimization, simulated annealing, or closed-form solutions, and "analysis by synthesis" techniques can be used. Last but not least, the matrix coefficient estimates may be constrained in various ways, for example, by limiting the range of values, regularization terms, superposition of energy conservation requirements, etc.
[0037] Transformation and Filter Bank Requirements Depending on the application and whether an object or a channel is to be reconstructed, certain requirements may be imposed on the transform or filter bank frequency resolution for the filter bank unit 41 of Fig. 3. In most practical applications, the frequency resolution is adapted to the assumed resolution of the human auditory system in order to give the best perceived audio quality for a given bit rate (determined by the number of parameters) and computational effort. It has been found that the human auditory system can be considered as a filter bank with a non-linear frequency resolution. These filters are called critical bands (Non-Patent Document 9) and are approximately logarithmic in nature. At low frequencies, the critical bands are less than 100 Hz wide, while at high frequencies, the critical bands can be wider than 1 kHz.
[0038] When it comes to filter bank design, this nonlinear behavior can present challenges: if the frequency resolution is constant over frequency, transforms and filter banks can be implemented very efficiently using symmetries in their processing structures.
[0039] This implies that the transform length or number of subbands is determined by the critical bandwidth at low frequencies, and a mapping of DFT bins to so-called parameter bands can be used to mimic the nonlinear frequency resolution. Such a mapping process is described, for example, in
[10] and
[11] . One drawback of this approach is that a very long transform is required to satisfy the low-frequency critical bandwidth constraint, while at high frequencies the transform is relatively long (or inefficient). An alternative solution to increase the frequency resolution at low frequencies is to use a hybrid filterbank structure. In such a structure, a cascade of two filterbanks is used, where the second filterbank increases the resolution of the first, but only in the lowest subbands ([3]).
[0040] Fig. 5 shows one form of hybrid filter bank structure 41 similar to that described in [3]. The input signal z[n] is first processed by a complex-valued quadrature mirror filter decomposition bank (CQMF) 71. The signal is then downsampled by a factor Q, e.g. 72, to give subband signals Z[k,b], where k is the subband sample index and b is the subband frequency index. At least one of the resulting subband signals is further processed by a second (Nyquist) filter bank 74. Meanwhile, the remaining subband signals are delayed 75 to compensate for the delay introduced by the Nyquist filter bank. In this example, the cascade of filter banks gives eight subbands (b=1,...,8), which are mapped to six parameter bands p=1,...,6 with nonlinear frequency resolution. The bands 76 merged together form a single parameter band (p=6).
[0041] The benefit of this approach is the lower computational complexity compared to using a single filter bank with many more (narrower) subbands, however the drawback is that the overall system delay increases significantly and, as a result, memory usage is also significantly higher and power consumption increases.
[0042] Limitations of conventional technology Returning to FIG. 4, the prior art uses a channel, object, or presentation signal [Y J 〕 to the basic signal Z s It can be seen that the concept of matrix processing 62, possibly augmented by the use of a decorrelator, is utilized to reconstruct from the set of . This leads to the following matrix formulation, which describes the prior art in a general way:
[0043]
number
[10] for parametric stereo coding or [4] for multi-channel decoding. Furthermore, this technique can also be used to recover inter-channel phase differences by using the complex-valued matrix coefficients (see
[11] ,
[12] ).
[0044] As shown in Fig. 6, in effect, using complex-valued matrix coefficients implies that the desired delay 80 is represented by a piecewise constant phase approximation 81. If we assume that the desired phase response is a pure delay 80 with a phase that decreases linearly with frequency (dashed line), the prior art complex-valued matrix processing operations give a piecewise constant approximation 81 (solid line). This approximation can be improved by increasing the resolution of the matrix M, but this has two important drawbacks. It requires an increase in the resolution of the filter bank, which leads to higher memory usage, higher computational complexity, longer latency and therefore higher power consumption. It also requires sending more parameters, which leads to higher bit rates.
[0045] All these shortcomings are particularly problematic for mobile and battery-powered devices, and it would be advantageous if a more optimal solution were available. [Prior art documents] [Non-patent literature]
[0046] [Non-Patent Document 1] Roger Dressler, "Dolby Pro Logic Surround Decoder, Principles of Operation", www.Dolby.com [Non-Patent Document 2] Wightman , FL , and Kistler , DJ ( 1989 ) “ Headphone simulation of free-field listening .” I. Stimulus synthesis , J. Acoust . Soc. Am. 85, 858–8
Outdoor Tools3
Outdoor Tools 4
Direct Environment 5
Outdoor Configuration6
Direct Environment 7
Outdoor Tools 8
Outdoor Tools9
Outdoor Tools 10
[0047] It is an object of the present invention in a preferred form to provide an improved form of encoding and decoding of audio signals for reproduction in a variety of presentations. [Means for solving the problem]
[0048] According to a first aspect of the present invention, there is provided a method for representing a second representation of an audio channel or an object as a data stream, comprising: (a) providing a set of elementary signals representing the first representation of the audio channel or object; and (b) providing a set of transformation parameters intended to transform the first representation into the second representation, the transformation parameters being specified for at least two frequency bands and further comprising a set of multi-tap convolution matrix parameters for at least one of the frequency bands.
[0049] The set of filter coefficients may represent a Finite Impulse Response (FIR) filter. The set of base signals is preferably divided into a series of time segments, and for each time segment a set of transformation parameters is provided. The filter coefficients may include at least one coefficient that may be complex valued. The first presentation or the second presentation may be intended for headphone reproduction.
[0050] In some embodiments, the transformation parameters associated with higher frequencies do not modify the signal phase, whereas for lower frequencies the transformation parameters modify the signal phase. The set of filter coefficients is preferably operable to process a multi-tap convolution matrix. The set of filter coefficients is preferably utilized to process a low frequency band.
[0051] The set of base signals and the set of transformation parameters are preferably combined to form the data stream. The transformation parameters may include high frequency audio matrix coefficients for a matrix operation of a high frequency portion of the set of base signals. In some embodiments, for a mid frequency portion of the high frequency portion of the set of base signals, the matrix operation may preferably include complex-valued transformation parameters.
[0052] According to a further aspect of the invention there is provided a decoder for decoding an encoded audio signal, the encoded audio signal comprising: a first presentation comprising a set of audio basis signals intended for reproduction of the audio in a first audio presentation format; and a set of transformation parameters for transforming the audio basis signals in the first presentation format to a second presentation format, the transformation parameters comprising at least high frequency audio transformation parameters and low frequency audio transformation parameters, the low frequency transformation parameters comprising multi-tap convolution matrix parameters, the decoder comprising: a first separation unit for separating the set of audio basis signals and the set of transformation parameters, a matrix multiplication unit for applying the multi-tap convolution matrix parameters to low frequency components of the audio basis signals to apply a convolution to the low frequency components to generate convolved low frequency components, a scalar multiplication unit for applying the high frequency audio transformation parameters to high frequency components of the audio basis signals to generate scalar high frequency components, and an output filter bank for combining the convolved low frequency components and the scalar high frequency components, the output filter bank generating a time domain output signal in the second presentation format.
[0053] The matrix multiplication unit is capable of modifying a phase of low frequency components of the audio fundamental signal. In some embodiments, the multi-tap convolution matrix transform parameters are preferably complex-valued. The high frequency audio transform parameters are also preferably complex-valued. The set of transform parameters may further include real-valued higher frequency audio transform parameters. In some embodiments, the decoder further includes a filter for separating the audio fundamental signal into the low frequency components and the high frequency components.
[0054] According to a further aspect of the invention, there is provided a method for decoding an encoded audio signal, the encoded audio signal comprising: a first presentation comprising a set of audio fundamental signals intended for reproduction of the audio in a first audio presentation format; and a set of transformation parameters for transforming the audio fundamental signals in the first presentation format to a second presentation format, the transformation parameters comprising at least high frequency audio transformation parameters and low frequency audio transformation parameters, the low frequency transformation parameters comprising multi-tap convolution matrix parameters, the method comprising: convolving low frequency components of the audio fundamental signals with the low frequency transformation parameters to generate convolved low frequency components; multiplying high frequency components of the audio fundamental signal with the high frequency transformation parameters to generate multiplied high frequency components; and combining the convolved low frequency components and the multiplied high frequency components to generate output audio signal frequency components for reproduction in the second presentation format.
[0055] In some embodiments, the encoded signal may comprise a plurality of temporal segments, and the method may preferably further comprise the steps of: interpolating transformation parameters of a plurality of temporal segments of the encoded signal to generate interpolated transformation parameters comprising interpolated low-frequency audio transformation parameters; and convolving a plurality of temporal segments of the low-frequency components of the audio fundamental signal with the interpolated low-frequency audio transformation parameters to generate a plurality of temporal segments of the convolved low-frequency components.
[0056] The set of transformation parameters of the encoded audio signal may preferably be time-varying, and the method may further comprise: convolving the low frequency components with the low frequency transformation parameters for a plurality of time segments to generate a plurality of sets of intermediate convolved low frequency components; and interpolating the plurality of sets of intermediate convolved low frequency components to generate the convolved low frequency components.
[0057] The interpolation may utilize an overlap-add method of the multiple sets of intermediate convolved low frequency components. [Brief description of the drawings]
[0058] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] Fig. 1 shows a schematic overview of the HRIR convolution process for two source objects, where each channel or object is processed by a pair of HRIR / BRIR. [Diagram 2] FIG. 1 illustrates a schematic diagram of a general parametric coding system supporting channels and objects. [Diagram 3] FIG. 3 shows diagrammatically one form of further detail of the channel or object reconstruction unit 30 of FIG. 2; [Figure 4] FIG. 1 shows the data flow for a method to convert a stereo loudspeaker presentation to a binaural headphone presentation. [Diagram 5] FIG. 1 is a schematic diagram of a hybrid analysis filter bank structure according to the prior art; [Figure 6] FIG. 1 shows a comparison of the desired phase response (dashed line) and the actual phase response (solid line) obtained with the prior art. [Figure 7] FIG. 1 is a schematic diagram of an exemplary encoder filter bank and parameter mapping system according to an embodiment of the present invention. [Figure 8]FIG. 2 is a schematic diagram illustrating a decoder filter bank and parameter mapping according to an embodiment. [Figure 9] FIG. 2 illustrates an encoder for stereo to binaural presentation conversion. [Figure 10] FIG. 2 shows a schematic diagram of a decoder for stereo to binaural presentation conversion. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0059] This preferred embodiment provides a method to reconstruct an object, channel or "presentation" from a set of elementary signals. This can be applied in a filter bank with low frequency resolution. An example is the conversion of a stereo presentation to a binaural presentation intended for headphone playback. This can be applied without a Nyquist (hybrid) filter bank. The reduced decoder frequency resolution is compensated for by a multi-tap convolution matrix. This convolution matrix only requires a small number of taps (e.g. two) and in practical cases is only required at low frequencies. This method (1) reduces the computational complexity of the decoder, (2) reduces the memory usage of the decoder, and (3) reduces the parameter bit rate.
[0060] In a preferred embodiment, a system and method is provided for overcoming undesirable decoder-side computational complexity and memory requirements by providing high frequency resolution in the encoder, utilizing a constrained (lower) frequency resolution in the decoder (e.g., using a frequency resolution significantly worse than that used in the corresponding encoder), and utilizing a multi-tap (convolution) matrix to compensate for the reduced decoder frequency resolution.
[0061] Typically, high frequency matrix resolution is only required at low frequencies, so multi-tap (convolution) matrices can be used at low frequencies, and regular (stateless) matrices can be used for the remaining (higher) frequencies. In other words, at low frequencies the matrices represent a set of FIR filters operating on each combination of input and output, while at high frequencies stateless matrices are used.
[0062] Encoder Filter Banks and Parameter Mapping FIG. 7 illustrates an exemplary encoder filter bank and parameter mapping system (90) according to one embodiment. In this exemplary embodiment 90, eight subbands (b=1,...,8), e.g., 91, are initially generated by a hybrid (cascaded) filter bank 92 and a Nyquist filter bank 93. The first four subbands are then mapped (94) to the same parameter band (p=1) to compute a convolution matrix M[k,p=1]. For example, the matrix now has an additional index k. The remaining subbands (b=5,...,8) are mapped to parameter bands (p=2,3) by using a stateless matrix M[p(b)] 95,96.
[0063] Decoder Filter Bank and Parameter Mapping Figure 8 shows a corresponding exemplary decoder filter bank and parameter mapping system 100. In contrast to the encoder, there is no Nyquist filter bank and no delays to compensate for the Nyquist filter bank delay. The decoder decomposition filter bank 101 produces only five subbands (b=1,...,5), e.g. 102. These are downsampled by a factor Q. The first subband is processed by a convolution matrix M[k,p=1] 103, while the remaining bands are processed by stateless matrices 104, 105 according to the prior art.
[0064] In the above example, the application of the Nyquist filter bank in the encoder 90 and the corresponding convolution matrix in the decoder 100 is only for the first CQMF subband, but the same process can be applied to multiple subbands, not necessarily limited to only the lowest subband(s).
[0065] Encoder embodiment One particularly useful embodiment is in the conversion of loudspeaker presentation to binaural presentation. Figure 9 shows an encoder 110 that uses the proposed method for presentation conversion. Given an input channel or object x i The set [n] is first transformed using a filter bank 111. The filter bank 111 is a Hybrid Complex Quadrature Mirror Bank (HCQMF), although other filter bank structures can equally be used. The resulting subband representation X i [k,b] is processed twice (112, 113).
[0066] First (113), the elementary signal Z intended for the output of the encoder s [k,b] 113. This output can be generated, for example, using amplitude panning techniques, such that the resulting signal is intended for loudspeaker playback.
[0067] Secondly (112), the desired transformed signal Y j The output generates a set of k,b signals [k,b] 112. This output can be generated using HRIR processing, for example, so that the resulting signal is intended for headphone playback. Such HRIR processing may be used in the filter bank domain, but can equally be performed in the time domain by HRIR convolution. The HRIRs are obtained from a database 114.
[0068] The convolution matrix M[k,p] is then s[k,b] is obtained by feeding it through a tapped delay line 116. Each tap of the delay line serves as an additional input to an MMSE predictor stage 115, which predicts the desired transformed signal Y j Calculate the convolution matrix M[k,p] that minimizes the error between [k,b] and the output of the decoder 100 of FIG. 8 that applies the convolution matrix. Then, the matrix coefficients M[k,p] are M=(Z * Z+εI) -1 Z * Y In this formulation, the matrix Z contains all the inputs of the tapped delay line.
[0069] Consider first the case of the reconstruction of one signal Y[k] for a given subband b, given A inputs from a tapped delay line:
[0070]
number
[0071]
number
[0072]
number
[0073] A further distinction can be made between complex-valued and real-valued stateless matrix processing. At low frequencies (typically below 1 kHz), a convolution process (A>1) is preferred to allow accurate reconstruction of inter-channel attributes aligned with the perceptual frequency scale. At intermediate frequencies up to about 2 or 3 kHz, the human auditory system is sensitive to inter-channel phase differences, but does not require very high frequency resolution for the reconstruction of such phase. This implies that a single-tap (stateless) complex-valued matrix is sufficient. For higher frequencies, the human auditory system is virtually insensitive to the fine structure phase of the waveform, and real-valued stateless matrix processing is sufficient. Reflecting the nonlinear frequency resolution of the human auditory system, the number of filterbank outputs mapped to one parameter band typically increases with increasing frequency.
[0074] In another embodiment, the first and second representations in the encoder are swapped, for example the first representation is intended for headphone reproduction and the second representation is intended for loudspeaker reproduction. In this embodiment, the loudspeaker representation (second representation) is generated by applying time-dependent transformation parameters in at least two frequency bands to the first representation, where the transformation parameters are specified as including a set of filter coefficients for at least one of the frequency bands.
[0075] In some embodiments, the first presentation is divided in time into a series of segments, and there is a separate matrix of transformation parameters for each segment. In one further refinement, if segment transformation parameters are not available, the parameters can be interpolated from previous coefficients.
[0076] Decoder embodiment Figure 10 shows an embodiment of decoder 120. Input bit stream 121 is split into a basic signal bit stream 131 and conversion parameter data 124. Thereafter, basic signal decoder 123 decodes the basic signal z[n]. It is then processed by decomposition filter bank 125. The resulting frequency domain signals Z[k,b] with subbands b = 1,…,5 are processed by matrix multiplication units 126, 129, and 130. Specifically, matrix multiplication unit 126 applies a complex-valued convolution matrix M[k,p = 1] to frequency domain signal Z[k,b = 1]. Further, matrix multiplication unit 129 applies a complex-valued single-tap matrix coefficient M[p = 2] to signal Z[k,b = 2]. Finally, matrix multiplication unit 130 applies a real-valued matrix coefficient M[p = 3] to frequency domain signals Z[k,b = 3,…5]. The matrix multiplication unit output signals are converted to time domain output 128 by synthesis filter bank 127. References to z[n], Z[k], etc. do not refer to any particular basic signal but rather to a set of basic signals. Thus, z[n], Z[k], etc. may be interpreted as z s [n], Z s [k], etc. Here, 0 ≤ s < N, where N is the number of basic signals.
[0077] In other words, matrix multiplication unit 126 determines the output sample of subband b = 1 of the output signal [^Y j [k]] from a weighted combination of the current samples of subband b = 1 of basic signal Z[k] and the previous samples of subband b = 1 of basic signal Z[k] (e.g., Z[k - a], where 0 < a < A and A is greater than 1). The weights used to determine the output sample of subband b = 1 of the output signal [^Y j [k]] correspond to the complex-valued convolution matrix M[k,p = 1] for the signal.
[0078] Further, matrix multiplier unit 129 [^Y jThe output sample of the subband b=2 of the fundamental signal Z[k] is determined from a weighted combination of the current samples of the subband b=2 of the fundamental signal Z[k]. j The weights used to determine the output sample for subband b=2 in [k] correspond to the complex-valued single-tap matrix coefficients M[p=2].
[0079] Finally, the matrix multiplier unit 130 produces an output signal Y j The output samples of the subbands b=3,...,5 of the fundamental signal Z[k] are determined from a weighted combination of the current samples of the subbands b=3,...,5 of the fundamental signal Z[k]. j The weights used to determine the output samples for subbands b=3,...,5 of [k] correspond to the real-valued matrix coefficients M[p=3].
[0080] In some cases, the fundamental signal decoder 123 operates on the signal with the same frequency resolution as provided by the analysis filterbank 125. In such cases, the fundamental signal decoder 125 may be configured to output a frequency domain signal Z[k] instead of a time domain signal z[n]. In that case, the analysis filterbank 125 may be omitted. Furthermore, in some cases it may be preferable to apply complex-valued single-tap matrix coefficients to the frequency domain signal Zs[k,b=3,...,5] instead of real-valued matrix coefficients.
[0081] In practice, the matrix coefficients M can be updated over time, for example by associating individual frames of the elementary signal with the matrix coefficients M. Alternatively or additionally, the matrix coefficients M may be time-stamped, indicating at which time or interval of the elementary signal z[n] the matrix should be applied. To reduce the transmission bit rate associated with the matrix updates, the number of updates is ideally limited, resulting in a matrix update distribution that is sparse in time. Such infrequent updates of the matrix require dedicated processing to ensure a smooth transition from one instance of the matrix to the next. The matrix M may be provided in association with a particular time segment (frame) and / or frequency domain of the elementary signal Z. The decoder may use various interpolation methods to ensure a smooth transition from subsequent instances of the matrix M over time. One example of such an interpolation method is to calculate overlapping windowed frames of the signal Z and for each such frame a corresponding set of output signals Y using the matrix coefficients M associated with that particular frame. Subsequent frames can then be summed using overlap-add techniques to provide a cross-fading transition. Alternatively, the decoder may receive a timestamp associated with the matrix M, which describes the desired matrix coefficients at a particular time. For audio samples intermediate between the timestamps, the matrix coefficients of matrix M may be interpolated using linear, cubic, bandlimited, or other interpolation means to ensure a smooth transition. In addition to interpolating through time, similar techniques may be used to interpolate the matrix coefficients through frequency.
[0082] Therefore, this paper focuses on audio channels or object X i The present invention describes a method (and a corresponding encoder 90) for representing a second representation of said audio channel or object X as a data stream transmitted or provided to a corresponding decoder 100. The method comprises: i The basic signal Z represents the first presentation of sAs outlined above, the step of providing a fundamental signal Z s is an audio channel or object X i from the first presentation may be determined using first rendering parameters G. The first presentation may be intended for loudspeaker playback or for headphone playback, while the second presentation may be intended for headphone playback or for loudspeaker playback. Thus, a conversion from loudspeaker playback to headphone playback (or vice versa) may be performed.
[0083] The method further comprises: s The output signal of the second presentation is expressed as Y j ], the transformation parameters M (in particular one or more transformation matrices) intended to transform the desired output signal Y j is an audio channel or object X i from the output signal {Y with ^} may be determined (as outlined in this paper) using the second rendering parameters H. The transformation parameters M are then calculated (for example using a minimum mean square error criterion) based on the j ], the desired output signal Y j may be determined by minimizing the deviation from
[0084] More specifically, the transformation parameters M may be determined in the subband domain (i.e. for different frequency bands). For this purpose, a fundamental signal Z[k,b] in the subband domain may be determined for B frequency bands using the encoder filter banks 92, 93. The number of frequency bands B is greater than 1, for example B is 4, 6, 8, 10 or more. In the example described here, B=8 or B=5. As outlined above, the encoder filter banks 92, 93 may comprise a hybrid filter bank providing a lower frequency band of the B frequency bands with a higher frequency resolution than a higher frequency band of the B frequency bands. Furthermore, a desired output signal Y[k,b] in the subband domain for the B frequency bands may be determined. The transformation parameters M for one or more frequency domains may be determined by a function of the resolution of the output signal [Y[k,b] in the one or more frequency bands (e.g. using a minimum mean square error criterion). j ], the desired output signal Y j may be determined by minimizing the deviation from
[0085] Thus, each of the transformation parameters M may be specified for at least two frequency bands, in particular for B frequency bands. Furthermore, the transformation parameters may include a set of multi-tap convolution matrix parameters for at least one of said frequency bands.
[0086] Thus, a method (and corresponding decoder) is described for determining an output signal of a second representation of an audio channel / object from a base signal of a first representation of said audio channel / object. The first representation may be used for loudspeaker reproduction and the second representation may be used for headphone reproduction (or vice versa). The output signal is determined using transformation parameters for different frequency bands, where the transformation parameters for at least one of said frequency bands include multi-tap convolution matrix parameters for at least one of said frequency bands. As a result of using multi-tap convolution matrix parameters for at least one of the frequency bands, the computational complexity of the decoder 100 may be reduced, in particular by reducing the frequency resolution of the filter bank used by the decoder.
[0087] For example, determining an output signal for a first frequency band using multi-tap convolution matrix parameters may include determining a current sample of the first frequency band of the output signal as a weighted combination of the current sample and one or more previous samples of the first frequency band of the fundamental signal, where weights used to determine the weighted combination correspond to the multi-tap convolution matrix parameters for the first frequency band, one or more of the multi-tap convolution matrix parameters for the first frequency band being typically complex-valued.
[0088] Furthermore, determining the output signal for a second frequency band may include determining a current sample of the second frequency band of the output signal as a weighted combination of current samples of the second frequency band of the fundamental signal (not based on previous samples of the second frequency band of the fundamental signal), where weights used to determine the weighted combination correspond to transformation parameters for the second frequency band, which may be complex-valued or real-valued.
[0089] Specifically, the same set of multi-tap convolution matrix parameters may be determined for at least two adjacent frequency bands of the B frequency bands. As shown in Fig. 7, a single set of multi-tap convolution matrix parameters may be determined for frequency bands provided by a Nyquist filter bank (i.e., for frequency bands with relatively high frequency resolution). In this way, the use of a Nyquist filter bank in the decoder 100 can be omitted, thereby reducing the computational complexity of the decoder 100 (while maintaining the quality of the output signal for the second presentation).
[0090] Furthermore, the same real-valued transform parameters may be determined for at least two adjacent high frequency bands (as shown in the context of Fig. 7), which may further reduce the computational complexity of the decoder 100 (while maintaining the quality of the output signal for the second presentation).
[0091] <interpretation> References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment," "some embodiments," or "in an embodiment" throughout this specification may, but do not necessarily, refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
[0092] As used herein, unless otherwise specified, the use of the ordinal adjectives "first," "second," "third," etc. to describe a common object indicates merely that different instances of a similar object are being referred to and is not intended to imply that the objects so described must be in a given order, whether in time, space, ranking, or in any other way.
[0093] In the claims and this specification, the terms "having" and "comprising" are both open terms, meaning that they include at least the elements / features mentioned, but do not exclude others. Thus, when used in the claims, the terms "having" and "comprising" should not be interpreted as being limited to the means or elements or steps listed. For example, the scope of an expression "a device having A and B" should not be limited to a device consisting of elements A and B only. The term "comprising" as used in this document is also an open term, meaning that they include at least the elements / features mentioned, but do not exclude others. Thus, "comprising" is synonymous with "having" and means "having".
[0094] As used herein, the term "exemplary" is used in the sense of providing an example rather than a denoting a quality, i.e., an "exemplary embodiment" is an embodiment that is provided as an example, but not necessarily an embodiment of exemplary nature.
[0095] In the foregoing description of exemplary embodiments of the invention, it should be understood that various features of the invention have been specifically grouped together in a single embodiment, drawing or description for the purposes of facilitating the flow of the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure is not to be interpreted as reflecting an intention that the claimed invention requires more features than are recited in each claim. Rather, as the appended claims reflect, inventive aspects lie in less than all features of a single disclosed embodiment as described above. As such, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of this invention.
[0096] Furthermore, although some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are intended to be within the scope of the present invention and to form different embodiments. Those skilled in the art will understand this. For example, in the claims, any of the claimed embodiments can be used in any combination.
[0097] Additionally, some of the embodiments are described herein as methods or combinations of method elements that may be implemented by a processor of a computer system or other means for performing the functions. Thus, the processor, together with the necessary instructions for performing such method or method element, constitutes a means for performing the method or method element. Additionally, the elements of the device embodiments described herein are examples of means for performing the functions performed by the elements to implement the invention.
[0098] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present invention may be practiced without such specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
[0099] Similarly, it should be noted that the term coupled, when used in the claims, should not be construed as being limited to only direct connections. The terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended to be synonymous with each other. Thus, the scope of the expression device A coupled to device B should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. It means that a path exists between the output of A and the input of B, and that the path may include other devices or means. "Coupled" may mean that two or more elements are in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but still cooperate or interact with each other.
[0100] Thus, having described what are believed to be preferred embodiments of the present invention, those skilled in the art will recognize that other and further modifications may be made without departing from the spirit of the present invention. It is intended to claim all such changes and modifications that fall within the scope of the present invention. For example, any formulas listed above are merely representative of procedures that may be used. Functions in the block diagrams may be added or deleted, and operations may be interchanged between functional blocks. Steps of methods described may be added or deleted within the scope of the present invention.
[0101] Various aspects of the present invention will be understood from the following enumerated example embodiments. [EEE1] 1. A method for representing a second presentation of an audio channel or an object as a data stream, comprising: (a) providing a set of base signals representing a first presentation of the audio channels or objects; (b) providing a set of transformation parameters intended to transform the first representation into the second representation, the transformation parameters further being specified for at least two frequency bands and including a set of multi-tap convolution matrix parameters for at least one of the frequency bands. method. [EEE2] The method of EEE1, wherein the set of filter coefficients represents a finite impulse response (FIR) filter. [EEE3] 3. The method of claim 1 or 2, wherein the set of elementary signals is divided into a series of time segments and for each time segment a set of transformation parameters is provided. [EEE4] 4. The method of any one of claims 1 to 3, wherein the filter coefficients include at least one coefficient that is a complex value. [EEE5] The method of any one of EEE1 to EEE4, wherein the first presentation or the second presentation is intended for headphone playback. [EEE6] The method of any one of EEE1 to EEE5, wherein transformation parameters associated with higher frequencies do not modify the signal phase, while for lower frequencies the transformation parameters modify the signal phase. [EEE7] 7. The method of any one of claims 1 to 6, wherein said set of filter coefficients is operable to process a multi-tap convolution matrix. [EEE8] The method of claim 7, wherein the set of filter coefficients is utilized to process a low frequency band. [EEE9] 9. A method according to any one of claims 1 to 8, wherein said set of base signals and said set of transformation parameters are combined to form said data stream. [EEE10] The method of any one of EEE1 to EEE9, wherein the transformation parameters include high frequency audio matrix coefficients for a matrix operation of a high frequency part of the set of base signals. [EEE11] The method of EEE10, wherein for intermediate frequency portions of the high frequency portion of the set of fundamental signals, the matrix operations include complex-valued transformation parameters. [EEE12] 1. A decoder for decoding an encoded audio signal, the encoded audio signal comprising: a first presentation including a set of audio elementary signals intended for reproduction of said audio in a first audio presentation format; a set of transformation parameters for transforming the audio fundamental signals in the first presentation format into a second presentation format, the transformation parameters including at least high frequency audio transformation parameters and low frequency audio transformation parameters, the low frequency transformation parameters including multi-tap convolution matrix parameters; The decoder in question: a first separation unit for separating said set of audio basis signals and said set of transformation parameters; a matrix multiplication unit for applying the multi-tap convolution matrix parameters to low frequency components of the audio fundamental signal; and applying the convolution to the low frequency components to generate convolved low frequency components; a scalar multiplication unit for applying the high frequency audio transformation parameters to high frequency components of the audio fundamental signal to generate scalar high frequency components; an output filter bank that combines the convolved low frequency components and the scalar high frequency components to generate a time-domain output signal in the second presentation format. decoder. [EEE13] The decoder of EEE12, wherein the matrix multiplication unit modifies a phase of the low frequency components of the audio fundamental signals. [EEE14] 14. The decoder of claim 12 or 13, wherein the multi-tap convolution matrix transform parameters are complex-valued. [EEE15] 15. A decoder according to any one of claims EEE12 to 14, wherein said high frequency audio transform parameters are complex valued. [EEE16] The decoder of EEE15, wherein the set of transformation parameters further includes real-valued, higher frequency audio transformation parameters. [EEE17] Decoder according to any one of claims EEE12 to EEE16, further comprising a filter for separating said audio fundamental signal into said low frequency and said high frequency components. [EEE18] 1. A method of decoding an encoded audio signal, the encoded audio signal comprising: a first presentation including a set of audio elementary signals intended for reproduction of said audio in a first audio presentation format; a set of transformation parameters for transforming the audio fundamental signals in the first presentation format into a second presentation format, the transformation parameters including at least high frequency audio transformation parameters and low frequency audio transformation parameters, the low frequency transformation parameters including multi-tap convolution matrix parameters; The method involves: convolving low frequency components of the audio fundamental signal with the low frequency transformation parameters to generate convolved low frequency components; multiplying high frequency components of the audio fundamental signal with the high frequency transformation parameter to generate multiplied high frequency components; combining the convolved low frequency components and the multiplied high frequency components to generate output audio signal frequency components for reproduction in a second presentation format. [EEE19] The encoded signal includes a plurality of time segments, and the method further comprises: interpolating transform parameters of a plurality of temporal segments of the encoded signal to generate interpolated transform parameters, including interpolated low frequency audio transform parameters; and convolving a plurality of time segments of the low frequency components of the audio fundamental signal with the interpolated low frequency audio transform parameters to generate a plurality of time segments of the convolved low frequency components. The method described in EEE18. [EEE20] The set of transformation parameters of the encoded audio signal is time-varying, the method further comprising: convolving the low frequency components with the low frequency transform parameters for a plurality of temporal segments to generate a plurality of sets of intermediate convolved low frequency components; and interpolating the plurality of sets of intermediate convolved low frequency components to generate the convolved low frequency components. The method described in EEE18. [EEE21] The method of any one of EEE19 and EEE20, wherein the interpolation utilizes an overlap-add method of the multiple sets of intermediate convolved low frequency components. [EEE22] The method of any one of EEE18 to 21, further comprising filtering the audio fundamental signal into the low frequency and high frequency components. [EEE23] A computer-readable non-transitory storage medium comprising program instructions for a computer to operate according to the method of any one of claims EEE1 to 11 and 18 to 22.
[0102] Several aspects will be described. [Aspect 1] 1. A method for representing a second presentation of an audio channel or an object as a data stream, comprising: (a) providing a base signal representing a first presentation of the audio channel or object; (b) providing transformation parameters intended to transform the fundamental signal of the first presentation into an output signal of the second presentation, each of the transformation parameters being specified for at least two frequency bands and comprising a set of multi-tap convolution matrix parameters for at least one of the frequency bands, the first presentation being for loudspeaker reproduction and the second presentation being for headphone reproduction or the first presentation being for headphone reproduction and the second presentation being for loudspeaker reproduction. method. [Aspect 2] 2. The method of aspect 1, wherein the multi-tap convolution matrix parameters represent a finite impulse response (FIR) filter. [Aspect 3] 3. The method of any one of the preceding aspects, wherein the fundamental signal is divided into a series of time segments and transformation parameters are provided for each time segment. Aspect 4 4. The method of any one of aspects 1-3, wherein the multi-tap convolution matrix parameters include at least one coefficient that is complex valued. Aspect 5 The step of providing the base signal includes determining the base signal from the audio channels or objects using first rendering parameters; The method includes determining a desired output signal for the second presentation from the audio channels or objects using second rendering parameters; and providing the transformation parameters includes determining the transformation parameters by minimizing a deviation of the output signal from the desired output signal. The method of any one of embodiments 1 to 4. Aspect 6 The step of providing the transformation parameters comprises: Determine subband domain basis signals for B frequency bands using the encoder filter bank; determining a desired output signal in a subband domain for the B frequency bands using the encoder filter bank; determining a same set of multi-tap convolution matrix parameters for at least two adjacent frequency bands of the B frequency bands. The method of embodiment 5. Aspect 7 the encoder filter bank includes a hybrid filter bank providing a low frequency band of the B frequency bands with a higher frequency resolution than a high frequency band of the B frequency bands; the at least two adjacent frequency bands are low frequency bands; The method of embodiment 6. Aspect 8 8. The method of embodiment 7, wherein the step of providing the transformation parameters comprises determining the same real-valued transformation parameters for at least two adjacent high frequency bands. Aspect 9 the at least two frequency bands include a lower frequency band and a higher frequency band; the transformation parameters specified for the higher frequency band do not modify the signal phase of the fundamental signal; the transformation parameters specified for the lower frequency band modify a signal phase of the fundamental signal; The method of any one of embodiments 1 to 8. Aspect 10 10. The method of any one of aspects 1-9, wherein the multi-tap convolution matrix parameters are utilized to process a low frequency band. Aspect 11 11. The method of any one of aspects 1-10, wherein the base signal and the transformation parameters are combined to form the data stream. Aspect 12 the transformation parameters include high-frequency audio matrix coefficients for a matrix operation of a high-frequency portion of the fundamental signal; 12. The method of any one of embodiments 1 to 11. Aspect 13 13. The method of claim 12, wherein for an intermediate frequency portion of the high frequency portion of the fundamental signal, the matrix operation includes complex-valued transformation parameters. Aspect 14 1. A decoder for decoding an encoded audio signal, the encoded audio signal comprising: a first presentation including audio fundamental signals intended for reproduction of the encoded audio signal in a first audio presentation format; and transformation parameters for transforming the audio fundamental signals in the first presentation format into output signals in a second presentation format, the transformation parameters comprising high-frequency audio transformation parameters and low-frequency audio transformation parameters, the low-frequency transformation parameters comprising multi-tap convolution matrix parameters, the first presentation format being for loudspeaker reproduction and the second presentation format being for headphone reproduction, or the first presentation format being for headphone reproduction and the second presentation format being for loudspeaker reproduction; The decoder in question: a first separation unit for separating the audio basis signals and the transformation parameters; a matrix multiplication unit for applying the multi-tap convolution matrix parameters to low frequency components of the audio fundamental signal; and for applying the convolution to the low frequency components to generate convolved low frequency components; a scalar multiplication unit for applying the high frequency audio transformation parameters to high frequency components of the audio fundamental signal to generate scalar high frequency components; an output filter bank that combines the convolved low frequency components and the scalar high frequency components to generate a time domain output signal in the second presentation format. decoder. Aspect 15 15. The decoder of claim 14, wherein the matrix multiplication unit modifies a phase of the low frequency components of the audio base signal. Aspect 16 16. The decoder of claim 14 or 15, wherein the multi-tap convolution matrix transform parameters are complex values. Aspect 17 17. The decoder of any one of aspects 14 to 16, wherein the high-frequency audio transform parameters are complex-valued. Aspect 18 20. The decoder of embodiment 17, wherein the transform parameters further include real-valued, high-frequency audio transform parameters. Aspect 19 19. The decoder of any one of aspects 14-18, further comprising a filter for separating the audio fundamental signal into the low frequency component and the high frequency component. Aspect 20 1. A method of decoding an encoded audio signal, the encoded audio signal comprising: a first presentation including audio fundamental signals intended for reproduction of the encoded audio signal in a first audio presentation format; and transformation parameters for transforming the audio fundamental signals in the first presentation format into output signals in a second presentation format, the transformation parameters including high-frequency audio transformation parameters and low-frequency audio transformation parameters, the low-frequency transformation parameters including multi-tap convolution matrix parameters, the first presentation format being for loudspeaker playback and the second presentation format being for headphone playback, or the first presentation format being for headphone playback and the second presentation format being for loudspeaker playback; The method involves: convolving low frequency components of the audio fundamental signal with the low frequency transformation parameters to generate convolved low frequency components; multiplying high frequency components of the audio fundamental signal with the high frequency transformation parameter to generate multiplied high frequency components; combining the convolved low frequency components and the multiplied high frequency components to generate output audio signal frequency components for the second presentation format. method. Aspect 21 The encoded audio signal includes a plurality of time segments, the method further comprising: interpolating transform parameters of a plurality of temporal segments of the encoded audio signal to generate interpolated transform parameters, including interpolated low frequency audio transform parameters; and convolving a plurality of time segments of the low frequency components of the audio fundamental signal with the interpolated low frequency audio transform parameters to generate a plurality of time segments of the convolved low frequency components. The method of embodiment 20. Aspect 22 The transformation parameters of the encoded audio signal are time-varying, and the convolution of low frequency components of the audio fundamental signal comprises: convolving the low frequency components of the audio fundamental signal with the low frequency transformation parameters for a plurality of time segments to generate a plurality of sets of intermediate convolved low frequency components; and interpolating the plurality of sets of intermediate convolved low frequency components to generate the convolved low frequency components. The method of embodiment 20. Aspect 23 23. The method of claim 20 or 22, wherein the interpolation utilizes an overlap-add method of the multiple sets of intermediate convolved low frequency components. Aspect 24 24. The method of any one of aspects 20-23, further comprising filtering the audio fundamental signal into the low frequency components and the high frequency components. Aspect 25 A computer-readable non-transitory storage medium comprising program instructions for a computer to operate according to the method of any one of embodiments 1 to 13 and 20 to 24.
Claims
1. receiving, by a decoder, an input bitstream; splitting the input bitstream into a base signal bitstream and transformation parameter data; decoding, by a base signal decoder, the base signal bitstream to provide a filterbank domain signal having a plurality of subbands; determining complex-valued matrix coefficients and real-valued matrix coefficients in response to the transformation parameter data; applying, by a first matrix multiplication unit, the complex-valued matrix coefficients to a first subband of the filterbank domain signal; applying, by a second matrix multiplication unit, the real-valued matrix coefficients to one or more remaining subbands of the filterbank domain signal; and converting the output signal from the matrix multiplication unit into a time domain output by a synthesis filter bank. method.
2. The method of claim 1 , wherein the complex-valued matrix coefficients represent a finite impulse response (FIR) filter.
3. 2. The method of claim 1, wherein the base signal bitstream comprises a series of time segments, and at least a portion of the transformation parameter data is provided for those time segments.
4. The method of claim 1 , wherein the obtaining of the base signal bitstream comprises a plurality of audio channels.
5. 2. The method of claim 1, wherein the transformation parameter data includes parameters for minimizing deviation of the output signal from a desired output signal.
6. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: receiving, by a decoder, an input bitstream; splitting the input bitstream into a base signal bitstream and transformation parameter data; decoding, by a base signal decoder, the base signal bitstream to provide a filterbank domain signal having a plurality of subbands; determining complex-valued matrix coefficients and real-valued matrix coefficients in response to the transformation parameter data; applying, by a first matrix multiplication unit, the complex-valued matrix coefficients to a first subband of the filterbank domain signal; applying, by a second matrix multiplication unit, the real-valued matrix coefficients to one or more remaining subbands of the filterbank domain signal; and converting the output signal from the matrix multiplication unit into a time domain output by a synthesis filter bank. Medium.
7. one or more processors; a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations; and wherein the operation comprises: receiving, by a decoder, an input bitstream; splitting the input bitstream into a base signal bitstream and transformation parameter data; decoding, by a base signal decoder, the base signal bitstream to provide a filterbank domain signal having a plurality of subbands; determining complex-valued matrix coefficients and real-valued matrix coefficients in response to the transformation parameter data; applying, by a first matrix multiplication unit, the complex-valued matrix coefficients to a first subband of the filterbank domain signal; applying, by a second matrix multiplication unit, the real-valued matrix coefficients to one or more remaining subbands of the filterbank domain signal; and converting the output signal from the matrix multiplication unit into a time domain output by a synthesis filter bank. system.
Citation Information
Patent Citations
audio decoding
JP2009536360A
Method and apparatus for generating binaural audio signals
JP2010541510A
