Apparatus and method for converting an audio stream - Patents.com

Abstract

1. An apparatus for converting an audio stream having two or more channels to another representation, comprising: means for converting the audio stream in a signal adaptive manner depending on the one or more parameters; and means for deriving one or more parameters describing an acoustic or psychoacoustic model of the audio stream, the parameters including at least information regarding a DOA, wherein the one or more parameters are derived from the audio stream.

Description

[Technical field]

[0001] An embodiment of the invention relates to an apparatus for converting an audio stream having two or more channels into another representation. Further embodiments relate to a corresponding method and a corresponding computer program.

[0002] Further embodiments relate to an apparatus for converting an audio stream in a directional audio coding system. Further embodiments relate to corresponding methods and computer program products.

[0003] Further embodiments relate to an encoder comprising one of the above defined devices to a corresponding method for encoding, and to a decoder comprising one of the above discussed devices and a corresponding method for decoding. Preferred embodiments relate generally to the technical field of compression of audio channels by prediction based on acoustic model parameters. [Background technology]

[0004] The prior art related to the embodiment is as follows: Directional Audio Coding (DirAC) A metadata-assisted EVS codec for spatial audio presented in the context of the 3GPP standardization body; It is derived mainly from two previously known audio coding methods.

[0005] A brief summary of both concepts follows. Directional Audio Coding DirAC is a parametric technique for encoding and reproducing spatial sound fields [1,2,3,4]. The justification for one source direction of arrival (DOA) and interaural coherence [4] is based on the psychoacoustic argument that human listeners can only process two cues per critical band [4]. It is therefore sufficient to reproduce two streams per critical band: a directional stream containing the coherent channel signal from one point source from a given direction, and a diffuse stream containing the incoherent diffuse signal [4].

[0006] The analysis stage on the encoder side is shown in the diagram of Fig. 1a. Fig. 1 shows an encoder claim with a bandpass filter 11 on the input side and two entities 12 and 13 for determining energy and intensity. Based on the energy and intensity, the spread is determined by a spread determiner 14, which can use for example a time average. The output of the spread determiner 14 is Φ. Based on the intensity, the direction (Azi and Ele) is determined by a direction determiner 15. The information of Φ, Azi and Ele is output as metadata.

[0007] The input is provided in the form of four B-format channel signals, which are analyzed with a filter bank (FB). For each band of this FB, the DOA of a point source and the divergence are extracted [3,4]. These two parameters, DOA expressed by azimuth and elevation angles, and divergence for each band comprise the DirAC metadata [3,4], the efficient compression of which has been addressed in Ref. [3,4,5].

[0008] As shown in Fig. 1b, from the B-format signal and the metadata, the two aforementioned streams are synthesized. The decoder 20 comprises a processor path 21 for processing the metadata ψ and a processing path 22 for processing the metadata Azi and Ele. Furthermore, the decoder 20 comprises a processing path 23 including a band-pass filter and a virtual microphone for processing the B-format signal (see Mic signal (W, X, Y, Z)). All three processing paths 21-23 are then combined by an entity 24 including a decorrelator to output the speaker channel signals. If it is desired to decode two loudspeakers, a directional stream can be obtained by panning a point source in the direction coded in the DirAC parameters [3, 4], for example using vector-based amplitude panning (VBAP) [6]. For the diffuse stream, it is necessary to feed the loudspeakers with uncorrelated signals [4].

[0009] Fig. 2 shows the DirAC encoder from (5). It includes a DirAC analysis 31 followed by a spatial metadata encoder 32. The DirAC analysis processes the B format and outputs diffuseness and directional parameters to the spatial meta encoder 32. In parallel, the B format is performed by an entity for beamforming / signal selection (see reference number 33). The output of the entity 33 is then processed by the EVS encoder 34. Fig. 3 shows the corresponding DirAC decoder. The DirAC decoder of Fig. 3 comprises a spatial metadata decoder 41 and an EVS decoder 42. Both decoded signals are then used by the DirAC synthesis 43 to output the speaker channels or the FOA / HOA.

[0010] An extension of this system to Higher Order Ambisonics (HOA) with multi-channel (MC) or object-based audio is presented by Fuchs et al. [5]. There, the authors propose to perform additional processing of the B-format input signal to select the appropriate downmix channel or to find the appropriate beam of the virtual microphone to capture the transport stream, as shown in Fig. 2, reference 33. These transport streams are then encoded using an EVS encoder. On the decoder side, a corresponding decoder is applied. The signal paths in the encoder and decoder can be seen in Figs. 2 and 3. Furthermore, an advanced encoding scheme (see Fig. 2, reference 32) is presented to ensure the transmission of metadata at the lowest possible bit rate without perceptible quality loss [5]. In contrast to the system of reference [2], the decoder output signal can be generated again in HOA format so that any renderer can be employed to obtain the headphone or loudspeaker signal.

[0011] Therefore, the stream of data transmitted from the encoder to the decoder must contain both an EVS bitstream and a DirAC metadata stream, and care must be taken to find an optimal distribution of the available bits between the metadata and the individual EVS coded channels of the downmix.

[0012] Metadata-assisted EVS codec An alternative approach to encoding and playback of spatial audio recordings that has been previously proposed in standardization bodies is the metadata-assisted EVS coder [7], also called spatial audio reconstruction (SPAR) [7]. Figure 4 shows the signal path from the encoder input to the decoder output. Similar to DirAC, the SPAR encoder extracts metadata and a downmix from the FOA or HOA input signal [7]. This processing is again performed in the FB domain [7].

[0013] Fig. 4 shows a metadata-assisted EVS coder for spatial audio as shown in [7]. The EVS coder 50 comprises a content ingestion engine 51 that receives M objects, HOA scenes and channels and outputs the M objects together with Nth Ambisonics channel to a SPAR encoder 52. The SPAR encoder comprises a downmix and WXYZ engine compression conversion. The SPAR metadata and FOA data are output together with the object metadata to an EVS and metadata encoder 53. This data stream is then processed by a mode switch 54 that delivers high and low immersion quality data (SPAR metadata and object metadata with FOA and predictive metadata) to the respective coders. The high immersion coders are marked with reference numbers 55a and 55b and the low immersion coders are marked with reference numbers 56a and 56b.

[0014] The downmix is ​​performed such that energy compression of the FOA signal is achieved (see Fig. 4) and then encoded using an EVS mono encoder with up to four instances. These steps are similar to the beamforming or channel selection and EVS encoding steps of DirAC in Fig. 2. On the decoder side, the FOA signal is reconstructed from the compressed downmix channels and metadata, including predictor coefficients (PCs) [7]. According to the pseudocode in [7], this is achieved by band-wise multiplication of a smaller number of channels with a gain matrix. The HOA signal can also be reconstructed using the transmitted SPAR metadata [7]. The metadata stream is compressed for transport by Huffman coding [7].

[0015] Head Tracking in Spatial Audio Reproduction When a spatial sound scene is played over headphones, it is necessary to track the listener's head movements and rotate the sound scene accordingly to create a consistent and realistic experience. To this end, a widely adopted technique is to rotate the scene in the Ambisonics domain by pre-multiplying the vectors of the channel signals with a rotation matrix [8,9,0]. This rotation matrix is ​​typically computed by the method in reference

[11] . An alternative approach is to render the output signal to virtual speakers and perform the rotation by amplitude panning [9,6]. Summary of the Invention

[0016] All of the above solutions have drawbacks, as explained below, the remedy for which is part of this invention.

[0017] In both of the above referenced systems, some key challenges are (i) selecting the best matching channels of the input signal for transmission over the EVS, (ii) finding a representation of these channels that reduces the redundancy between them, and (iii) distributing the available bitrate between the metadata and the individual EVS encoded audio streams such that the best possible perceptual quality is achieved. Since these decisions depend heavily on the signal characteristics, signal adaptation processing must be implemented.

[0018] It is an object of the present invention to enable a coding technique in which the amount of additional metadata required to enable reconstruction of the downmix channels is reduced while the coding efficiency is increased.

[0019] An embodiment of the present invention provides an apparatus for converting an audio stream having two or more channels into another representation. The apparatus comprises a converting means and a deriving means and / or a receiving means. The converting means is configured to convert the audio stream in a signal adaptive manner depending on one or more parameters. The deriving means is configured to derive one or more parameters describing an acoustic or psychoacoustic model of the audio stream (signal). It is noted that on the decoder side, prediction parameters can be received (see receiving means). Said parameters include at least information about DOA (Direction of Arrival), where one or more parameters may be derived from the audio stream, for example on the encoder side (or just received, for example on the decoder side).

[0020] According to a further embodiment, the means for deriving calculates the prediction coefficients based on a covariance matrix or on parameters of the acoustic signal or is configured to calculate the prediction coefficients.

[0021] According to an embodiment, the means for deriving is configured to calculate the covariance matrix from a model / acoustic model or in general based on the DOA or additional spreading coefficients or energy ratios.

[0022] It should be noted that according to an embodiment, the one or more parameters include predictive parameters.

[0023] The embodiments of the present invention are based on the principle that prediction coefficients at both the encoder side and the decoder side can be approximated from a model such as an acoustic model or acoustic model parameters. In a directional audio coding system, these parameters are always present at the decoder side, so that no additional metadata bits are transmitted for the prediction. Thus, the amount of additional metadata required to enable reconstruction of the downmix channels at the decoder side is significantly reduced compared to a naive implementation of the prediction. In other words, this means that the combination of deriving one or more parameters describing an acoustic model and transforming the audio stream in a signal adaptive manner provides a way to compress the downmix channels in a directional audio coding system or other applications through the application of inter-channel prediction based on an acoustic model of the input signal.

[0024] In the above embodiments, mainly DOA parameters are described. According to further embodiments, further spreadness information / spreading coefficients can be used. Thus, said parameters used by the transforming means and derived by the deriving means can include spreading coefficients or information on one or more DOAs or energy ratios. For example, one or more parameters are derived from the audio stream itself.

[0025] Regarding the prediction coefficients, according to a further embodiment, the prediction coefficients are real or complex spherical harmonic functions Y l,m Note that the calculation is based on

[0026] With regard to the covariance matrix, it is noted that according to a further embodiment, the means for deriving is configured to calculate the covariance matrix based on information about the diffusivity, the spherical harmonics and the time-dependent scalar-valued signal. For example, the calculation is It may be based on the formula in TIFF2025505460000002.tif779, where: TIFF2025505460000003.tif77 is the frequency and index TIFF2025505460000004.tif72 and where t is a spherical harmonic function with t = 0 and s(t) is a time-dependent scalar-valued signal. According to a further embodiment, the calculation may be, for example, It may be based on signal energy by using the formula in TIFF2025505460000006.tif1384, where TIFF2025505460000007.tif73 shows the signal energy.

[0027] Alternatively or additionally, The formula TIFF2025505460000008.tif781 may be used, where: TIFF2025505460000009.tif73 is the signal energy as well.

[0028] Alternatively or additionally, The formula TIFF2025505460000010.tif1387 may be used, and similarly for the y and z channels.

[0029] According to an embodiment, the energy TIFF2025505460000011.tif73 is calculated directly from the audio stream. Alternatively or additionally, the energy TIFF2025505460000012.tif73 is estimated from a model of the signal.

[0030] According to a further aspect, the audio stream is pre-processed by a parameter estimator or a parameter estimator comprising a metadata encoder or a metadata decoder and / or by an analysis filter bank.

[0031] According to a further embodiment, the input audio stream is a higher order Ambisonics signal and the parameter estimation is based on all or a subset of these input channels. For example, this subset may include the first order channels. Alternatively, this subset may consist of planar channels of any order or any other choice of channels.

[0032] As mentioned above, an embodiment provides an encoder comprising the above mentioned device. A further embodiment provides a decoder comprising the above mentioned device. On the encoder side, the device may comprise a converting means configured to perform a mixing, e.g. a downmix of the audio stream. On the decoder side, the converting means is configured to perform a mixing, e.g. an upmix or an upmix generation of the audio stream.

[0033] The above mentioned device may also be used for transforming an audio stream in a directional audio coding system. According to an embodiment, the device comprises means for transforming and means for deriving. The means for transforming are configured to transform the audio stream in a signal adaptive manner dependent on one or more acoustic model parameters. The means for deriving are configured to derive one or more acoustic model parameters of a model of the audio stream, parameterized by DOA and / or diffuseness and / or energy ratio parameters. Said acoustic model parameters are transmitted to recover all channels of the audio stream and include at least information on the DOA. The transmitted audio stream is derived by transforming all or a subset of the channels of the audio stream. According to an embodiment, the transmitted parameters are quantized before transmission. According to an embodiment, the parameters are dequantized after transmission. According to a further embodiment, the parameters can be smoothed over time. According to a further embodiment, the quantized parameters may be compressed by entropy coding.

[0034] Concerning the transformation, it is noted that according to a further embodiment, the transformation is calculated such that the correlation between the transport channels is reduced. According to an embodiment, the inter-channel covariance matrix of the input of the audio stream is estimated from a model of the signal of the audio stream. For example, a transformation matrix is ​​derived from the covariance matrix of the model of the audio stream signal. The covariance matrix can be calculated using different methods for different frequency bands. Concerning the transformation performed by the means for transforming, it is noted that according to an embodiment, at least one of the transformation methods is a multiplication of a vector of the audio channels with a constant matrix. According to another embodiment, the transformation method uses a prediction based on the inter-channel covariance matrix of the audio signal vector. According to another embodiment, at least one of the transformation methods uses a prediction based on the inter-channel covariance matrix of a model signal described by DOA and / or spreading coefficients and / or energy ratios.

[0035] According to another embodiment, and mainly applicable to an apparatus for transforming an audio stream in a directional audio coding system, the scene to be encoded by the audio stream (signal) is A vector of audio transport channel signals is premultiplied by a rotation matrix; the model parameters are transformed in response to a transformation of the transport channel signal; and The non-transport channels of the output signal are reconstructed using the parameters of the transformed model. It can be rotated in such a way.

[0036] As mentioned above, the device can be applied to an encoder and a decoder. Another embodiment provides a system comprising an encoder and a decoder, the encoder and the decoder being configured to independently calculate prediction matrices and / or downmix and / or upmix matrices from estimates of an acoustic model or transformation parameters.

[0037] According to a further embodiment, the above-mentioned technique can be implemented by a method. Another embodiment is a method for converting an audio stream having two or more channels into another representation, comprising the steps of: - deriving or receiving one or more parameters describing an acoustic or psychoacoustic model of the audio stream from the audio stream, said parameters including at least information regarding the DOA; - transforming the audio stream in a signal adaptive manner dependent on one or more parameters; The present invention provides a method comprising:

[0038] Another embodiment is a method for transforming an audio stream in a directional audio coding system, comprising: - deriving one or more acoustic model parameters of a model of the audio stream (audio stream parameterized by DOA and diffuseness or energy ratio parameters), where the acoustic model parameters are derived by recovering all channels of the input audio stream and transmitting the transmitted audio stream including at least information about the DOA, and transforming all or a subset of the channels of the audio stream; - transforming the audio stream in a signal adaptive manner dependent on one or more acoustic model parameters; The present invention provides a method comprising:

[0039] According to a further embodiment the method may be computer implemented. Thus, an embodiment provides a computer program for performing the method according to the above disclosure when executed on a computer.

[0040] Embodiments of the present invention are described below with reference to the accompanying drawings. [Brief description of the drawings]

[0041] [Figure 1a] FIG. 1 is a schematic diagram of DirAC analysis and synthesis. [Figure 1b] FIG. 1 is a schematic diagram of DirAC analysis and synthesis. [Diagram 2] 1 is a schematic diagram of a DirAC encoder. [Diagram 3] FIG. 1 is a schematic diagram of a DirAC decoder. [Figure 4] FIG. 1 is a schematic diagram of a metadata-assisted EVS for spatial audio. [Figure 5a] FIG. 13 shows the covariance matrix elements for one frequency band as a function of frame number (time) for a signal containing only one panned point source; the model matrix and the exact matrix match very well (to illustrate the embodiment). [Figure 5b] FIG. 13 shows covariance matrix elements for one frequency band as a function of frame number (time) for a signal from an EigenMike recording (where the model and exact matrices show good quality agreement), to illustrate an embodiment. [Figure 6] 1 is a schematic diagram of an apparatus for converting an audio stream (as part of a decoder and / or encoder) according to a basic embodiment; [Figure 7a] FIG. 2 is a schematic diagram of a DirAC system with predictive coding of transport channels according to a further embodiment; [Figure 7b] FIG. 2 is a schematic diagram of a DirAC system with predictive coding of transport channels according to a further embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings, in which the same reference numbers are used for objects having the same or similar functions, and the descriptions thereof are interchangeable or mutually applicable.

[0043] Before describing the embodiments of the present invention, a description of some features of the present invention will be given separately.

[0044] Channel Compression It is known that for the compression of transport channels, the optimal decorrelation, and therefore the energy compression, can be obtained by the Karhunen-Loeve Transform (KLT) (see, for example,

[12] ). The KLT transforms the signal vector into a base of eigenvectors of the inter-channel covariance matrix. For a B-format input signal with the format TIFF2025505460000013.tif17136, the inter-channel covariance matrix The elements of TIFF2025505460000014.tif21136 are The transformation matrix is ​​given by TIFF2025505460000015.tif9136 and similar for other channel combinations. In the KLT, matrix 2 is diagonalized, completely removing all inter-channel correlations and thus obtaining the least redundant representation of the signal. However, there are two difficulties that prevent the implementation of the KLT in most real-world systems: the computational complexity of the required eigenvector calculations and the metadata bit usage for the transmission of the resulting transformation matrix are often considered too high.

[0045] prediction As a compromise, one can only remove the correlation between x, y, and z and the w channel via the prediction matrix. TIFF2025505460000016.tif17136This method does not require matrix diagonalization, and the three prediction coefficients Only TIFF2025505460000017.tif712 is transmitted. Depending on the frame length and signal characteristics, the amount of metadata for this technique can still be significant. Our experiments have shown this to be on the order of 10kbps. This is particularly noteworthy since these metadata are transmitted along with the metadata required for the DirAC system itself, increasing the overall bit requirements.

[0046] This naturally raises the question of how these two metadata streams are connected. The invention described below clarifies the link between the predictions aimed at compression of DirAC or SPAR transport channels and the model parameters transmitted in DirAC, allowing for decoder-side reconstruction of the full HOA input signal. We provide a route to reuse of metadata already transmitted as part of the DirAC system for compression of transport channels. Our method can thus improve the perceptual quality of DirAC compared to passive downmix with static selection of transport channels, while avoiding additional metadata transmission.

[0047] Head Tracking Both approaches to scene rotation as mentioned above have significant drawbacks. In the former case, the computational complexity is very high due to the sample-by-sample matrix multiplication of the signals. In the latter case, the quality is not optimal [9]. It is therefore desirable to reduce the complexity of the former method without excessively compromising the quality. The present invention provides a route to apply rotations in a low-dimensional space. Within the framework of the two aforementioned systems for parametric coding of spatial audio, this can be achieved by combining a rotation of a subset of channels in the Ambisonics domain with an appropriate transformation in the metadata domain.

[0048] It has been established above that transport channel compression can be achieved by reducing correlation via a transform derived from the covariance matrix. The following description shows how such a transform can be obtained independently at both the encoder and decoder side from readily available DirAC model parameters or general acoustic model parameters.

[0049] According to an embodiment, the covariance matrix may be determined from the model signal.

[0050] This can be considered as one of the parameter bands of directional audio coding (see above). For simplicity, the notation omits the frequency band index. First, we focus on the non-diffuse directional part of the signal. TIFF2025505460000018.tif13136, with the compound angle variable The direction of arrival (DOA) of the sound from a point source on the unit sphere specified by TIFF2025505460000019.tif726. The sound pressure from this source on the unit sphere is The time-dependent signal is calculated by the formula in TIFF2025505460000020.tif5136TIFF2025505460000021.tif5136 TIFF2025505460000022.tif78 and Dirac distribution on a sphere The resulting image is given in TIFF2025505460000023.tif79.

[0051] We consider the directional part from a panned point source. Consider a B-format or First Order Ambisonics (FOA) signal that contains a directional part with no correlation between the individual channels and a non-correlated diffuse part. For this reason, the signal vector of the directional part is TIFF2025505460000025.tif21136, where: TIFF2025505460000026.tif77 is a spherical harmonic function with order and exponent numbers l and m.

[0052] This result can be easily read off from the expansion of the seven Dirac functions up to first order in spherical harmonics (see also

[13] ). TIFF2025505460000027.tif9136 The full B format signal, together with the diffusion part, It will look like TIFF2025505460000028.tif29136.

[0053] Diffusion part TIFF2025505460000029.tif712 components The prefactor of TIFF2025505460000030.tif135 is It results from the normalization of the signal.

[0054] Given this model signal, we can now easily evaluate the covariance matrix elements. For the off-diagonal matrix elements, we TIFF2025505460000031.tif5136. Here, the integral The terms containing integers over 1323 vanish because the diffuse components are assumed to exhibit no correlation with s(t) or with each other. Using TIFF2025505460000033.tif731, this can be calculated as follows: TIFF2025505460000034.tif5136 Diagonal matrix elements TIFF2025505460000035.tif710 is TIFF2025505460000036.tif5136, and the diffusion energy TIFF2025505460000037.tif79 is defined as being similar to directional energy. The other diagonal matrix elements follow similarly.

[0055] Figures 5a and 5b show the covariance matrix elements as a function of time for a signal-panned point source and an EigenMike recording, respectively. In the case of the point source (Figure 5a), the agreement is very accurate, as can be seen for the comparison of the DirAC model signal (dashed blue line) with the exact calculated signal (solid red line). In the case of the EigenMike recording, the model captures the signal features qualitatively.

[0056] Prediction in DirAC Using equations 4, 12, and 13, the direct and diffuse energies TIFF2025505460000038.tif78 and If we express TIFF2025505460000039.tif79 by the total signal energy E, the remaining parameters are the angles always present in the DirAC decoder. TIFF2025505460000040.tif76 and only the spread or energy ratio. Thus, the need to transmit additional prediction coefficients can be completely avoided.

[0057] Alternatively, the model can be enabled only for a subset of the frequency bands. For other bands, the prediction coefficients are calculated from the exact covariance matrix and transmitted explicitly. This can be useful when very accurate predictions are needed for the most perceptually relevant frequencies. It is often desirable to reproduce the input signal more accurately at lower frequencies, e.g. below 2 kHz. The choice of crossover crossover frequency can be motivated by two different opinions:

[0058] First, it is known that sound source localization depends on different mechanisms for low and high frequencies

[14] . While interaural phase difference (IPD) is evaluated at low frequencies, interaural level difference (ILD) dominates for sound source localization at higher frequencies

[14] . It is therefore more important to achieve high accuracy of prediction and more accurate reproduction of phase at lower frequencies. As a result, one may wish to resort to more demanding but more accurate transmission of prediction parameters for lower frequencies.

[0059] Second, the resulting perceptual audio coder for the downmix channel often reproduces the low frequency band more accurately than the high frequency band, due to the above arguments. For example, at low bit rates, the higher frequencies can be quantized to zero and restored from the lower frequency copy

[15] . Therefore, it may be desirable to implement the crossover frequencies according to the internal parameters of the core coder employed to provide a consistent quality across the system.

[0060] The signal path of the resulting DirAC system is shown in Figure 7a / b. The main improvement compared to the systems of Figures 2 and 3 presented earlier is the adaptive compression of the transport channels using the acoustic model parameters. After the usual estimation of the DOA angles and the diffusivity in each band, the model covariance matrix and the prediction coefficients are calculated according to equations 12 to 14. The input channels are then mixed and coded using EVS. At the decoder side, the prediction coefficients are calculated again from the transmitted model parameters and the transformation is inverted. The non-transport channels are then reconstructed by the DirAC decoder as described above.

[0061] Low-complexity head tracking TIFF2025505460000041.tif720, order Let L be the vector of output channel signals in HOA of TIFF2025505460000042.tif73. Therefore, the dimension of this vector is N=(L+1) 2 To perform a scene rotation in the conventional way, the signal is first reconstructed with a DirAC or SPAR decoder, and a rotation matrix of size N × N is applied to each sample of the signal. Multiplied by TIFF2025505460000043.tif716.

[0062] Where: Let TIFF2025505460000044.tif717 be the signal vector of the transported channel after applying the inverse transformation as shown in FIG. 7, reference numeral 110d. The vector The dimensions of TIFF2025505460000045.tif717 are: Most of the channels in TIFF2025505460000046.tif721 are reconstructed parametrically, so M <Nである。ここで、 All channels in TIFF2025505460000047.tif717 are in order TIFF2025505460000048.tif713 belongs to the basis function (spherical harmonic function) of order Select TIFF2025505460000049.tif75 and click TIFF2025505460000050.tif75 for all channels Applying rotation via premultiplication of TIFF2025505460000051.tif717. Thus, All channels in TIFF2025505460000052.tif712 will be unaffected by the rotation and the signal vector will be left in an inconsistent state.

[0063] The key novelty of our invention is that The solution is to take advantage of the properties of TIFF2025505460000053.tif716: it is block diagonal, each of which belongs to a particular order l, The matrix elements for TIFF2025505460000054.tif711 are: is identical to the same rotation applied to any vector in TIFF2025505460000055.tif76

[11] . Thus, Before reconstructing the channel, which is TIFF2025505460000056.tif712, TIFF2025505460000057.tif716 We can apply the blocks in TIFF2025505460000058.tif711 to the DOA vector 5. As a result, these channels are reconstructed including scene rotation and with full dimensionality. The need to perform matrix multiplications in TIFF2025505460000059.tif74 can be avoided, significantly reducing computational complexity.

[0064] The above-mentioned technique can be used by an apparatus as shown in Fig. 6. The apparatus 100 may be a part of an encoder or a decoder, and includes at least a means for converting 110 and a means for deriving 120. The apparatus 100 is applicable to the encoder and decoder sides. First, the function of the apparatus on the encoder side will be described.

[0065] Let us assume that the device 100, which is part of an encoder, receives a HOA representation. This representation is provided to the entities 110 and 120. A pre-processing of the HOA signal, for example by an analysis filter bank or a DirAC parameter estimator, is performed (not shown). One or more parameters describing an acoustic or psychoacoustic model of the input audio stream HOA. For example, they may include information on at least the direction of arrival (DOA), or optionally on the energy ratio edge of the spread or insertion.

[0066] Entity 120 performs the derivation of one or more parameters, eg prediction parameters / coefficients.

[0067] The diffuseness and / or the direction of arrival may be parameters of the acoustic model mentioned above. The prediction coefficients may be calculated by the entity 120 on the basis of the acoustic model or on the basis of parameters describing the acoustic model. According to further embodiments, an intermediate step may be used. The prediction coefficients according to further embodiments are calculated on the basis of a covariance matrix which is also calculated by the means for deriving 120, for example from the acoustic model. Often such a covariance matrix is ​​calculated on the basis of information on the diffuseness, spherical harmonics and / or time-dependent scalar-valued signals. For example, the formula In TIFF2025505460000060.tif1383, TIFF2025505460000061.tif78 is the order and exponent TIFF2025505460000062.tif72 and where s(t) is a spherical harmonic function with TIFF2025505460000063.tif74, and s(t) is a time-dependent scalar-valued signal. The description of the calculation of the covariance matrix has been given in great detail above. According to further embodiments, the additional calculation methods mentioned above can be used.

[0068] This means that, according to an embodiment, entity 120 performs the following calculations: Extraction of acoustic or psychoacoustic model parameters such as DOA or diffuseness from the audio stream HOA. Derive the covariance matrix based on the set parameters of the acoustic model Calculation of predicted parameters based on the covariance matrix, which predicted parameters can be used by another entity, for example entity 110. The output of entity 120 is therefore parameters, in particular predicted parameters, which are forwarded to entity 110.

[0069] Entity 110 is adapted to perform a transformation, for example a downmix generation, which is based on an input signal, here the HOA signal, however in this case the transformation is applied in a signal-adaptive manner that depends on one or more parameters as derived by entity 120.

[0070] Due to the novel approach, where parameters, e.g. inter-channel prediction coefficients, are derived from an acoustic signal model or parameters of an acoustic signal model, it is possible to perform transformations such as mixing / downmixing in a signal-adaptive manner. For example, this principle can be used to develop an extension of the DirAC system for spatial audio signals. This extension improves the quality compared to a static selection of a subset of channels of the HOA input signal as transport channels. Furthermore, this reduces the metadata bit usage compared to previous approaches to signal-adaptive transformations that reduce inter-channel correlation. The metadata savings can in turn free up more bits for the EVS bitstream, further improving the perceptual quality of the system. The additional computational complexity is negligible. These advantages result directly from the derivation of the mathematical connection between the signal model considered in the DirAC system and the prediction coefficients that are usually transmitted as side information in predictive coding schemes.

[0071] Although the principle is described in the context of an encoder, it can also be applied on the decoder side. On the decoder side, the device also comprises a transforming means and a means (see reference number 120) for deriving one or more parameters used in the transforming means 110. For example, the decoder receives metadata, including information on an acoustic / psychoacoustic model or parameters of the acoustic / psychoacoustic model (typically parameters allowing to determine prediction coefficients), together with a coded signal, such as an EVS bitstream. The EVS bitstream is provided to the transforming means 110, where the metadata is used by the deriving means 120. The deriving means 120 determines based on metadata parameters, including for example information on the DOA. For example, the determined parameters may be prediction parameters. It is noted that the metadata is derived from the audio stream, for example on the encoder side. These parameters / prediction parameters are then used by the transforming means 110. This transforming means 110 may be configured to perform an inverse transformation, such as an upmixing, to output a decoded signal, such as a FOA signal. This FOA signal can then be further processed to determine a HOA signal or a direct speaker signal. Further processing may include, for example, DirAC synthesis, which includes an analysis filterbank.

[0072] Note that the prediction coefficients may be calculated in the decoder in the same way as in the encoder, in which case the parameters may be pre-processed by a metadata decoder.

[0073] With reference to Figs. 7a and 7b, a detailed implementation of the above scheme at the decoder side and encoder side is described.

[0074] Fig. 7a shows an encoder 200 according to an embodiment with a central entity means 110e for transforming and means 120e for deriving one or more parameters, the means for transforming 110e being implemented as a downmix generation processing HOA data received from an input of the encoder 200. These data are processed taking into account parameters received from the entity 120e, for example prediction coefficients. The output of the downmix generation can be adapted to a bit allocation entity 212 and / or a synthesis filter bank 214. Both data streams processed by the entities 212 and 214 are forwarded to an EVS coder 216, which performs the coding and outputs the coded stream to a multiplexer 230.

[0075] Entity 120e comprises in this embodiment two entities, namely an entity for determining a model and / or a model covariance matrix marked with reference sign 121 and an entity for determining prediction coefficients marked with reference sign 122. According to an embodiment, entity 122 performs the determination of the covariance matrix based on one or more model parameters, such as for example the DOA. Entity 122 determines the prediction coefficients based for example on the covariance matrix.

[0076] Entity 120e may, according to a further embodiment, receive the HOA signal or a derivative of the HOA signal, for example preprocessed by DirAC parameter estimator 232 and analysis filterbank 231. The output of DirAC parameter estimator 232 may provide information about the direction of arrival (DOA as mentioned above). This information is then used by entity 120e, in particular entity 121. According to a further embodiment, the estimated parameters of entity 232 may also be used by metadata encoder 233, an encoded metadata stream being multiplexed by multiplexer 230 together with the EVS coded stream to output an encoded HOA signal / encoded audio stream.

[0077] Fig. 7b shows a decoder 300 with a demultiplexer 330 at the input according to an embodiment. The decoder 300 comprises central entities 120d and 110d. Entity 110d is arranged to perform a transformation, for example an inverse transformation, such as an upmixing of the signal received from the demultiplexer 330. The received input signal may be an EVS encoded signal, which is decoded by entity 316 and further processed by an analysis filter bank 314. The output of the transformer 110d is a FOA signal, which can then be further processed by DirAC synthesis taking into account the metadata received via the demultiplexer 330. For this purpose, the metadata path may comprise a metadata decoder 333.

[0078] A DirAC synthesis entity is marked by reference numeral 335, the output of which may be further processed by a synthesis filter bank 336 to output a HOA signal or a headphone / speaker signal.

[0079] The metadata, e.g. the metadata decoded by the metadata decoder 333, is used to determine the parameters obtained by the entity 120d, which in this case included two entities for determining the model / model covariance matrix marked by reference sign 121 and an entity for determining the prediction coefficients / general parameters (marked by reference sign 122). The output of the entity 120d is used for the transformation performed by the entity 110d.

[0080] Further aspects can be described below. The above-mentioned embodiments start from the assumption that an audio stream having more than one channel should be converted to another representation. The above-mentioned embodiments may also be applied for converting an audio stream in a directional audio coding system. Thus, the embodiments provide an apparatus and a method for converting an audio stream in a directional audio coding system, where: a) Acoustic model parameters are sent to recover all channels of the input signal; b) the parameters include at least one (or more) of DOA and diffuseness; c) the transmitted audio stream is derived by transforming all or a subset of the channels of the input signal; d) the transformation is derived from a model of the input signal parameterized by DOA and diffuseness parameters; e) This transform is computed in a signal-adaptive manner independently at both the encoder and decoder side.

[0081] According to an embodiment, the sound scheme comprises: a) a vector of transport channel signals is pre-multiplied by a rotation matrix in the appropriate domain; b) the model parameters and / or prediction coefficients are transformed in response to a transformation of the transport channel signal; and c) The non-transport channels of the output signal are reconstructed using these transformed model parameters and / or prediction coefficients. It can be rotated in this way.

[0082] In a general embodiment, a) the transform is derived from parameters describing an acoustic or psychoacoustic model of the signal, b) these parameters include at least one DOA and a spread; and c) The transform is computed in a signal-adaptive manner. Thus, the present invention relates to an apparatus and method for converting an audio stream having two or more channels into another representation.

[0083] According to further embodiments, the transformation is calculated such that correlation between the transport channels is reduced. For example, an inter-channel covariance matrix can be used, where the inter-channel covariance matrix of the input signal is estimated from a model of the signal. According to further embodiments, the transformation matrix is ​​derived from the covariance matrix of the model, such as a matrix calculated using different methods for different frequency bands.

[0084] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the most important method steps may be performed by such an apparatus.

[0085] The encoded audio signal of the present invention may be stored on a digital storage medium and may be transmitted over a transmission medium, such as the Internet, a wireless transmission medium or a wired transmission medium.

[0086] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to execute the respective method. The digital storage medium may therefore be computer readable.

[0087] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0088] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer. The program code can, for example, be stored on a machine readable carrier.

[0089] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0090] In other words, therefore, an embodiment of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0091] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0092] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0093] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0094] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0095] Further embodiments according to the invention comprise an apparatus or a system arranged to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0096] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0097] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and explanations of the embodiments herein.

[0098] References [1] Ville Pulkki. Directional audio coding in spatial sound reproduction and stereo upmixing. In Audio Engineering Society Conference: 28th International Conference: The Future of Audio Technology-Surround and Beyond, Jun 2006.

[0099] [2] Ville Pulkki. Spatial sound reproduction with directional audio coding. J. Audio Eng. Soc, 55(6):503-516, 2007. V. Pulkki, M-V. Laitinen, J. Vilkamo, J. Ahonen, T. Lokki, , and T. Pihlajamaeki. Directional audio coding - perception-based reproduction of spatial sound. 2009.

[0100] [3] Andrea Eichenseer, Srikanth Korse, Oliver Thiergart, Guillaume Fuchs, Markus Multrus, Stefan Bayer, Dominik Weckbecker, Juergen Herre, and Fabian Kuech. Parametric coding of object-based audio using directional audio coding. Internal document Fraunhofer IIS, 2020.

[0101] [4] Toni Hirvonen, Jukka Ahonen, and Ville Pulkki. Perceptual compression methods for metadata in directional audio coding applied to audiovisual teleconference. In Audio Engineering Society Convention 126, May 2009.

[0102] [5] Guillaume Fuchs, Juergen Herre, Fabian Kuech, Stefan Doehla, Markus Multrus, Oliver Thiergart, Oliver Wuebbolt, Florin Ghido, Stefan Bayer, and Wolfgang Jaegers. Apparatus and method for encoding or decoding directionalaudio coding parameters using quantization and entropy coding. United States Patent Application Publication US 2020 / 0265851 A1, August 2020.

[0103] [6] Ville Pulkki. Virtual sound source positioning using vector base amplitude panning. J. Audio Eng. Soc, 45(6):456-466, 1997.

[0104] [7] Dolby Laboratories Inc. Dolby vrstream audio profile candidate - description of bitstream, decoder, and renderer plus informative encoder description. Technical report, Dolby Laboratories Inc., 2018.

[0105] [8] Markus Noisternig, Alois Sontacchi, Thomas Musil, and Robert Holdrich. A 3d ambisonic based binaural sound reproduction system. In Audio Engineering Society Conference: 24th International Conference: Multichannel Audio, The New Reality, Jun 2003.

[0106] [9] Maximilian Neumayer. Evaluation of soundfield rotation methods in the context of dynamic binaural rendering of higher order ambisonics. Master’s thesis, Technische Universitaet Berlin, 2017.

[0107]

[10] Adam McKeag and David S. McGrath. Sound Field Format to Binaural Decoder with Head Tracking. Audio Engineering Society, August 1996.

[0108]

[11] Joseph Ivanic and Klaus Ruedenberg. Rotation matrices for real spherical harmonics. direct determination by recursion. The Journal of Physical Chemistry, 100(15):6342-6347, 1996.

[0109]

[12] Dai Yang, Hongmei Ai, C. Kyriakakis, and C.-C.J. Kuo. High-fidelity multichannel audio coding with karhunen-loeve transform. IEEE Transactions on Speech and Audio Processing, 11(4):365-380, 2003.

[0110]

[13] https: / / dlmf.nist.gov / 1.17#E25.

[0111]

[14] M. Risoud, J.-N. Hanson, F. Gauvrit, C. Renard, P.-E. Lemesre, N.-X. Bonne, and C. Vincent. Sound source localization. European Annals of Otorhinolaryngology, Head and Neck Diseases, 135(4):259-264, 2018.

[0112]

[15] Sascha Disch, Andreas Niedermeier, Christian R. Helmrich, Christian Neukam, Konstantin Schmidt, Ralf Geiger, Je´re´mie Lecomte, Florin Ghido, Frederik Nagel, and Bernd Edler. Intelligent gap filling in perceptual transform coding of audio. In Audio Engineering Society Convention 141, Sep 2016.

[0113]

[16] Sascha Disch, Andreas Niedermeier, Christian R. Helmrich, Christian Neukam, Konstantin Schmidt, Ralf Geiger, Je´re´mie Lecomte, Florin Ghido, Frederik Nagel, and Bernd Edler. Intelligent gap filling in perceptual transform coding of audio. In Audio Engineering Society Convention 141, Sep 2016.

Claims

1. An apparatus (100) for converting an audio stream having two or more channels into another representation, comprising: - a means for deriving (120, 120e, 120d) or a means for receiving one or more parameters describing an acoustic or psychoacoustic model of the audio stream, the means for deriving (120, 120e, 120d) being configured to calculate prediction coefficients as the one or more parameters; and means (110, 110e, 110d) for transforming said audio stream in a signal-adaptive manner dependent on said one or more parameters, The one or more parameters include at least information regarding at least one DOA; The apparatus (100), wherein the means for converting (110, 110e, 110d) is configured to perform a downmix of the audio stream on the encoder (200) side and / or the means for converting (110, 110e, 110d) is configured to perform an upmix generation of the audio stream on the decoder (300) side.

2. The apparatus (100) of claim 1 , wherein the prediction coefficients are calculated based on a covariance matrix or based on the one or more parameters.

3. The prediction coefficient is In particular, The formula is calculated based on the beads, and the elements of the matrix are and and but, and The apparatus (100) of claim 2, wherein the function is a real spherical harmonic function having an order and exponent of

4. 4. The apparatus (100) of claim 1, 2 or 3, wherein the one or more parameters further comprise at least information regarding a spreading factor or one or more DOAs or energy ratios and / or the one or more parameters are derived from the audio stream.

5. The apparatus (100) of claim 1, wherein the means (120, 120e, 120d) for deriving is configured to calculate a covariance matrix or a covariance matrix from the acoustic model or the psychoacoustic model.

6. The apparatus (100) according to any one of claims 1 to 5, wherein the deriving means (120, 120e, 120d) is configured to calculate a covariance matrix based on the DoA and a spreading coefficient or an energy ratio.

7. The means for deriving (120, 120e, 120d) is based on information about the diffusivity, the spherical harmonics and the time-dependent scalar-valued signal, in particular wherein: is the degree and exponent and where s(t) is a time-dependent scalar-valued signal; and / or Based on signal energy, especially where ψ represents the diffusivity, represents the signal energy for the audio stream; and / or wherein is the signal energy, and / or 7. The apparatus (100) of claim 6, configured to calculate a covariance matrix based on the formula:

8. The signal energy is calculated directly from the audio stream, and / or The energy The apparatus (100) of claim 7, wherein is estimated from the model of the audio stream.

9. 9. The apparatus (100) according to any one of claims 1 to 8, wherein the audio stream is pre-processed by a parameter estimator (232) or a parameter estimator (232) comprising a metadata encoder (233) or a metadata decoder (333) and / or by an analysis filter bank.

10. The apparatus (100) according to any one of claims 1 to 9, wherein the means (110, 110e, 110d) for converting is configured for mixing the audio streams on the encoder (200) side.

11. The apparatus (100) of claim 1 , wherein the one or more parameters comprise a predictive parameter.

12. An apparatus (100) for converting an audio stream in a directional audio coding system, comprising: - means for deriving (120, 120e, 120d) or receiving one or more acoustic model parameters of a model of the audio stream, the one or more parameters being transmitted to recover all channels of the audio stream and including at least information regarding DoA; and means (110, 110e, 110d) for transforming the audio stream in a signal adaptive manner dependent on the one or more acoustic model parameters, the audio stream is derived by transforming all or a subset of the channels of the audio stream; The apparatus (100), wherein the means for converting (110, 110e, 110d) is configured to perform a downmix of the audio stream on the encoder (200) side and / or the means for converting (110, 110e, 110d) is configured to perform an upmix generation of the audio stream on the decoder (300) side.

13. The apparatus (100) of claim 12, wherein the one or more parameters are quantized before transmission.

14. The apparatus (100) according to claim 12 or 13, wherein the one or more parameters are dequantized after transmission.

15. The apparatus (100) of any one of claims 12 to 14, wherein the parameters are smoothed over time.

16. The apparatus (100) according to any one of claims 12 to 15, wherein the transform is calculated such that correlation between transport channels is reduced by use of a Karhunen-Loeve transform or a prediction matrix.

17. 17. The apparatus (100) of any one of claims 12 to 16, wherein an inter-channel covariance matrix of an input of the audio stream is estimated from a model of the signal of the audio stream.

18. 18. The apparatus (100) of any one of claims 12 to 17, wherein a transformation matrix is ​​derived from a covariance matrix of a model of the audio stream.

19. The apparatus (100) of any one of claims 12 to 18, wherein the transformation matrix is ​​calculated using different methods for different frequency bands.

20. 20. The apparatus (100) according to any one of claims 12 to 19, wherein at least one of the transformation methods used by the means for transforming is multiplication of a vector of audio channels with a constant matrix.

21. 21. The apparatus (100) according to any one of claims 12 to 20, wherein at least one of the transformation methods used by the means for transforming uses prediction based on the inter-channel covariance matrix of audio signal vectors of audio channels.

22. 22. The apparatus (100) of claim 12, wherein at least one of the conversion methods used by the converting means uses a prediction based on the inter-channel covariance matrix based on the DOA and an additional spreading factor or energy ratio.

23. 23. The apparatus (100) of any one of claims 12 to 22, wherein the means (120, 120e, 120d) for deriving the one or more parameters is configured to process all or a subset of the channels of a first order or higher order Ambisonics input signal of the audio stream.

24. A sound scene of the audio stream, A vector of audio transport channel signals is premultiplied by a rotation matrix; the model parameters and / or the prediction coefficients are transformed in response to said transformation of the transport channel signal; and The non-transport channels of the output signal are reconstructed using the parameters of the transformed model and / or prediction coefficients.

24. The device (100) according to any one of claims 12 to 23, wherein the device (100) is rotatable in such a manner.

25. An encoder (200) comprising an apparatus (100) according to any one of claims 1 to 24.

26. A decoder (300) comprising an apparatus (100) according to any one of claims 1 to 24.

27. A system comprising an encoder (200) according to claim 25 and a decoder (300) according to claim 26, wherein the encoder (200) is configured to calculate a prediction matrix and / or a downmix and the decoder (300) is configured to calculate independently of each other an upmix matrix from estimated parameters or the one or more parameters of the acoustic model.

28. 1. A method for converting an audio stream having two or more channels to another representation, comprising the steps of: - deriving or receiving one or more parameters describing an acoustic or psychoacoustic model of an audio stream from the audio stream, the deriving step comprising calculating prediction coefficients as the one or more parameters, the one or more parameters including at least information regarding DOA; - transforming the audio stream in a signal adaptive manner depending on the one or more parameters, the transforming comprising a downmix of the audio stream on the encoder (200) side and / or the transforming comprising an upmix of the audio stream on the decoder (300) side; The method includes:

29. 1. A method for converting an audio stream in a directional audio coding system, comprising: - deriving or receiving one or more acoustic model parameters and diffuseness or energy ratio parameters of a model of the audio stream parameterized by DOA, the acoustic model parameters being transmitted to reconstruct all channels of an input audio stream and including at least information about DOA, the transmitted audio stream being derived by transforming all or a subset of the channels of the audio stream; - transforming the audio stream in a signal adaptive manner dependent on one or more acoustic model parameters, the transforming comprising a downmix of the audio stream on the encoder (200) side and / or the transforming comprising an upmix of the audio stream on the decoder (300) side; The method includes:

30. 30. A computer program for carrying out the method according to claim 28 or 29, when the computer program is executed on a computer.

Images